NagaYu/signpost-classifier

Signpost label-quality classifier (hashed backend)

Judges whether an accessibility label actually identifies the control it is attached to -- the question left over after axe-core and Xcode's Accessibility Inspector have both passed. There is a name; does the name mean anything on this screen?

Repository: NagaYu/signpost-classifier

Code and the full evaluation	https://github.com/NagaYu/signpost
Dataset it was trained on	https://huggingface.co/datasets/NagaYu/signpost-label-quality
Try it in your browser	https://huggingface.co/spaces/NagaYu/signpost

It classifies one label into eight classes: ok plus seven ways a label fails (generic, role_echo, identifier, duplicate, verbose, missing, mismatch), and reports the character span of the offending part.

How to use it

from signpost.context import ContextBuilder, ContextConfig
from signpost.model import QualityClassifier

clf = QualityClassifier.load("path/to/this/checkpoint")
builder = ContextBuilder(ContextConfig(level="full"))  # must match the checkpoint
bundles = builder.build_screen(screen)               # a signpost.types.ScreenTree
for bundle, prediction in zip(bundles, clf.predict(bundles)):
    print(bundle.name, prediction.error_type, round(prediction.confidence, 3))

or from the command line, over an axe run, an XCUITest dump or a uiautomator dump:

signpost check tree.json --model path/to/this/checkpoint

The context level is pinned in the checkpoint and enforced on every call: feeding it bundles built at another level raises rather than returning numbers that look fine and mean nothing.

What was measured

Held-out origins -- no site or app on the test side appears in training -- with 860 labels, 309 planted defects and 511 labels the strict quality gate calls good.

what was measured	value	on
of the planted defects the rule baselines missed, share Signpost reported	54.6% [48.3%, 60.8%]	238 missed defects
the same, with the right error type	53.4%	238 missed defects
good labels this model reported anyway (over-flagging)	0.59% [0.20%, 1.71%]	511 good labels
the same, for the rule baselines	0.00%	511 good labels
hard set: defects every syntactic check still passes	40.6%	n=96
error-type accuracy over the eight classes	0.824	-
macro F1 over the eight classes	0.710	-
median time to judge one label	0.054 ms	17585 labels/s

The operating threshold is 0.644, chosen on the training split as the most talkative cut whose measured over-flagging stayed inside a 2% budget. The test numbers above were computed at that threshold and it was never tuned on them.

Per class

class	precision	recall	F1	support
ok	0.79	0.99	0.88	515
generic	1.00	0.69	0.82	55
role_echo	0.75	0.19	0.30	48
identifier	0.94	0.57	0.71	54
duplicate	0.74	0.39	0.51	36
verbose	1.00	1.00	1.00	36
missing	1.00	0.96	0.98	27
mismatch	0.94	0.32	0.48	53

Does the context help?

The same backend, the same hyperparameters, the same seed and the same screens, trained once per context level. The only thing that varies is what the model was allowed to look at.

context level	what the model may see	macro F1	over-flagging	hard set
none	role and name -- what a rule checker sees	0.611	2.90%	10.4%
local	+ value, hint, visible text, icon asset, traits	0.612	2.12%	10.4%
full	+ siblings, neighbouring text, screen title, inferred action	0.774	0.97%	49.0%

Training

• Corpus: signpost/data/seed_screens.jsonl (synthetic)
• Split: by origin, 30% of nodes held out, stratified by language; train and test share no origin, checked before training.
• Defects: planted by signpost/inject.py at a rate of 35%, with a different seed on each side of the split so a test defect is not a training defect relocated.
• Backend: hashed, class weighting balanced, seed 0.
• No base model. The hashed backend is a matrix of floats over hashed context features, trained from scratch. There is no encoder to attribute, which is why the frontmatter carries no base_model.
• Fit: 0.7 s on Darwin arm64 python 3.13.5.
• No rewrite adapter is included. The template rewriters in signpost.rewrite need no model and are what signpost check uses by default.

Limitations -- read these before quoting a number above

• The corpus is synthetic. Every screen was composed by scripts/make_seed_corpus.py from a vocabulary written for this project, and every defect was planted by a generator this project also wrote. A model evaluated here is partly being measured on how learnable that generator is. Real screens are messier: more abbreviations, more house style, more labels that are bad in ways nobody thought to plant. Treat these figures as an upper bound on a real codebase rather than an estimate of one.
• Some classes it is bad at. Weakest in this run: role_echo recall 0.19 on 48 rows; mismatch recall 0.32 on 53 rows. In the project's reference run (python benchmarks/run.py, seed 0) the two weakest are role_echo recall 0.21 and mismatch recall 0.32 -- and mismatch is the class this project exists for. A defect this model misses is a defect nobody is told about.
• It is not a replacement for anything. Not for axe-core, not for Xcode's Accessibility Inspector or Android's lint checks -- they are better than this model at the classes they cover, missing above all -- and not for review by people who use assistive technology. It narrows what a human has to look at.
• It does not know what anyone will experience. Every finding points at a documented guideline (WCAG 2.2, Apple's and Google's own guidance) through signpost.cite, and says nothing about how any person will perceive a screen.
• duplicate is a property of a screen, not of a label. A row taken out of its screen cannot be judged for that class.
• The threshold is a policy, not a fact. A team gating a merge queue can afford fewer false alarms than a team reviewing a hundred screens by hand; calibrate_threshold is how they pick their own, and set_threshold is how they impose it.

Reproducing this checkpoint

python scripts/train.py --backend hashed --seed 0 --test-frac 0.30 --inject-rate 0.35

Same seed, same bytes: the hashed backend trains full-batch with no sampling anywhere in the loop.

Licence

mit. The corpus is synthetic and carries no third-party redistribution question; signpost/collect.py is the path for real pages and keeps per-row provenance.