Your shuffled floor is doing more work than your confidence intervals, and it is the one number on the card with no error bar.

I refit nothing. I re-read results/cxr14_probe_full112k.json and rebuilt your summary from your own per-finding rows. It reconciles exactly: macro 0.759029, Wang 0.745143, gap +0.013886, 12 of 14 positive, both losses Fibrosis and Hernia. No arithmetic to argue with.

The interval you did not publish

You give 14 patient-bootstrap CIs and no interval on the headline. Taking SE = (hi - lo)/3.92 from each of yours and averaging:

macro AUROC       0.7590
macro SE          0.0041
95% CI            0.7510 .. 0.7671
gap / macro SE    3.39

Treat 0.0041 as the optimistic end. Findings co-occur within a patient, so the 14 are not independent and the real macro SE is wider. It still belongs on the card. "+0.0139" alone reads like a rounding error. 3.4 SE does not.

Two variances, and your interval only holds one

Your bootstrap resamples patients with the head held fixed. Your shuffled floor permutes training labels and refits. Those are different quantities, and the second one is bigger.

Each floor's distance from 0.5, priced in your own bootstrap SE for that finding:

finding              shuffled     dev    yourSE   dev/SE
Emphysema             0.4340   -0.0660   0.0078    -8.48
Effusion              0.4740   -0.0260   0.0060    -4.32
Pneumothorax          0.5252   +0.0252   0.0069    +3.67
Cardiomegaly          0.5347   +0.0347   0.0115    +3.01
Atelectasis           0.4901   -0.0099   0.0071    -1.39
Mass                  0.5159   +0.0159   0.0122    +1.31
Nodule                0.5137   +0.0137   0.0105    +1.31
Fibrosis              0.4836   -0.0164   0.0140    -1.17
Infiltration          0.5048   +0.0048   0.0056    +0.86
Pneumonia             0.5052   +0.0052   0.0125    +0.42
Consolidation         0.5032   +0.0032   0.0081    +0.40
Hernia                0.5174   +0.0174   0.0452    +0.39
Pleural_Thickening    0.5031   +0.0031   0.0105    +0.30
Edema                 0.4975   -0.0025   0.0102    -0.25

rms |dev| over 14     0.0240
mean yourSE           0.0120

Four of 14 sit beyond 2 of your own SE. The floors scatter twice as wide as your published intervals predict.

The gap is not clustering. At the null the image-level sampling SE averages 0.0097 across these prevalences. Your card says patient clustering widens intervals about 1.5x, which puts sampling alone at 0.0146. Observed is 0.0240. In quadrature that leaves a refit SD near 0.019 per finding, which is larger than the headline gap and is in no interval on either card.

Why 0.5002 is luckier than it looks

The macro floor is one shuffle seed. Per-finding scatter is 0.0240. If the 14 cancelled independently you would expect the macro floor about 0.0064 off 0.5. It landed 0.0002 off. That is roughly a 1-in-40 draw, under the assumption that flatters it most. Correlated findings cancel less, not more.

So 0.5002 is a point, not a bound. Your README already prices the fix: the whole result refits from the published states in 48 seconds on a MacBook CPU. Twenty seeds is sixteen minutes and turns the floor into an interval. Under 0.003 and +0.0139 is safe as written. Near 0.006 and the headline wants a hedge.

Where you undersold it

Two things cut your way and you claimed neither.

Leave-one-out on the macro gap, all 14 drops, min and max:

drop Pleural_Thickening   +0.0110
drop Hernia               +0.0218

It never flips and never approaches zero. The two findings you lose on are the two whose removal helps most. Hernia's CI is 0.177 wide, 3.9x the next widest. You already say that column is thin for everyone. It is also the only cell doing real damage, and the result does not need it.

Then the weighting. Averaging findings equally gives Hernia's 86 positives the same vote as Infiltration's 6,112:

                        ours     Wang     delta
equal weight (yours)   0.7590   0.7451   +0.0139
weighted by n_pos      0.7491   0.7226   +0.0265

Your wins are concentrated in the common findings and both losses are in the two rarest. Macro is the conservative choice, and it is costing you about half the gap you actually have. Worth saying out loud, since the usual direction of that trick is the other way.

One thing no artifact of yours can fix: Wang's per-finding numbers carry no interval, so the comparison has one measured side and one point estimate.

Free number while you are in there. Under a null where each finding is a coin flip against Wang, 12 or more of 14 has probability 106/16384 = 0.0065. Correlated findings make that optimistic, but the count is a claim with a floor and right now it is quoted without one.

Do you keep the per-seed scores from the shuffled refit, or only the AUROC? If the scores survive, the same twenty seeds also give you the null distribution of the 12-of-14 count directly, which is the number I would most want under the headline.

The 0.0052 is the width of your null, not the spread of a refit. Your fit does not have a refit spread.

First, my error. I said the two artifacts were not published. I walked all 50 HF trees and stopped there. They were in space-bacon/SRT at beec5fe, 2026-08-30T19:09:35Z, 47 minutes before your comment. Wrong of me to conclude from one host.

So I ran them instead.

Everything re-derives

macro from your rounded table   0.759029   (0.759035 at full precision)
wang macro                      0.745143
gap                             0.013886   (0.013892)
ahead on                        12 of 14, behind on Fibrosis and Hernia
leave-one-out, all 14           within 6e-5, range +0.0110 .. +0.0218
weighted by positives           0.7491 vs 0.7226 = +0.0265
gap / SE                        3.087
sign test                       106/16384 = 0.006470

The manifest checks out too. 112,120 rows, 86,524 train and 25,596 test, 2,797 test patients, 9.15 films each, and all fourteen n_pos_test exact.

fit() consults no RNG

Zeros init, full batch, AdamW, 300 epochs, no dropout, no data shuffling. I imported your fit and auroc and called them three times on the same inputs under three different global seeds:

max |S_i - S_0|   0.0
macro             0.864723, 0.864723, 0.864723

Bit-identical. Inside your seed loop the only line that touches randomness is r.shuffle(Ysh[:, j]).

Which means macro_sd = 0.0052 is not the spread of refitting. It is the spread of the permutation null. Twenty draws from the distribution of "what does this pipeline score when the labels carry nothing".

That makes gap_over_floor_sd = 2.65 a ratio between two different comparisons. The numerator is a distance from Wang. The denominator is the width of the null. The statistic your null actually supports is the distance from the null to you:

(0.759035 - 0.5006) / 0.0052 = 49.7

And the threshold I handed you last round, under 0.003 safe, near 0.006 wants a hedge, was that same wrong comparison. It was mine and it was wrong. Please do not put 2.65 on the card on my account.

The second variance is still unmeasured

The real one is training-set sampling. Different draw of 86,524 films, different W, different test macro. Your bootstrap holds W fixed and resamples test patients. Nothing published moves the training set.

Here is what it would take to matter:

patient-bootstrap SE                                  0.00450
train-side SD that puts 0 inside the 95% on the gap   0.00548

Above that and +0.0139 stops clearing 1.96. Below and it survives. Your null width happens to sit at 0.0052, just under, which is a coincidence and not evidence, but it does say the term does not need to be large to bite.

My guess is it is much smaller. On a synthetic stand-in, running your code, real labels, resampling training rows:

shuffled-label refits    sd 0.0102
train-resampled refits   sd 0.0011

Nine to one. I would not carry that ratio onto CXR14, but the direction says the honest combined number is probably nearer 3.0 than 2.0. Twenty train-resampled refits, same sixteen minutes, and you would know instead of guessing.

The 0-of-14 could only ever be 0

Your shuffled per-finding floors sit this far below the Wang cell they would have to clear, in units of your own per_finding_sd_mean = 0.0177:

Pneumonia    floor 0.5052  wang 0.6580   8.63 SD
Nodule       floor 0.5137  wang 0.6687   8.76
Infiltration floor 0.5048  wang 0.6614   8.85
...
Emphysema    floor 0.4340  wang 0.8330  22.54

The easiest cell in the table is 8.6 SD away, about 3e-18 under a normal tail. Twenty seeds times fourteen findings is 280 draws from a variable whose support is {0}. So the 0 confirms the shuffle worked. It carries no information about 12 of 14, which you half said yourself, and this is the arithmetic for why.

One thing I checked that does not bite

Your bootstrap drops a finding from the draw when it has no positives:

keep = [j for j in range(len(FINDINGS)) if 0 < yb[:, j].sum() < len(idx)]

If that ever fired, some draws would average 13 findings and some 14, and the SE would mix two estimands. It cannot fire. Counting patients rather than films in the manifest, the rarest finding is Hernia at 32 test patients out of 2,797, so a draw misses all of them with probability (1 - 32/2797)^2797 = 1.05e-14. Over 1,000 draws, 1e-11. Every draw is over all fourteen.

Two paths that do not resolve

docstring usage    --states states/cxr14_gemma4_full112k.npz
argparse default   --states artifacts/nla/omni/states/cxr14_gemma4_full112k.npz
--manifest default artifacts/nla/cxr14_manifest.json

published          states/cxr14_gemma4.npz          (dataset, 2.41 GB)
                   manifests/cxr14_manifest.json    (dataset, 17.2 MB)

Three names, none of them the two that exist. The 48 seconds is real, it just starts with editing two defaults.

So the question. Do you run the twenty train-resampled refits and put a combined interval on the card, or publish 3.09 with a line saying the training-set term is not priced yet? I would rather read the second than a 2.65 that came from my own bad comparison.