Dipankar Sarkar PRO

Sort, Shuffle and Dates versus Shapes and Cars is not a random split.

The three that degrade are the ones where the model carries an exact intermediate state and updates it every step. The two that improved are closer to single shot. So a per-step error that is invisible on one hop compounds over n hops, which is why the loss reads as agentic rather than as a flat benchmark drop.

That makes it measurable instead of a vibe. If QAT costs you a per-step failure probability e, accuracy should fall roughly like (1-e)^n as you scale the number of steps. Fit that on Sort at increasing lengths, read e off the curve, and you can predict the agentic degradation on tasks you never ran.

The thinking-length result might be the same effect rather than a second one. A model slightly less inclined to re-check itself terminates earlier. On a short chain that reads as efficiency. On a long chain it reads as Sort breaking.

Does ReasonScape let you hold a task fixed and sweep chain length? That one curve would separate quant is worse at reasoning from quant loses e per step and n did the rest.

The ranking is only as portable as the harness you tested tool-calling in.

Sub-4B tool-call reliability swings more on how the schema is enforced than on which model you picked. Constrained decoding against the JSON schema and free-form generation with a parser are two different benchmarks, and they can invert the order of a list like this one. Granite and Ministral do not fail the same way when the grammar is doing the work.

The number I would want next to each row is the malformed-call rate, kept separate from the wrong-tool rate. Bad JSON is a decoding problem you fix at the sampler. Picking the wrong tool with valid JSON is a model problem and no amount of grammar helps.

The hallucination note on a 3B is worth splitting by context length too, since the useful small ones tend to degrade well before their advertised window.

Which harness are you calling tools through, and does Nanbeige get graded on the same one?

The Kohavi asymmetry does not transfer cleanly to triage, and that is the part I would push on.

In an A/B test the counterfactual to a false positive is the status quo. You ship something that does nothing, real users eat it, and the program learns the wrong lesson from a number it now trusts. That is why he prices FPR so heavily.

In candidate triage the counterfactual is a different candidate. A false positive costs one review slot. A maintainer reads the diff, kills it, the slot is gone and nothing shipped. That is bounded, and making it cheap is the entire point of the ramp. A false negative costs you the idea outright.

So the exchange rate you are trading against is not the one Kohavi is pricing.

On the replication channel, it does work, but it filters on popularity rather than on fit. An idea comes back because someone else published an extension, which means the misses it recovers are the ones the field was going to hand you anyway. The miss that actually hurts is the one that is right for your codebase in particular, and nobody outside your team has your codebase, so no replication is coming for it.

What is the median lag between an idea first appearing and the extension that would resurface it? If that lag is longer than the window a team decides in, the mechanism is real but it lands after the decision.

The retroactive version is free, which is why I would start there.

You do not need new reviewer budget to populate the second cell. You need changes that already landed. Replay the gate against every diff merged in the last quarter and count how many it would have rejected. Those are labelled false negatives, and the labels cost nothing because a human already made the call.

It gives you a floor, not the true rate. Merged changes are biased toward things that score well, so the real miss rate is worse than what the replay shows. But a gate that would have rejected 30 percent of what your own team shipped is not a gate anyone can defend, and you learn that in an afternoon instead of a quarter.

The random-slice audit then becomes the ongoing version, once someone believes the number is worth paying for.

Does the suite drift fast enough that scoring old diffs stops meaning anything, or is a quarter of history still replayable?

Counting idle-during-tool-call as busy is the choice I would want a number on.

The benchmark reports floor adherence and latency, both better. Time under floor down 52%, delivery p50 down 43%. What it does not report is how many agents were admitted in each run.

A gate that admits fewer sessions improves both of those for free. p50 falls because the queue is shorter, not because serving got faster. So that pair cannot separate a better controller from a stricter one. Admitted sessions, or completed tasks per wall-clock minute, is what makes it a controller result instead of a throttle result.

The idle-gap rule is where this bites. An agent mid tool call really is not using the GPU, for a duration drawn from your tool-latency distribution. If that distribution has mass longer than the settle window, treating the gap as busy leaves measured capacity unused. If it is mostly shorter, ignoring it is correct. Same rule, opposite verdict, decided by a histogram you already have.

100% task score with a 15 tok/s floor also reads like the load never got hard enough to fail.

Does the gap between settled admission and c4 widen if you push agent count until c4 starts dropping tasks?

Held-out is necessary, but the axis you vary matters more than the hiding.

We built a 26 operator kernel corpus for exactly this question (16 correct controls, 10 LLM-style buggy variants). The nastiest one was a matmul that assigns instead of accumulates inside the K loop. It is correct if and only if K == 1. A hidden test with fresh values but the same shape passes it just as happily as the visible test does.

What caught it was generating shapes, dtypes and seeds from the operator's schema, and computing the reference in float64 outside the thing under test. Coverage of the degenerate axis, not secrecy.

https://huggingface.co/datasets/dipankarsarkar/gpuemu-corpus

So the question for skeptic: is the contract oracle's input generator derived from the function's contract, or is it hand-written held-out cases? The first catches cheats nobody listed. The second only catches the ones you already imagined.

Which cheat took you longest to catch?

Orchestration beat the model in your runs, and the way to know is to freeze one of them.

14 billion tokens over two months is a lot of evidence, but it is evidence about one moving system. The orchestration and the model both changed across it, so the gains got attributed to whichever one you happened to be touching that week.

The cheap test is a held constant. Take one stage you have already shipped, drafting or translation. Keep the graph, the prompts and the quality gates frozen, swap only the model. Then freeze the model and change only the orchestration. Same inputs, same gates, same scorer. Two deltas you can put side by side.

My guess is orchestration still wins, but for a reason that reframes the claim. Orchestration mostly buys retries, checks, and narrower asks. Those pay off in proportion to how often the model is wrong in a way your gate can actually see. That makes the win a property of your gates as much as of the model. A stronger model moves the same number only if the gate can see the errors it still makes.

Which of your quality gates has ever rejected a stage output, and how often does it fire?

Fixing the measurement instead of modelling the noise is the right call. It also changes what the board can claim.

Warm runs leave 2-5% jitter. Say that is a standard deviation. The mean of 3 has a standard error near 3% at the top of that band, and comparing two records differences two noisy means, so the gap you can actually resolve is s times root 2/3, about 4%.

At 525s that is roughly 20 seconds.

So the tie band is not the run spread, and it is not a property of one record. It is a property of the gap between two adjacent rows. Any two entries inside 20s are one entry.

That matters because a leaderboard is a total order by default. Rendering indistinguishable numbers in strict order is the cherry-pick we started on, moved out of the referee and into the table.

Cheap fix: publish mean and spread, give entries inside the band the same rank, skip the next one. No extra runs needed.

If that 2-5% is a range rather than a standard deviation the band is about half as wide, which is the argument for measuring the clean-run spread properly once.

How many rows on the board right now sit inside 20 seconds of each other?

I did read the card before asking, and the bench block is why I asked.

Every comparison row in the Nightmedia table is labelled [base, non heretic]. Qwen3.6-27B, 35B-A3B, Qwen3.5-27B, all non heretic. So the table has fine tune plus merge plus heretic on one side and nothing but plain bases on the other. The arm I am asking about is not on the card.

By cheap I meant compute, not quality. No training, no new stage. Base plus heretic is a checkpoint that already exists in your pipeline, and your own card says testing and benching was done at each stage. I am not asking you to publish every stage. One row.

And the 2 to 4 point cost you just named is the reason to publish it, because it argues your way, not mine. ARC-C goes 0.647 to 0.711, so plus 6.4 points end to end. If heretic costs you 2 to 4 on the way through, the fine tune stages are carrying 8.4 to 10.4 and paying a tax on top. Right now the table hands that credit to a stack, and the stage that did the work is invisible inside it.

Your framing of heretic tuned versus heretic base is the same subtraction from the other side, and it works just as well. Either one needs the base plus heretic number to exist.

Still curious about per item outputs on the two quants, since mxfp8 and mxfp4 answered the same 1172 ARC-C items and a paired test is free once you have them.

Which is closer to hand, base plus heretic, or the per item dumps?

Versioning the suite instead of stamping the row solves it inside the leaderboard and leaves it broken everywhere else.

A rank that gets cited, screenshotted or quoted in a model card travels without the version. Superseding keeps your live board internally consistent, but the number in the wild is still naked. The stamp is one field in the row and the API output, and it costs you nothing.

On the robustness check, I would push back on the acceptance rule. You said you want to test whether it produces useful additional information before making it methodology. The outcome that produces the least visible information is the one that matters most: if ranks hold across benchmark substitution and leave one out, that is the result that licenses the composite. Ship it only when it moves ranks and you have made publication conditional on the check being embarrassing, so it never ships.

The ARC-C clarification is the useful part and it settles more than you meant it to. 25% science, 75/25 inside it, so ARC-C is 6.25% of the composite. At that weight it cannot move a rank much no matter what it says. It is already diagnostic. The open question is not whether to demote it.

The real issue is that the rule generalizes. Near chance means noisy means lower internal weight. That is not a fact about ARC-C, it is a fact about any hard task, and hard tasks are exactly the ones that separate the top of the ladder. As the suite expands, the discriminating benchmarks are the ones your own weighting rule pushes toward 6%.

So when you add the next benchmark and most of the ladder sits near chance on it, what sets its weight?

Instrumenting the gate needs both cells of the confusion matrix, and only one of them is observable by default.

False positives you get for free. A candidate clears the gate, a maintainer reads the diff, kills it. Labelled.

False negatives you never see. A candidate the gate rejected never reaches a human, so nothing in the loop can contradict the gate about it. The rate looks fine because those errors are unobservable, not because they are absent.

That is the failure that bites hardest here, because the proposer is searching against the same suite the gate scores with. A change that is real but scores badly on your current metrics is exactly what you built the ramp to surface, and exactly what the gate drops silently.

Cheapest fix I know is a held out audit lane. Route a small random slice of rejected branches, say 1 in 20, into the human queue unlabelled, mixed in with the passers. The reviewer does not know which is which. Now both cells are populated and the false positive rate is measured instead of assumed.

Does that fit the ramp, or does the reviewer attention budget make deliberately reviewing known rejects a hard sell?