FlipActPreprint, 2026

Sensitivity or Criterion?Measuring and Moving When Language Models Act

Paper arXiv Code and data soon BibTeX

Two models can have the same accuracy and need opposite fixes.

TL;DR
  1. One accuracy hides two failures. Before an agent acts it must decide whether to act at all. A model that fails to see when acting is warranted and one that sees it and holds back get the same score.
  2. FlipAct measures them separately. A sensitivity and a criterion for every model, on an evidence scale computed by a rule, so the numbers are comparable across models.
  3. The criterion can be moved on its own. Most open models hold back more than the evidence warrants. Steering and distillation shift that, with no significant change in sensitivity.
part one, background

Why one accuracy is not enough

Three short steps from psychology: what an accuracy hides, how signal-detection theory splits it, and where the same split shows up in agents.

one

Two observers, one accuracy, opposite mistakes

Start with a case where the stakes are obvious. Two doctors each read the scans of 100 patients who have a tumour and 100 who are healthy. Their accuracy is 65% and 64%. By that number they are the same doctor.

the same 200 cases

Accuracy adds the two kinds of error together. What to do about them depends on which kind you have.

two

Sensitivity and criterion: try it on yourself

Signal-detection theory was worked out in the 1950s for radar operators and for listeners in hearing experiments. It says every yes-or-no judgement under uncertainty comes from two separate things.

Sensitivity d′

How well you can tell a signal from none. It is the distance between the two curves.

low: hard to tell apart
high: easy to tell apart

Criterion c

How sure you want to be before you say yes. It is where you put the line.

to the left: says yes freely
to the right: holds back

Make a miss expensive and people move the line left. Make a false alarm expensive and they move it right. What they can see has not changed. Try it: a patch of static will flash, and half the time it hides a faint blob.

you willdifficulty
the stakes

Why, and the scoring table

Why three settings? They change only which mistake is expensive. The pictures and sounds stay just as hard. If the theory holds, how well you can tell stays the same from one setting to the next, and how readily you say yes moves. Run two of them and compare.

What a trial looks like
Nojust static
Yes, made obviousbrighter in the middle. Only for reference.
what a Yes looks like at this difficulty
  1. You answer yes or no, and see right away whether you were right.
  2. After 20 trials you get two numbers: how well you can tell, and how freely you say yes.

Now the full picture, with both knobs. It is the same for a listener, for the two doctors above, and for a language model deciding whether to call a tool.

the setting
the observeror move the sliders
balanced accuracy

Moving the line trades one error for the other. Only pulling the curves apart removes errors.

three

The same split, wherever an agent decides to act

The doctors' problem is every agent's problem: act, or hold back. Pick a decision and a kind of model, and see which error it makes and what that error costs.

the decision
the model
what a miss costs here
what a false alarm costs here
part two, the paper

FlipAct

What the paper builds and what it finds: an evidence scale computed by rule, a sensitivity and a criterion for 44 open models, and two ways to move the criterion alone.

four

Evidence computed by rule, not written by a model

Signal-detection theory already gives d′ and c, so why a new benchmark? On a benchmark with yes-or-no labels, the criterion it recovers is relative to that benchmark's mix of examples, and no zero is shared across benchmarks. To compare criteria across models, every question needs an evidence score on one scale with a fixed zero. FlipAct computes it from qualitative physics: directions and coarse step sizes, no physical units.

A state

Slightly, very or dangerously off, in one of two directions. Normal is zero.

A tool

Pushes the state one way, by one, two or three steps.

A score, ΔU

How much closer to normal the tool leaves the state. Yes exactly when ΔU > 0.

build a questionpick a state and a tool, and the rule does the rest

The state v, from dangerously dry (−3) to dangerously wet (+3). Normal is 0.
The tool
One quad: same knowledge, flipped state

The design unit is a 2 × 2 quad: two tools, each shown in the state it fixes and in the state it worsens. Every question of a quad needs the same two facts, what the tool does and what the room needs. Only the state changes, and with it the answer, so a model that knows what a humidifier is for must still read the state.

149,840candidate entities
2,876records annotated by language models
1,638pass the screens
1,422tool pairs
6,176questions, answers and ΔU by rule
1,880after balance selection

Language models help only in the two indigo steps. The rule decides every state, answer and score.

five

Reading a model: a slope and a crossing point

Because every question carries a score, a model's Yes-rate can be fitted as a curve over that score. The slope a is its sensitivity, in the role of d′. The score τ where the curve crosses one half is its criterion. The rule's boundary sits at zero for every model, so a positive τ means the model says Yes less often than the rule warrants.

a model that is
Steeper means the model separates helpful from harmful actions more cleanly.
Above zero is conservative, below zero is permissive.
says Yes to helpful actions
says Yes to harmful ones

The curve is an illustration of the fit, drawn from the two numbers you set. Measured Yes-rates look more like a step, as the footnotes below say.

0.34 to 0.58

Sensitivity behaves like a capability

In the Qwen3 family, bigger models tell helpful from harmful actions better. The ranking agrees with sensitivity on other benchmarks.

18 of 21

The criterion is conservative by default*

Most open models hold back more than the evidence warrants.

+2.18 vs −1.10

It is a per-model setting, not a size effect†

Two releases of Mistral-Small-24B, the same size, land on opposite sides of zero.

Models read the sign of the evidence, not its size.‡ Actions that leave the state no better are accepted about as often as helpful ones.

  1. *Counted over the 21 models, run with reasoning off, whose criterion is well determined. Whether this is only a preference for the word No is answered in the questions below.
  2. †Measured on FlipAct. It does not consistently predict a model's criterion on other benchmarks: across models the correlation with When2Call is near zero (r = −0.01). What carries over is the shift an intervention causes, shown in the next section.
  3. ‡Yes-rates jump between a score of −1 and 0 and are flat on either side. In 9 of the 21 models the Yes-rate never reaches one half, so τ is where the fitted curve, not the data, crosses it. The paper therefore reads τ by its sign and rank.
six

Moving the criterion, and only the criterion

If the criterion is a separate setting, it should move while sensitivity stays put. Three interventions, two of which aim at exactly that.

Steering

At inference. A direction read from the model's own activations is added during generation. Weights stay fixed.

moves the criterion

Criterion distillation

Into the weights. The model is distilled from its own steered copy.

moves the criterion

Sensitivity learning

Training on the computed labels does the opposite job. The curve gets steeper, and its crossing point moves close to zero.

raises sensitivity, moves the criterion too
the student
shown
sensitivity a
criterion τ
held-out accuracy
23 of 23

It carries to other benchmarks

After distillation, the criterion moves toward acting on every scored model and benchmark, across six external benchmarks.

±0.02

Sensitivity is kept*

On five external benchmarks, the criterion-free AUC is equivalent to no change within this bound.

up to +6 pts

Accuracy follows the starting point†

Models that start conservative gain accuracy. Models that start permissive do not.

  1. *AUC is computed from answer log-probabilities, which five of the six benchmarks provide; each change passes a two one-sided equivalence test at ±0.02. On FlipAct itself, no student's sensitivity changes significantly after distillation. An equivalence test bounds that change within ±0.12 for two of the four students and cannot decide for the other two.
  2. †Balanced accuracy. Where a student starts permissive it falls by up to four points or stays within its error, except Mistral-24B on When2Tool, which falls about ten: its hit rate is already at ceiling, so acting more only adds false alarms. A criterion shift should take its direction from the target benchmark.

Steering, with the weights fixed

On a tool-calling benchmark, steering moves all five models toward acting.‡

−0.40−0.200+0.20+0.40+0.60+0.80← acts more freelyholds back →Qwen3-8B−0.32Qwen3.5-27B+0.01gemma-2-9b+0.59Mistral-24B−0.13phi-4−0.33criterion c on When2Call
unsteeredsteeredswipe to see all →
Show the numbers
call ratecriterion c
ModelunsteeredsteeredunsteeredsteeredΔ accuracy
Qwen3-8B0.560.60−0.18−0.32−0.022
Qwen3.5-27B0.470.50+0.10+0.01+0.005
gemma-2-9b0.240.29+0.78+0.59+0.002
Mistral-24B0.370.55+0.39−0.13−0.032
phi-40.570.59−0.30−0.33−0.038
  1. ‡Acting more is not always better. Steered toward acting on five benchmarks, the five models lose balanced accuracy on 12 of the 25 model and benchmark pairs. Choosing the direction and strength from fifty labeled examples of each benchmark leaves one pair worse.

A when-to-act benchmark says more when it reports d′ and c beside accuracy.

faq

Questions readers ask

How does this relate to ROC, AUC and F1?

ROC. Keep sensitivity fixed and slide the criterion from one end to the other. The hit rate and false-alarm rate trace one ROC curve. A higher d′ gives a curve that bows further toward the top-left corner. Changing the stakes moves a person along the same curve.

AUC. The area under that curve measures the same thing as d′, with no criterion in it. When the two distributions are Gaussian with equal variance, AUC = Φ(d′/√2): a d′ of 0, 1, 2 and 3 gives an AUC of 0.50, 0.76, 0.92 and 0.98. AUC needs a score for every example. d′ needs only one hit rate and one false-alarm rate, at the price of the Gaussian assumption. The paper reports d′ and c on every benchmark, and adds AUC on the five that provide answer log-probabilities.

F1 and accuracy. Both are read at one criterion, so they mix how well a model can tell with how readily it says yes, and both depend on the share of positive examples. Two models with the same d′ can differ widely in F1 only because their criteria differ.

Is a conservative criterion just a preference for the word No?

Partly, for some models. A criterion collects every bias in the response, including a preference for one answer word, so the paper asks every question again in reversed wordings such as “Is using the tool a mistake?”, where No means act. A preference for No then pushes the two wordings in opposite directions, which separates it from a threshold on the evidence. For the three Qwen3.5 models and GLM-4-32B the preference explains the conservative reading. The other 12 of the 16 models tested stay conservative without it.

The same question applies to distillation. On FlipAct most of the distilled shift is a shift in the preference for Yes. On When2Call and BFCL-miss-func, whose answers have no Yes, the criterion-distilled student still moves further than one distilled from a teacher with only its Yes logits raised, for both students tested this way.

cite

BibTeX

@article{dong2026flipact,
  title   = {Sensitivity or Criterion? Measuring and Moving When Language Models Act},
  author  = {Dong, Sixun and Hu, Yebowen and Yin, Ming and Chen, Yiran and Liu, Fei and Chen, Chen},
  journal = {arXiv preprint arXiv:2610.15415},
  year    = {2026},
  url     = {https://arxiv.org/abs/2610.15415}
}

Questions or collaboration: [email protected]