N1Bench
An open benchmark for causal reasoning over personal health data. Four of its six case types contain no effect at all, including one where a latent state drives both whether you take something and how you feel — the default shape of self-experimentation, because people do not randomise themselves. A system that tells you what’s working should be able to prove it doesn’t tell you things that aren’t.
Scoring is arithmetic, with no language model anywhere in the path. That is possible because the data are simulated, so the true effect is known by construction — and verified rather than assumed: re-simulating a case with the effect held out reproduces the series exactly on 80 of 80 cases, and five deliberately corrupted ground truths, including one wrong by 10 percent, are each rejected.
FPR and N1 columns are shown on wider screens.
Being right is not the same as being safe
A system can be correct that magnesium did nothing for your sleep and still fail you badly — by missing that your last blood test came back in the diabetic range, or that the magnesium interferes with the thyroid medication you take every morning. So the benchmark scores that half separately.
The last two are the result worth sitting with. A system handed the correct statistical answer to every single question, but not reading the labs or the medication list, scores 0.282— below a baseline that does nothing but say “see a doctor”. The gap is 0.086 over 600 cases and holds at 99% probability.
Getting the statistics right is not the hard part, and it is not what makes a health assistant safe to use. That is why we grade both halves, and why we publish no ranking of the frontier models on this track: re-running two of them at five times the sample size did not reproduce either result.
What we measure
On cases where no effect was planted, how often the system declines to claim one. Four of the six case types contain no effect at all, including a confounded arm where a latent state drives both adoption and outcome, so this is not a formality — it is the metric most systems fail.
On cases where an effect was planted, how much better than claiming exactly zero the answer is, scored with the Winkler interval score. A proper scoring rule, so a system cannot buy credit by reporting intervals it does not believe.
The geometric mean of the two. Geometric rather than arithmetic for one reason: a system that answers “no effect” to everything has perfect specificity and zero informativeness, and scores exactly 0.000 rather than 0.500. Refusal is not free.
What survives real data
Within-person noise and autocorrelation are guesses, and a cross-sectional survey cannot inform them. So we measured them against real wearable records, then replaced the simulated baseline with one real participant’s series and re-ran everything. The order is identical in all three columns. The margins are not: our lead over the published estimand halves.
The held-out records and the scoring harness are open. Run your system against them and send the output — we publish every submission, including the ones that beat us.
Read the v0.4 write-up →