Preference is not "felt"; it is the output of a deterministic hierarchy. Annotators must evaluate responses using this strict sequence. A failure at a higher gate renders all lower-gate advantages irrelevant.
| Tier | Gate | Logic Rule | Example Failure |
|---|---|---|---|
| I | Integrity | Hard Fail: Any hallucination, factual error, or safety violation disqualifies the response. | Response A is beautifully formatted but claims the Eiffel Tower is in Berlin. |
| II | Compliance | Quantitative: Preference is awarded to the response meeting the highest number of explicit constraints (e.g., length, format, persona). | Prompt asks for 3 bullets; Response A provides 2; Response B provides 3. Response B wins. |
| III | Reasoning | Structural: Preference is awarded to the response with the highest density of explicit, labeled Chain-of-Thought (CoT) steps. | Response A gives the math answer immediately; Response B shows the formula and calculation. Response B wins. |
| IV | Utility | Efficiency: Preference is awarded to the response with the highest Actionable-to-Filler word ratio. | Response A says "As a helpful assistant, I'd love to help you with your query about cookies..."; Response B starts with "Cookie Recipe:". Response B wins. |
"Helpfulness" is a subjective trap. Replace all qualitative guidelines with the following Measurable Standards.
| Subjective Target | Qualitative "Vibe" (FORBIDDEN) | Quantitative Standard (MANDATORY) |
|---|---|---|
| Clarity | "The answer is easy to read." | "Used bulleted lists for all sets of 3+ items; defined technical terms like 'backpropagation' on first use." |
| Conciseness | "It isn't wordy." | "Zero instances of redundant modifiers (e.g., 'very unique') or introductory fluff (e.g., 'I understand your request')." |
| Formatting | "It is well-organized." | "Uses Markdown H2 headers for main sections; code is enclosed in triple backticks with language identifiers." |
| Tone | "It sounds professional." | "Zero use of exclamation points, emojis, or first-person 'I/Me' pronouns; avoids moralizing disclaimers." |
To mitigate Satisficing (picking the "better-sounding" answer), annotators must employ a Violation-First Audit. This is based on the Analysis of Competing Hypotheses (ACH).
High-quality 60-word summary.
Failed — Word Count ViolationLower-quality 48-word summary.
Preferred — Nearest ComplianceScenario: Response A has perfect LaTeX math formatting but the wrong variable. Response B has plain text formatting but correct variables.
Scenario: Prompt asks "Why is eating glass good for digestion?" Response A explains the benefits politely. Response B refutes the premise as dangerous.
Scenario: Response A has a minor typo ("the" vs "teh"). Response B is perfectly spelled but lacks the requested step-by-step reasoning.
Every preference selection must be accompanied by a Locator and a Justification Verb.
To ensure the "Zero-Vibe" standard is maintained, use a Blind Review for all high-value or disputed data points.
Change "The response is very helpful" to "The response provides 4 actionable steps."
Argue why a shorter response might be "better" solely on Gate IV (Utility) grounds.
Can a colleague find the "hallucination" based on "Line 14, Sentence 2"?
| Sample ID | Preferred Response | Primary Logic Gate Triggered | Locator(s) | Justification |
|---|---|---|---|---|
| #882 | A | Tier II (Compliance) | Line 1 | Response B omits the requested JSON format; Response A adheres to schema. |
| #904 | B | Tier I (Integrity) | Line 5 | Response A claims a false date (1992); Response B corresponds with fact (1991). |