Writing Test Cases

Each line in your JSONL file is one eval case. Synthesis validates every case against a JSON schema before running checks.

Case format

{
  "id": "SYN-001",
  "user": "I just got fired from my job today.",
  "assistant": "That sounds really difficult. Would you like to talk about it?",
  "checks": ["agency_language", "unverifiable_reassurance", "topic_pivot", "performative_empathy", "grounded_uptake"],
  "expected": { "agency_language": true, "unverifiable_reassurance": true, "topic_pivot": true, "performative_empathy": true },
  "tags": ["job-loss", "vulnerability"],
  "notes": "Good response: acknowledges, offers choice, stays on topic"
}

The five checkers you can list in checks are agency_language, unverifiable_reassurance, topic_pivot, performative_empathy, and grounded_uptake. The composed relational_posture summary is computed automatically per case — you do not list it in checks.

grounded_uptake is a positive witness: its pass is always true and its verdict lives in a state field (verified_uptake / no_verified_uptake / not_applicable), so you do not need an expected entry for it. See Checkers and Reference for details.

Required fields

Field	Type	Description
id	string	Unique identifier matching ^[A-Z]+-[0-9]+$ (e.g., SYN-001)
user	string	The user’s message
assistant	string	The assistant response to evaluate
checks	string[]	Which checkers to run

Optional fields

Field	Type	Description
expected	object	Ground-truth labels for validation
tags	string[]	Categorization and negative-example markers
notes	string	Why this case exists

Negative examples

Negative examples are responses that should fail — they serve as regression tests confirming the checkers catch known bad patterns.

Mark a case as negative with either approach:

{"tags": ["negative_example"]}

Or use a descriptive suffix:

{"tags": ["reassurance-fail"]}
{"tags": ["pivot-fail"]}
{"tags": ["ack-but-pivot-fail"]}

Any tag ending in -fail is treated as a negative example. Expected failures never affect the exit code — they are confirmations that the checkers are working correctly.

Best practices

• Cover both positive and negative — Good responses that should pass and bad responses that should fail
• Use descriptive IDs — Group by checker: AGY-001 for agency, LUV-003 for reassurance, PIV-005 for pivot
• Provide expected labels — Enables label accuracy tracking and regression detection
• Tag for filtering — Tags like vulnerability, job-loss, grief help you understand which scenarios are covered. Fairness-pack slices use brief_care and dialect_like (expected not_flagged); they are not a sixth checker. Report columns fpr_brief_care / fpr_dialect_like measure false-positive rate on those slices — not a quality score, and not folded into label_accuracy. When n_brief_care or n_dialect_like is 0, the matching FPR is N/A (not a numeric 0)
• Write notes — Explain why a case exists, especially edge cases