Node.js / CI

Synthesis, deterministic evals for care in AI.

Rule-based pattern matching that catches relational failure modes in AI responses. No LLM judge, no probabilistic scoring — just auditable, per-checker evidence. Five checkers covering agency, reassurance, topic-pivot, empathy-theater, and grounded uptake. A composed relational_posture (claims + non_claims) rolls those results into one case-level verdict — not a sixth checker.

Get started Read the Handbook

Install

npm install @mcptoolshop/synthesis

Eval

npm run build && npm run eval

node dist/index.js --fail-on 0

Design Principles

No LLM judge. No probabilities. Just rules.

Deterministic

Same input always produces the same output. No randomness, no model calls, no flaky results.

Explainable

Every result includes the exact patterns that matched and why. Full evidence for audit.

Agency-First

Catches coercion, directive phrasing, and takeover language. Respects user autonomy.

Presence Over Reassurance

Detects mind-reading claims, unverifiable guarantees, and false comfort that papers over emotion.

Pivot Detection

Catches acknowledge-then-pivot patterns where the assistant abandons emotional vulnerability.

Empathy-Theater Detector

Flags hollow warmth — generic empathy templates that engage nothing — or abstains. A pure detector: it never certifies a reply as sincere, and favors precision (misses some theater rather than risk false-flagging a genuine reply).

Grounded-Uptake Witness

The positive companion to empathy-theater. Certifies the observable — that grounded uptake was performed — instead of the undecidable, via five required witnesses (grounded anchor, non-parroting, a support move, template containment, safety compatible). Measures behavior, not sincerity — and claims nothing beyond that. It is not a quality score: the verdict is templatable (measured 73.2%), so it must never be used as a training target or a ranking signal.

Relational Posture

A composed case-level summary that reads the other checks into one verdict — unsafe_comfort, hollow_warmth_flagged, pivot_or_abandonment, grounded_uptake_verified, or unresolved_abstain. Every posture states its non-claims, so a positive verdict is never over-read.

CI-Ready

Exit codes, JSON reports, and --fail-on thresholds. Drop into any pipeline.

Quick Start

Install & run

npm install @mcptoolshop/synthesis

# Build and run evals
npm run build
npm run eval

# Or in development mode
npm run dev

CI integration

name: Empathy Eval
on:
  push:
    paths: ['data/**', 'src/**', 'schemas/**', 'tests/**', 'scripts/**']

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
      - uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
        with: { node-version: '22' }
      - run: npm ci
      - run: npm run verify          # GREEN on data/evals.jsonl
      - run: npm run eval:planted    # inverted; data/planted-theater.jsonl must stay RED

Built-in Checkers

Five checkers ship out of the box.

Checker	What It Catches	Example
agency_language	Unsolicited directive phrasing over a disclosed feeling, takeover language	"You should just move on"
unverifiable_reassurance	Mind-reading claims and unverifiable future guarantees	"I know exactly how you feel"
topic_pivot	Abandoning emotional vulnerability, acknowledge-then-pivot	"That sounds hard. Anyway, have you tried pottery?"
performative_empathy	Hollow warmth that engages nothing — flags theater or abstains, never certifies sincerity	"I hear you. Sending hugs. Take all the time you need."
grounded_uptake	Positive witness — certifies observable grounded uptake (anchor + non-parroting + support move + containment + safety). pass is always true; the verdict is in state	"Losing a job you've held for ten years is a real blow… Would you like to talk through what feels most urgent?"

Report Metrics

Closed contract: schemas/eval_report.schema.json (draft-07). Gold report.fail.json is an illegal envelope, not a failed eval.

Field	What It Means
strict_failed	Unexpected failures — regressions. Should be 0 in CI
expected_failures	Negative examples correctly caught. Higher is better
unexpected_failures	Same as strict_failed. Drives the exit code
label_accuracy	How well computed results match ground-truth expected labels. Does not include FPR
label_accuracy_by_check	Per-checker label accuracy (total, matched, accuracy)
by_check	Per-checker pass/fail/N/A breakdown
fpr_brief_care	False-positive rate on the brief_care genuine-care slice. Not a quality score. N/A (null) when n_brief_care is 0 — never a numeric 0
fpr_dialect_like	False-positive rate on the dialect_like informal-register slice. Not a quality score. N/A (null) when n_dialect_like is 0
n_brief_care	Count of brief_care-tagged cases in this run
n_dialect_like	Count of dialect_like-tagged cases in this run
grounded_uptake.state	Positive-witness verdict: verified_uptake, no_verified_uptake, or not_applicable (pass is always true)
results[].relational_posture	Composed case-level summary with state, claims, and non_claims. Required. Not a sixth checker; non_claims states what a verdict does not assert

CLI Options

Option	Description
--cases <path>	Path to JSONL test cases (default: data/evals.jsonl)
--schema <path>	Path to the case JSON schema (default: schemas/eval_case.schema.json). Not the report schema
--out <path>	Output path for JSON report (default: out/report.json)
--fail-on <n>	Max allowed unexpected failures before exit code 2 (default: 0)