Skip to content
Synthesis

Getting Started

Synthesis is a deterministic eval framework that catches relational failure modes in AI assistant responses. No LLM judge, no probabilistic scoring — just rule-based pattern matching that produces auditable evidence.

Installation

Section titled “Installation”

From npm:

Terminal window
npm install @mcptoolshop/synthesis

Or clone and build from source:

Terminal window
git clone https://github.com/mcp-tool-shop-org/synthesis.git
cd synthesis
npm install
npm run build

Run your first eval

Section titled “Run your first eval”

The quickest way to see Synthesis in action:

Terminal window
npm run build
npm run eval

This loads the bundled GREEN pack from data/evals.jsonl, runs all five checkers, and writes a JSON report to out/report.json.

Exit code 0 means no unexpected failures.

A separate planted pack lives at data/planted-theater.jsonl. Run it with npm run eval:planted. That harness is inverted: every planted row must stay RED (schema-invalid or theater-flagged). Do not mix planted RED rows into data/evals.jsonl — npm run eval and npm run verify stay GREEN against the evals pack only.

Development mode

Section titled “Development mode”

For faster iteration without a build step:

Terminal window
npm run dev

What happens during an eval

Section titled “What happens during an eval”
  1. Load — Synthesis reads your JSONL test cases and validates each one against the JSON schema
  2. Check — Each case runs through the checkers specified in its checks array
  3. Compare — If expected labels are provided, computed results are compared against ground truth
  4. Report — A structured JSON report is written with per-case results, evidence, and aggregate metrics

Next steps

Section titled “Next steps”