v0.1.2 · prototype

A desk for your dataset, before it trains anything.

Point it at a folder of training images. It finds the repeats, the odd ones out and the descriptions that only echo their prompt, and writes a report and a review queue. Nothing is removed automatically.

Get started Read the Handbook

Install

pip install "dataset-desk @ git+https://github.com/mcp-tool-shop-org/dataset-desk@v0.1.2"

Index

dataset-desk index --project art/ --backend none --out desk-out/art

Report

dataset-desk report --project art/ --backend none --out desk-out/art

What it checks

Two defects spoil a training set quietly: repeated pictures, and descriptions that are not descriptions.

Duplicates, three ways

Byte-identical copies by sha256, near-copies by a 256-bit dHash, and semantic repeats by embedding cosine. Each pair is listed for a person to judge.

A description gate with no model

Flags text that echoes the generation prompt, wraps the subject line, is too short, repeats another description, or runs past SigLIP2's 64 tokens.

A blind worksheet

Each flagged image sits beside its nearest siblings, with no prompt and no old text in view, so the rewrite describes the picture.

Content and look, apart

Outliers are ranked twice: by what the image shows, and by a 30-number look channel of palette, tone and native texture.

A lite tier with no torch

numpy and Pillow only. Duplicates, the look channel, the gate and the report all run on a laptop CPU.

Polite on a shared machine

CPU by default with capped thread pools; a GPU needs a grant id; it never writes into the project.

Usage

Five commands. Every one reads the project and writes only under --out.

Lite pass (no model)

dataset-desk index  --project art/ --backend none --out desk-out/art
dataset-desk report --project art/ --backend none --out desk-out/art

With an embedding index

pip install "dataset-desk[models] @ git+https://github.com/mcp-tool-shop-org/dataset-desk@v0.1.2"
dataset-desk index  --project art/ --backend siglip2 --out desk-out/art
dataset-desk search --project art/ --backend siglip2 --out desk-out/art \
  --text "a squat bog creature"

Duplicates as JSON

dataset-desk dupes --project art/ --backend siglip2 --out desk-out/art > dupes.json

Coverage of a subject list

dataset-desk coverage --project art/ --backend siglip2 --out desk-out/art \
  --subjects wanted.txt

Tiers

Pick the backend your machine can carry.

Backend	Needs	Gives you
none	numpy, Pillow	copies, near-copies, look outliers, the description gate, the report
siglip2	the models extra; SigLIP2 so400m	all of the above, plus search, coverage, semantic repeats, content outliers
eg2	the models extra; EmbeddingGemma 2	the same, with joint text + picture search

Trust

Question	Answer
What does it read?	The folder named by --project: images, records/*.json, wave metadata, PNG generation data.
What does it write?	Only under --out. An --out inside the project is refused.
Does it delete anything?	No.
Network?	None in the lite tier. A model backend fetches its pinned weights on first use unless HF_HUB_OFFLINE=1.
Telemetry?	None.