dataset-desk Handbook
dataset-desk is a review desk for visual datasets: the folders of images, and their descriptions, that train a style LoRA or test an image search. It reads a project, writes a report and a review queue, and leaves every decision to a person. It never removes or edits anything in the project.
The two defects it looks for
Section titled “The two defects it looks for”Repeated pictures. A training set that holds the same image twice, or two renders a few pixels apart, teaches that image twice and something else less. dataset-desk finds three kinds of repeat:
| Kind | How it’s found | What it usually is |
|---|---|---|
| Byte-identical | sha256 of the file | the same render saved into two folders |
| Near | 256-bit dHash, within 10 bits | a re-save, a small crop, a seed neighbour |
| Semantic | embedding cosine | the same subject and pose, rendered differently |
Descriptions that aren’t descriptions. When an image’s description is its generation prompt, or one sentence pasted onto dozens of images, it says nothing about that picture. A model trained on it learns the prompt; a search benchmark scored on it measures recall of the prompt. The description gate catches these without any model.
What you get back
Section titled “What you get back”report-<backend>.md: duplicates, the least typical approved images (by content and by look), the description audit, and how well each description finds its own image.description-review.jsonl: one line per approved record, with the gate’s verdict and reasons.description-worksheet.md: the flagged images, each beside its nearest siblings, ready to be re-described.
How this handbook is laid out
Section titled “How this handbook is laid out”- Getting started: install it and run the lite pass.
- Usage: each command, and how to read the report.
- The description gate: every test, with its exact arithmetic.
- Reference: flags, outputs, error codes and exit codes.
- Architecture: the modules, the caches and the backends.
- Security: what it reads, writes and never touches.
Status
Section titled “Status”v0.1.2 is a public prototype. It is tested offline (152 tests, 96% line coverage) and not yet calibrated on real data, so its thresholds are starting points. The design record and its review history are in the studio’s research library: rnd / experiments / dataset-desk.