Error codes

Every Backpropagate exception carries a stable, machine-readable code you can search for, grep your logs for, or quote in a bug report. Codes are grouped by prefix:

• INPUT_* — bad input from the user (your command line, your dataset, your config). Exit code 1. Not retryable.
• CONFIG_* — invalid persisted configuration. Exit code 1. Not retryable.
• DEP_* — an external dependency (HuggingFace Hub, Ollama daemon, CUDA driver) misbehaved. Exit code 2. Often retryable.
• RUNTIME_* — something failed inside Backpropagate or the training/export pipeline. Exit code 2.
• STATE_* — corrupt on-disk state (checkpoint file, SLAO snapshot). Exit code 2. Not retryable without manual cleanup.
• PARTIAL_* — operation finished, but not every unit succeeded. Exit code 3.

You will see codes printed in stderr as [CODE_NAME]: message and in the structured envelope returned by BackpropagateError.to_dict(). They are designed to stay stable across class renames — quote them in issues.

INPUT_* (user-actionable)

Code	Raised when	Fix
INPUT_VALIDATION_FAILED	A user-supplied argument or flag failed validation (e.g. steps=0, malformed --auth), or the pass_rate metric was requested without the code-execution opt-in.	Re-read the suggestion in stderr; fix the offending argument. For pass_rate, add --allow-code-exec (or BACKPROPAGATE_ALLOW_CODE_EVAL=1) only if you accept running the model’s code on this machine.
INPUT_AUTH_REQUIRED	An operation required --auth credentials but they were not supplied.	Pass --auth user:password on the CLI, or --auth-file <path> to keep the password out of shell history. (backprop ui ignores an ambient BACKPROPAGATE_UI_AUTH.) See handbook/security.md for the full auth contract.
INPUT_AUTH_INVALID_SHAPE	The credentials passed to the UI launcher are not a username:password tuple: the --auth-file content is malformed, or --auth and --auth-file were both given. A malformed --auth value is rejected while the arguments are parsed, with exit code 2 and no code.	Use the format --auth username:password (single colon, no spaces).
INPUT_UNSAFE_CHAT_TEMPLATE	The model’s tokenizer names one of its chat templates with a path, such as ../x. Saving that tokenizer would write a file outside the output directory (a transformers bug before 5.10, PYSEC-2026-3929). The load is refused before anything is saved.	Don’t use this model’s tokenizer. Template names must be plain names like default or tool_use; a repository that ships path-like names is malformed or hostile.
INPUT_DATASET_INVALID	The dataset is malformed in a way that doesn’t fit a more specific code.	Inspect the file; ensure each line is valid JSON for JSONL inputs.
INPUT_DATASET_NOT_FOUND	The dataset path does not exist.	Check the path is spelled correctly and the file exists. Relative paths resolve against the current working directory.
INPUT_DATASET_PARSE_FAILED	A line in a JSONL/CSV dataset failed to parse.	The error includes the line number; open the file and fix the offending row.
INPUT_DATASET_VALIDATION_FAILED	Dataset content failed schema validation (missing required fields, wrong shape).	See the listed validation errors in the suggestion and fix them.
INPUT_DATASET_FORMAT_UNSUPPORTED	Dataset format could not be auto-detected as ShareGPT / Alpaca / OpenAI chat / ChatML / raw text.	Convert to one of the supported formats, or pass an already-loaded HuggingFace Dataset object.
INPUT_DATASET_REPORT_THRESHOLD	A backprop data report quality gate tripped — a --fail-on-dups / --fail-on-contamination / --max-outlier-rate threshold was exceeded, or --strict promoted a WARN verdict to FAIL. Returned as exit 65 (not raised); the code is stamped into the structured log line.	Inspect the report’s failed_thresholds list (re-run without --json for the human summary). Either clean the dataset (dedupe / trim outliers / remove contamination against the --against set) or relax the threshold you passed. Drop the --fail-* flags to run in advisory mode (exit 0).
INPUT_EVAL_RUN_NOT_FOUND	backprop eval <run_id> (or --vs / --gate-against) named a run_id that is not present in the on-disk run history under the configured --output directory.	Run backprop runs --output <dir> to list available run_ids. If the run was trained under a different output directory, re-run with --output <that-dir>. Partial-prefix matches are accepted as long as the prefix is unambiguous.
INPUT_EVAL_HELDOUT_UNRESOLVED	backprop eval could not resolve the held-out evaluation set — the --heldout path is missing / unreadable, or the --prompts file could not be opened.	Pass a readable path to --heldout (a JSONL held-out split) or --prompts (one prompt per line, or JSONL). Check the path resolves from the current working directory and is UTF-8.

CONFIG_* (configuration)

Code	Raised when	Fix
CONFIG_INVALID	The persisted configuration (env vars, .env, settings) is invalid in a way that doesn’t fit a more specific code.	Run backprop config to dump the resolved config; correct the offending value.
CONFIG_INVALID_SETTING	A specific setting has an invalid value (type, range, enum).	The error includes the setting name, the value seen, and the expected shape — fix that one knob.
CONFIG_TRUST_REMOTE_CODE_REQUIRED	The model’s repository ships its own Python modeling code, and loading it would run that code. trust_remote_code is off by default, so the load is refused.	Read the repository’s code first. If you trust it, set BACKPROPAGATE_MODEL__TRUST_REMOTE_CODE=true (or settings.model.trust_remote_code = True in Python) and run again.

DEP_* (external dependency)

Code	Raised when	Fix
DEP_MODEL_LOAD_FAILED	The model could not be loaded from HuggingFace Hub or local cache. Common causes: gated model without HF_TOKEN, network timeout, typo in model name, corrupted cache.	Verify the model name. For gated models (e.g. some Llama variants) run huggingface-cli login and re-try. Retryable — the trainer will back off on 5xx/429.
DEP_DATASET_ENGINE_MISSING	An optional dependency needed to read a dataset file format (pandas for CSV; pandas + a pyarrow parquet engine for parquet) is not installed.	Install the missing extra: pip install pandas pyarrow (parquet) or pip install pandas (CSV).
DEP_GPU_NOT_AVAILABLE	CUDA / a compatible GPU could not be detected.	Confirm nvidia-smi works. Reinstall PyTorch with the CUDA wheel that matches your driver (pip install torch --index-url https://download.pytorch.org/whl/cu121 etc.).
DEP_OLLAMA_REGISTRATION_FAILED	register_with_ollama(...) could not reach the Ollama daemon or the registration call returned an error.	Start the daemon: ollama serve (or install Ollama from https://ollama.com). Default endpoint is localhost:11434. Retryable.
DEP_MLX_UNAVAILABLE	v1.5 T3.1 (experimental) — the MLX (Apple-Silicon) rail was selected (resolved backend='mlx') but the mlx_lm toolchain is not importable on this host. mlx-lm is Apple-Silicon-ONLY (macOS + arm64), so on a Windows / Linux / Intel-Mac host the [mlx] extra cannot install and this fires. Raised by MLXBackend.run() at the subprocess-launch attempt; a forced backend='mlx' on a non-Apple host is normally intercepted earlier at Trainer construction as CONFIG_INVALID_SETTING, so reaching this code usually means a corrupted mlx-lm install on a real Mac.	On an Apple-Silicon Mac: install the extra — pip install 'backpropagate[mlx]'. On a non-Apple host: use backend='auto' (the default — routes to CUDA) or backend='cuda'; backend='mlx' cannot run there. Set the rail via --backend (CLI) or BACKPROPAGATE_TRAINING__BACKEND (env).
DEP_FSDP_UNAVAILABLE	v1.7 — --full-ft-offload (FSDP2 CPU-offload full fine-tuning) was requested but the FSDP toolchain (torch.distributed FSDP / accelerate, with a usable NCCL backend) is not available. The common case is Windows-native: FSDP CUDA collectives need NCCL, which is Linux/WSL2-only (gloo can’t carry CUDA collectives). Raised by the trainer before the offload run starts — it never silently runs without offload and OOMs.	Run --full-ft-offload under WSL2 / Linux (the measured recipe — needs ~64 GB host RAM), OR drop --full-ft-offload and use a model within the pure-GPU full-FT ceiling, OR use --mode lora (QLoRA fits 7B–34B on a 32 GB card).

RUNTIME_* (internal runtime)

Code	Raised when	Fix
RUNTIME_TRAINING_FAILED	Training failed in a way that doesn’t fit a more specific code.	Re-run with --verbose (or BACKPROPAGATE_DEBUG=1) to see the unredacted traceback. The run_id printed at startup correlates this failure with every log line and checkpoint — quote it in any bug report.
RUNTIME_TRAINING_ABORTED	Training was aborted (user interrupt, GPU pause/abort, etc.). The error includes steps_completed and the last checkpoint_path so you can resume.	If the abort was from GPU temperature or VRAM pressure, fix the underlying issue and resume from the checkpoint with backprop resume <run_id> (the run_id is printed in the error).
RUNTIME_EVAL_GATE_REGRESSED	backprop eval --gate-against <baseline_run_id> determined the evaluated run regressed the held-out metric beyond the allowed --max-regression. This is the eval-gate that protects continual-merge / SLAO campaigns. Returned as exit 65 (not raised); the code is stamped into the structured log line.	The after-run is worse than the baseline by more than --max-regression. Inspect the diff (re-run with --vs <baseline> for the side-by-side). Either keep the baseline (reject this run / merge), raise --max-regression if a small regression is acceptable, or retrain with a higher learning rate / more steps / cleaner data.
RUNTIME_VRAM_CALIBRATION_FAILED	backprop estimate-vram --calibrate could not measure the model on this GPU: no CUDA GPU, too little free VRAM for even the smallest probe, the model failed to load, or too few probes fitted in memory. Nothing is stored (exit 2).	Close other GPU programs and retry, or measure a smaller model. Estimates use the built-in formula until a measurement exists.
RUNTIME_EVAL_FAILED	backprop eval failed to complete the evaluation — the model could not be loaded, the held-out forward pass crashed, or sample generation raised. Distinct from a clean regression (RUNTIME_EVAL_GATE_REGRESSED); here the eval did not finish. Surfaced via the catch-all exit-code mapper (exit 2, or 137 on CUDA OOM / 69 on a Hub failure).	Re-run with --verbose (or BACKPROPAGATE_DEBUG=1) for the full traceback. Confirm the run’s checkpoint loads via backprop show-run <run_id> and that the model fits in VRAM at eval time (lower --num-samples / --max-new-tokens if you OOM during generation).
RUNTIME_UI_AUTH_NOT_ENFORCED	backprop ui --share (or --host <non-loopback>) was invoked without --auth user:password. The v1.2.0+ contract refuses to start the UI rather than expose an unauthenticated tunnel. (The pre-v1.2.0 BACKPROPAGATE_SECURITY__REQUIRE_AUTH_FOR_SHARE=false opt-out is a no-op under the Reflex UI — held only for forward-compat with the Gradio era — and will not relax this gate.)	Pass --auth user:password (or --auth-file <path> in v1.3+ to keep the credential out of shell history). If you don’t actually need a public URL, use SSH port-forwarding instead: ssh -L 7860:localhost:7860 <host>. See handbook/security.md for the full threat model and the four supported auth modes.
RUNTIME_EXPORT_FAILED	Export failed in a way that doesn’t fit a more specific code.	Re-run with --verbose; verify the model loaded cleanly first via backprop info. If the export step fired before trainer.train() returned, check the LoRA-export-without-training warning above your stderr — save() no longer silently writes untrained weights as of v1.4.
RUNTIME_LORA_EXPORT_FAILED	LoRA adapter export failed.	Confirm the trainer actually has LoRA adapters attached — trainer.train() must have run before trainer.export(...) or trainer.save(...). As of v1.4, save() emits a WARN line if you call it before train(), naming the path it would have written; check your stderr for that line.
RUNTIME_GGUF_EXPORT_FAILED	GGUF export failed.	Install the export extra: pip install backpropagate[export]. On first run, llama-cpp-python may need to build from source — Windows needs Visual C++ Build Tools + CMake; Linux needs build-essential + cmake. See troubleshooting → llama-cpp-python build failed.
RUNTIME_MERGE_EXPORT_FAILED	Merging the LoRA back into the base model failed.	Most commonly VRAM pressure during the merge — try a smaller base model, free GPU memory first (nvidia-smi to see what’s holding it), or pass device_map="cpu" to merge on CPU at the cost of speed.
RUNTIME_GPU_ERROR	Generic GPU error that doesn’t fit a more specific GPU code.	Run backprop info to confirm CUDA / GPU / driver are wired up correctly. Then re-run training with --verbose to see which op tripped. If you see a CUBLAS error, see the CUDA troubleshooting page → CUBLAS errors.
RUNTIME_GPU_OOM	Out-of-memory on the GPU during training, eval, or export.	The B-002 OOM-recovery path (enabled by default) auto-shrinks batch size up to 3 times before giving up. If you still see this code: lower --batch-size manually, reduce the model’s max_seq_length setting (most VRAM lives in attention — pass Trainer(max_seq_length=1024) or set BACKPROPAGATE_MODEL__MAX_SEQ_LENGTH=1024), or pick a smaller model preset (Qwen 2.5 3B fits in ~8GB). To opt out of auto-recovery and let OOM bubble up: Trainer(oom_recovery=False).
RUNTIME_FULL_FT_MODEL_TOO_LARGE	Trainer(mode="full") was requested for a model whose parameter count exceeds the card-aware full-fine-tuning ceiling — derived from detected VRAM (16 GB→4B, 24 GB→5B, 32 GB→6B pure-GPU; --full-ft-offload lifts it to ≈8B / 7B-class on a 32 GB card). Fires at Trainer.__init__ (preset/model-id lookup) AND again at Trainer.load_model() (authoritative model.num_parameters() check).	The error is contrastive: (1) if the model clears the offload ceiling but not the pure-GPU one, add --full-ft-offload (Python full_ft_offload=True) to spill params+optimizer to host RAM (slower; Linux/WSL2; ~64 GB host RAM); (2) raise the ceiling explicitly with --full-ft-ceiling-billions; (3) past even the offload ceiling, use --mode lora — LoRA/QLoRA fits 7B–34B on a 32 GB card (recommended per Biderman 2024 + Thinking Machines 2025). See full fine-tuning for the VRAM math + the decision tree.
RUNTIME_FP8_UNSUPPORTED	v1.5; verified on Blackwell (RTX 5090, sm_120) in v1.6 — the FP8 compute path (--fp8 / BACKPROPAGATE_TRAINING__FP8 / Trainer(fp8=True)) was enabled but FP8 conversion / the first float8 GEMM failed at runtime — most often a half-installed or ABI-mismatched torchao whose import torchao.float8 fails AFTER the capability gate already promised FP8 (so it is NOT degraded silently to bf16). Other FP8 failure axes (no CUDA, sm < 9, conversion error) instead degrade to bf16 with a warning and do not raise this.	Re-run without --fp8 (fp8=False) to train in bf16 — correct, just without the FP8 speed/memory win. If you want FP8: reinstall a torchao matching your torch build — pip install --force-reinstall 'backpropagate[fp8]'; FP8 needs base-layer inner dims divisible by 16, a Hopper/Blackwell (sm_90+) GPU, and torch>=2.11 for the compiled torchao kernels (2.10 uses the slower _scaled_mm fallback). Note --fp8 is --mode lora + --method sft only; mode='full', any non-sft method (orpo / simpo / kto), or explicit 4-bit combined with FP8 are rejected earlier as CONFIG_INVALID_SETTING.
RUNTIME_GPU_TEMPERATURE_CRITICAL	GPU exceeded the configured safety threshold and training was paused or aborted. Retryable.	Wait for the GPU to cool (training auto-resumes once it does); if it keeps tripping, improve case airflow and / or lower batch size to reduce sustained load. See GPU safety for the temperature thresholds.
RUNTIME_GPU_MONITORING_FAILED	The GPU monitor could not query NVML.	Install pynvml: pip install pynvml. If pynvml is installed and you still see this, your driver may not expose NVML — confirm with nvidia-smi.
RUNTIME_SLAO_ERROR	Generic SLAO failure.	Re-run with --verbose to see which merge step tripped. The run_index field in the error includes the run number — pair it with the on-disk checkpoint for that run when reporting.
RUNTIME_SLAO_MERGE_FAILED	SLAO weight merge failed at a specific run.	The error includes run_index; inspect the on-disk checkpoint for that run.
SLAO_MERGE_DIVERGED	A SLAO merge produced weights with non-finite values (NaN/Inf). Defensive check raised before the bad weights enter the model.	Reduce the learning rate, shorten steps_per_run, or pick a different merge_mode.
PEFT_API_INCOMPATIBLE	The installed peft version does not expose the API Backpropagate expects.	Upgrade peft: pip install -U peft.
UI_OUTPUT_DIR_FORBIDDEN	The BACKPROPAGATE_UI__OUTPUT_DIR env var (or default) resolved to a system / credential directory like /etc, ~/.ssh, C:\Windows\System32.	Pick a non-system directory under your home (e.g. ~/.backpropagate/ui-outputs or ~/work/backprop-out).

STATE_* (corrupt state)

Code	Raised when	Fix
STATE_CHECKPOINT_INVALID	A checkpoint file could not be saved or loaded — missing manifest, corrupt safetensors, mid-write crash.	Delete the offending checkpoint directory; stray .partial directories are safe to remove. Leave <path>.backup/ and <path>.backup.json (1.8.2+): they hold the previous save, which the next save restores. See troubleshooting.
STATE_SLAO_CHECKPOINT_INVALID	A SLAO checkpoint snapshot is corrupt.	Same fix — remove the bad snapshot and re-run. SLAO will rebuild from the previous good snapshot.

PARTIAL_* (mixed success/failure)

Code	Raised when	Fix
PARTIAL_BATCH_OPERATION	A per-item loop (e.g. multi-run training, batch export) finished with some items failing. The error lists the first N failures and a success-rate percentage.	Inspect the listed errors; re-run the failed items individually.
PARTIAL_SUCCESS	The CLI ran to completion but not every unit succeeded. Maps to exit code 3.	The operation completed enough to be useful — inspect logs, decide whether to retry the failed units.

Reading codes programmatically

Every BackpropagateError exposes a structured envelope:

from backpropagate.exceptions import BackpropagateError

try:
    trainer.train("data.jsonl", steps=100)
except BackpropagateError as e:
    envelope = e.to_dict()
    # {
    #   "type": "GPUMemoryError",
    #   "code": "RUNTIME_GPU_OOM",
    #   "message": "Insufficient GPU memory: need 14.2GB, have 12.0GB",
    #   "retryable": False,
    #   "suggestion": "Try reducing batch size, ...",
    #   "details": {"required_gb": 14.2, "available_gb": 12.0},
    # }

The retryable flag tells callers whether a naive retry is safe. True for DEP_OLLAMA_REGISTRATION_FAILED, RUNTIME_GPU_TEMPERATURE_CRITICAL, DEP_MODEL_LOAD_FAILED; False for everything else by default.

See also

• Troubleshooting — symptoms-first reverse index (start here if you don’t yet know which code fired).
• Environment variables — every BACKPROPAGATE_* knob, including the security and UI sandbox knobs referenced above.