📉 Sequential Forgetting Leaderboard

How much does sequential fine-tuning destroy what the model already learned? Lower magnitude = better retention. Every number links to the raw run file in the benchmark dataset; transcriptions are hand-checked (provenance). Suites are not cross-comparable; each table ranks within its own protocol.

Suite A — 5 real-world domains · Mistral-7B · 3 seeds

method	base model	domains	seeds	metric	value %	status	source	notes
`modular_crma`	Mistral-7B	5	3	holdout_NLL_drift	-0.166	valid_multiseed	multiseed_results_combined.json	per-task LoRA + CRMA backbone; drift of saved snapshots re-run under final backbone
`frozen_base`	Mistral-7B	5	3	holdout_NLL_drift	1.948	valid_multiseed	multiseed_results_combined.json	no adaptation control
`naive_sequential_lora`	Mistral-7B	5	3	holdout_NLL_forgetting	42.96	valid_multiseed	multiseed_results_combined.json	single LoRA trained sequentially across domains

Suite B — Medical→Legal→Code→Finance

method	base model	domains	seeds	metric	value %	status	source	notes
`modular_crma`	Mistral-7B-v0.3	4	1	holdout_NLL_drift	-0.1	valid_single_run	ablation_v8.1_7b_results.md	avg of per-task drift table; NAIVE ref is the same run's forgetting avg
`modular_crma`	TinyLlama-1.1B-Chat-v1.0	4	1	holdout_NLL_drift	-0.1	valid_single_run	ablation_v8.1_results.md
`naive_sequential_lora`	TinyLlama-1.1B-Chat-v1.0	4	1	holdout_NLL_forgetting	225.3	valid_single_run	ablation_v8.1_results.md
`naive_sequential_lora`	Mistral-7B-v0.3	4	1	holdout_NLL_forgetting	351.4	valid_single_run	ablation_v8.1_7b_results.md

Suite B history — CL-technique stacks (within-version comparisons only)

method	base model	domains	seeds	metric	value %	status	source	notes
`cl_stack_v5_10component`	TinyLlama-1.1B	4	1	holdout_NLL_forgetting	58.4	valid_single_run	full_ablation_history_v2_v8.md	10-component stack; post-data-fix suite - NAIVE dropped 185.8->88.8 from data fixes alone
`cl_stack_v3_ewc_gradproj`	TinyLlama-1.1B	4	1	holdout_NLL_forgetting	91.3	valid_single_run	full_ablation_history_v2_v8.md	EWC + gradient projection; pre-data-fix suite - compare only to its own NAIVE column
`cl_stack_v7_replay_kd_freeze`	Mistral-7B	4	1	holdout_NLL_forgetting	109.3	valid_single_run	full_ablation_history_v2_v8.md	replay + knowledge distillation + bottom-layer freeze, Mistral-7B

Suite C — MQuAKE 5-skill retention · Qwen3-4B

method	base model	domains	seeds	metric	value %	status	source	notes
`modular_vault_slots`	Qwen3-4B-Instruct-2507	5	1	accuracy_BWT	0.0	valid_single_run	bwt_mquake_vault5_s42.json	retention matrix R; BWT_k = R_final,k - R_k,k; mean over 4 earlier skills

Invalid & incomplete runs (disclosed)

Buggy or unfinished runs are relabeled, not deleted. Highlight: our early O-LoRA arm appeared to win (−2.0% forgetting) until we found a gradient-clipping bug that had frozen the model — so O-LoRA is listed as invalid, never validly measured here, not as beaten.

method	base model	domains	seeds	metric	value %	status	source	notes
`cl_stack_v2_olora_ewc_gradproj_replay`	TinyLlama-1.1B	2	1	holdout_NLL_forgetting	-2.0	invalid_disclosed	full_ablation_history_v2_v8.md	INVALID: PiSSA-init O-LoRA grad norms ~126678 vs clip 100 (ratio 0.00079) froze the model; the -2.0% 'win' is an artifact. O-LoRA has never been validly measured here
`cl_stack_v4_cumulbasis`	TinyLlama-1.1B	2	1	holdout_NLL_forgetting	27.8	incomplete_disclosed	full_ablation_history_v2_v8.md	INCOMPLETE: run cut off mid-Phase-3; Phase-2-only numbers
`cl_stack_v6_sma`	TinyLlama-1.1B	2	1	holdout_NLL_forgetting	61.5	incomplete_disclosed	full_ablation_history_v2_v8.md	INCOMPLETE: Sparse Memory Adapter crashed (OOM) at Phase 3; Phase-2-only numbers

Submit your method

Run your method on a suite (protocol).
Score it with scoring/score.py (--nll or --matrix).
Open a PR on the dataset repo adding your raw log, a results.csv row, and a provenance line.

Single-seed submissions are accepted and labeled valid_single_run. If your run later turns out buggy, it moves to the disclosed section — that's the deal for everyone, including us.

Maintained by ModelBrew — fine-tuning without catastrophic forgetting (patent-pending CRMA adapters). The modular_crma rows are our method; independent replications welcome.