---
license: apache-2.0
base_model: distilbert/distilgpt2
library_name: peft
tags:
  - lora
  - peft
  - adapter
  - mixture-of-experts
  - model-merging
  - task-arithmetic
  - sparse-model-nexus
pipeline_tag: text-generation
---

# Sparse Model Nexus — Composable LoRA Quanta

**One compositional algebra spanning the LLM ⟷ quantum sparsity continuum.**

This repo is the *operational* form of the [Sparse Model Nexus knowledge graph](https://sparse-model-nexus.pages.dev) (Pharosiraptor graph `1c0cc1b9-d95a-4448-b7d7-a08e1e347820`). It ships three tiny **LoRA "quanta"** over a shared frozen `distilgpt2` base and a Python package implementing a four-operator algebra to compose them.

A **quantum** is a minimal, self-contained, composable model unit — a LoRA adapter — the realization of the graph's `Definition::SparseLimitModel`. Every composition of quanta is itself deployable, which is the whole point of the continuum: the same algebra spans one quantum to a dense mixture.

## The quanta (seed specialists)

| Adapter | Domain | Mirrors graph node |
|---|---|---|
| `adapters/code_specialist` | Code / algorithms / SQL | `SLM-quantum: code-specialist` |
| `adapters/biomed_specialist` | Biomedical prose | `SLM-quantum: biomed-specialist` |
| `adapters/shared_common` | General English | `SLM-quantum: shared-common` |

Each is rank-8 LoRA on `c_attn` (~150K trainable params), trained on CPU. See `adapters/train_report.json` for start/end loss.

## The operator algebra

From the graph's `Operator::` nodes, implemented in `sparse_nexus.py`:

| Op | Name | Semantics | Algebra |
|---|---|---|---|
| **⊕** | `mixture(width)` | weighted blend of several quanta (MoE-style) | assoc + commut; identity = null-expert |
| **∘** | `sequential(depth)` | chain quanta — one's output feeds the next | assoc, non-commut |
| **▷** | `route(conditional)` | gated dispatch: predicate picks a quantum | router-as-morphism |
| **⊗** | `merge(weight-space)` | task-arithmetic merge into one new quantum | partial |

All compositions respect a **maximal self-organized criticality** constraint (`Criticality::MAXIMAL`): every quantum should be load-bearing (Shapley-marginal > 0). Dead experts / routing collapse is the anti-pattern.

## Quickstart

```bash
pip install torch transformers peft
python -c "
from sparse_nexus import Nexus
nx = Nexus()
print(nx.route('def quicksort(xs):')['output'])          # ▷ -> code_specialist
print(nx.mixture('Explain:', {'code_specialist':0.5,'shared_common':0.5})['output'])  # ⊕
print(nx.merge(['code_specialist','biomed_specialist','shared_common'],
               name='omni', prompt='A concise explanation:')['output'])  # ⊗
"
```

Load a single quantum with vanilla PEFT:

```python
from transformers import AutoModelForCausalLM
from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained("distilgpt2")
model = PeftModel.from_pretrained(base, "adapters/code_specialist")
```

## Why this design speeds adoption

- **Tiny + CPU-trainable.** Each quantum is ~150K params; the whole bundle downloads in seconds.
- **Standard PEFT format.** Drops into any `transformers` + `peft` pipeline; no custom runtime.
- **Composable by construction.** Add a new domain = train one more quantum; the operators compose it with the rest for free.

## Tarski residual (honest bound)

Composite capability is not the colimit of the quanta's closures — emergence across a composition is **undecidable** in general, and weight-space factorization is non-unique. Optimality of a composition over unknowable future workloads holds only in expectation, never as a true all-futures optimum. This is inherited verbatim from the graph's `TarskiResidual::nexus` node.

## Provenance

Generated as SwarmInvocation `exp-20260705-nexus-adapter` (caller: megapraxis). Full graph, executive + operational views: https://sparse-model-nexus.pages.dev
