edge0-35b

The flagship first tier of the edge0 platform: a 35B-class sparse mixture-of-experts model built on Qwen3.6-35B-A3B (K=4 tier). Through streaming expert loading, a trained prerouter, and LoRA adaptation, the full quantized weight set resides on disk and is loaded on demand, so the model can be served on a single device.

The performance profile is based on benchmarks of the current release adapter version and is pinned by default in Qwen35Config (src/edge0/models/edge0_35b/__init__.py).

Model profile

Item	Value
Parameter scale	35B-class
Number of layers	40
Number of experts	256 routed experts + 1 always-resident shared expert
top_k (K)	4
Routing	SOFTMAX_TOPK (softmax → top-k → renormalize, norm_topk_prob=True)
Expert quantization	4-bit affine, group 64
Weight layout	WeightLayout.SEPARATE (gate/up/down stacked as separate tensors)
Expert weight path	language_model.model.layers.N.mlp.switch_mlp
Prerouter	33 heads (owners 6..38), start_layer 7, hidden 512, fp16
Decode invocation	patch_call=True, staged decode across tokens, K=4
LoRA	r=16, alpha=32.0
Prefill chunk	2048
Hot window	4
Streaming prefetch history	on (prefetch_history=True)
Serving port	8000
Measured throughput	14.9–17.7 tok/s (M4 Pro)
Measured peak activation memory	≈ 2.9 GiB

Note: all values are taken from Qwen35Config._defaults() and LayerOptions.staged_k4(). The head count comes from the explicit owners list (6 through 38, 33 heads in total); feature_topk="executed" means the top-k features fed to the heads are exactly the set actually routed at decode time.

Staged decode

This tier uses the LayerOptions.staged_k4() preset:

• Fixed-slot staged decode (staged=True, staged_n=4, staged_sync=True), with no per-layer host synchronization.
• staged_replace=False: routing is supplied by the trained prerouter heads (the MoE blocks route via prerouter logits), so the staged set and the routed set are exactly identical and the slot-table mapping discards nothing.
• On-demand prefill (full_layer_prefill=False): prefill and decode take the same on-demand expert loading path (aligned with the production profile's QWEN_PREFILL_FULL=0), so peak memory matches deployment;
• Always-resident hot-expert pinning is off (hot_per_layer=0, aligned with the production profile's QWEN_HOT=0).

Usage

Serving via the CLI

edge0 serve /path/to/checkpoint --host 127.0.0.1 --port 8000

Optional arguments:

• --no-prerouter: disable the prerouter (prerouter=None).
• --no-lora: disable LoRA (lora="").
• --flask: switch to the Flask transport (requires flask to be installed; supports SSE streaming).

For single-turn chat in the terminal, use chat instead:

edge0 chat --name edge0-35b --model-dir /path/to/checkpoint --prompt "Hello"

To view this tier's default profile:

edge0 models

Python API

from edge0 import AutoEngine, AutoModel, AutoConfig

# an engine ready to generate
engine = AutoEngine.from_pretrained(
    "/path/to/checkpoint", name="edge0-35b",
)
ids = engine.generate([248044])          # uses the default sampling from the config
text = engine._tok.decode(ids)
engine.close()

# weights only (streaming experts + prerouter + LoRA installed)
model = AutoModel.from_pretrained("/path/to/checkpoint", name="edge0-35b")

# config only
cfg = AutoConfig.from_pretrained("/path/to/checkpoint", name="edge0-35b")

AutoEngine / AutoModel / AutoConfig can all omit name and resolve automatically from the model_type in the checkpoint's config.json or from the directory basename (see src/edge0/registry.py).

HTTP API

edge0 serve exposes a single OpenAI-compatible model endpoint. The engine serves one exclusive request at a time; generation is serialized through a FIFO queue.

Method	Path	Description
GET	/healthz	Health check
GET	/v1/models	List loaded models
POST	/v1/chat/completions	Chat completions (supports stream)
POST	/v1/completions	Not supported; returns 400

Non-streaming chat

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "edge0-35b",
    "messages": [{"role": "user", "content": "Introduce yourself"}],
    "temperature": 0.6,
    "max_tokens": 256
  }'

Response fields (non-streaming):

{
  "id": "chatcmpl-...",
  "object": "chat.completion",
  "created": 1750000000,
  "model": "edge0-35b",
  "choices": [{
    "index": 0,
    "message": {"role": "assistant", "content": "..."},
    "finish_reason": "stop"
  }],
  "usage": {"prompt_tokens": 12, "completion_tokens": 30, "total_tokens": 42}
}

Optional request fields: model, messages (with role/content; content supports multiple text segments that are concatenated automatically), temperature, top_p, top_k, max_tokens, seed, stream.

Streaming chat (requires Flask)

curl -N http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "edge0-35b",
    "messages": [{"role": "user", "content": "Count to five"}],
    "stream": true
  }'

Each token emits one data: {"object":"chat.completion.chunk", ...} SSE event, and the stream ends with data: [DONE].

Configuration overrides

from_pretrained supports overriding any public field (unknown fields raise a TypeError).

from edge0 import AutoEngine

engine = AutoEngine.from_pretrained(
    "/path/to/checkpoint",
    name="edge0-35b",
    port=9090,                    # override the default port 8000
    target_tok_s=14.0,            # override the acceptance throughput target
    prerouter=None,               # disable the prerouter
    lora="",                      # disable LoRA
    prefill_chunk=1024,           # smaller prefill chunk
)

The corresponding CLI overrides are --no-prerouter / --no-lora (see _engine_kwargs in src/edge0/cli.py). To override engine parameters, call Qwen35Config.from_pretrained(model_dir, **overrides) directly.