No description
  • Shell 64.4%
  • Python 31.2%
  • Dockerfile 4.4%
Find a file
2026-11-22 00:34:33 +01:00
.agents/skills docs: n-gram speculation numbers 2026-11-22 00:34:33 +01:00
patches patches: fuse PAD into small-K mul_mat; fuse MUL_MAT_ID scale on src1 alias 2026-11-22 00:17:50 +01:00
.env.example up.sh: wait for ready, smoke-test, restart cleanly, never prompt 2026-11-21 22:55:14 +01:00
.gitignore gitignore: models, TLS material, local settings, .agents/ except skills 2026-11-21 22:23:27 +01:00
AGENTS.md docs: setup runbook for agents; scan covers merged PRs and forks 2026-11-21 22:55:14 +01:00
build.sh build.sh: clone, pin e50c254a2, apply patches/, build the image 2026-11-21 22:23:27 +01:00
Caddyfile compose: podman and docker variants, settings from .env 2026-11-21 22:23:27 +01:00
check.sh check.sh: preflight for a fresh host 2026-11-21 22:55:14 +01:00
compose.docker.yaml compose: n-gram speculative decoding (ngram-mod) 2026-11-22 00:34:33 +01:00
compose.yaml compose: n-gram speculative decoding (ngram-mod) 2026-11-22 00:34:33 +01:00
Containerfile build.sh: clone, pin e50c254a2, apply patches/, build the image 2026-11-21 22:23:27 +01:00
down.sh up/down: engine-agnostic start and stop 2026-11-21 22:23:27 +01:00
download.sh download: verify SHA-256, write the merged attn_qrg model 2026-11-21 22:23:27 +01:00
engine.sh up.sh: wait for ready, smoke-test, restart cleanly, never prompt 2026-11-21 22:55:14 +01:00
gen-tls.sh up/down: engine-agnostic start and stop 2026-11-21 22:23:27 +01:00
merge-qrg.py download: verify SHA-256, write the merged attn_qrg model 2026-11-21 22:23:27 +01:00
models.sha256 download: verify SHA-256, write the merged attn_qrg model 2026-11-21 22:23:27 +01:00
README.md docs: n-gram speculation numbers 2026-11-22 00:34:33 +01:00
up.sh up.sh: wait for ready, smoke-test, restart cleanly, never prompt 2026-11-21 22:55:14 +01:00

Inkling 2 Mini on an RX 7900 XTX

Serves Inkling 2 Mini (26.4 B parameters, a looped mixture of experts (MoE)) with a patched llama.cpp on an AMD RX 7900 XTX through ROCm. It exposes an OpenAI-compatible API on 127.0.0.1:8000.

One request decodes at 60–63 tokens/s on prose and 70–110 on code edits (n-gram drafting, output unchanged), prefill runs at 847 tokens/s; 4 slots share a 256 K-token KV pool in 23 GiB of VRAM and reach 145 tokens/s together.

The image is llama.cpp master e50c254a2 plus patches/: Inkling 2 loop and DSpark support, and fused kernels for the model's small "glue" operations. The model file is rewritten so that three attention projections run as one matrix product (merge-qrg.py). The rationale for each patch is in its commit message.

Use

Requirements: Linux, an RX 7900 XTX with /dev/kfd and /dev/dri accessible to your user, podman with podman-compose or docker with compose, about 60 GiB of disk and 32 GiB of RAM (the server maps 16 GiB; each build job takes 1–2 GiB, JOBS caps them).

./check.sh             # preflight: GPU, device access, engine, disk, RAM
./build.sh             # pinned llama.cpp + patches/ -> image
./download.sh          # model + vision projector, SHA-256 check, merged copy
./up.sh                # waits until ready; creates .env with a random key; TLS=1 adds HTTPS on :8443
./down.sh

engine.sh picks podman (compose.yaml, for SELinux) or docker (compose.docker.yaml, untested: it should work; if it does not, hand it to your agent). Set ENGINE=docker to override. compose.override.yaml is merged in if present. .env sets the bind address, the ports, the key and the image. Agents: see AGENTS.md for timings and troubleshooting.

Built with Claude Code (Anthropic).