- Shell 64.4%
- Python 31.2%
- Dockerfile 4.4%
| .agents/skills | ||
| patches | ||
| .env.example | ||
| .gitignore | ||
| AGENTS.md | ||
| build.sh | ||
| Caddyfile | ||
| check.sh | ||
| compose.docker.yaml | ||
| compose.yaml | ||
| Containerfile | ||
| down.sh | ||
| download.sh | ||
| engine.sh | ||
| gen-tls.sh | ||
| merge-qrg.py | ||
| models.sha256 | ||
| README.md | ||
| up.sh | ||
Inkling 2 Mini on an RX 7900 XTX
Serves Inkling 2 Mini (26.4 B parameters, a looped mixture of experts (MoE)) with a patched llama.cpp on an AMD RX 7900 XTX through ROCm. It exposes an OpenAI-compatible API on 127.0.0.1:8000.
One request decodes at 60–63 tokens/s on prose and 70–110 on code edits (n-gram drafting, output unchanged), prefill runs at 847 tokens/s; 4 slots share a 256 K-token KV pool in 23 GiB of VRAM and reach 145 tokens/s together.
The image is llama.cpp master e50c254a2 plus patches/: Inkling 2 loop and DSpark support, and fused kernels for the model's small "glue" operations. The model file is rewritten so that three attention projections run as one matrix product (merge-qrg.py). The rationale for each patch is in its commit message.
Use
Requirements: Linux, an RX 7900 XTX with /dev/kfd and /dev/dri accessible to your user, podman with podman-compose or docker with compose, about 60 GiB of disk and 32 GiB of RAM (the server maps 16 GiB; each build job takes 1–2 GiB, JOBS caps them).
./check.sh # preflight: GPU, device access, engine, disk, RAM
./build.sh # pinned llama.cpp + patches/ -> image
./download.sh # model + vision projector, SHA-256 check, merged copy
./up.sh # waits until ready; creates .env with a random key; TLS=1 adds HTTPS on :8443
./down.sh
engine.sh picks podman (compose.yaml, for SELinux) or docker (compose.docker.yaml, untested: it should work; if it does not, hand it to your agent). Set ENGINE=docker to override. compose.override.yaml is merged in if present. .env sets the bind address, the ports, the key and the image. Agents: see AGENTS.md for timings and troubleshooting.
Built with Claude Code (Anthropic).