OnPremo

Fit the model to the machine — before it loads.

A small Rust control plane around llama.cpp: inspect, admit, run — within a declared memory budget. Memory-budget-aware local LLM runtime for constrained Linux devices (Raspberry Pi class). Offline. Private.

  • Admission before load
  • Defined downgrade order
  • Reject with reason codes
  • No cloud required

Install

One-liner installers download a prebuilt binary, verify SHA-256, and put onpremo on your PATH.

Linux x86_64

Desktop / server / CI hosts

curl -fsSL https://onpremo.org/install-linux-x86_64.sh | sh

Linux ARM64

Pi 3 / 4 / 5 (64-bit OS)

curl -fsSL https://onpremo.org/install-linux-arm64.sh | sh

Linux ARMv7

32-bit Pi OS

curl -fsSL https://onpremo.org/install-linux-armv7.sh | sh

macOS Apple Silicon

Development host (arm64)

curl -fsSL https://onpremo.org/install-macos-apple-silicon.sh | sh

macOS Intel

Development host (x86_64)

curl -fsSL https://onpremo.org/install-macos-intel.sh | sh

Windows (PowerShell)

Install under %USERPROFILE%\.onpremo\bin

powershell -c "irm https://onpremo.org/install-windows.ps1 | iex"

Build from source

Requires Rust, cmake, and a C++ toolchain for the llama.cpp backend.

curl -fsSL https://onpremo.org/install-from-source.sh | sh
Installers verify SHA-256 of published assets. ARM and Windows binaries appear as cross-builds are validated — see bin/FLAVORS for what this release actually ships. Checksums live in SHA256SUMS. To uninstall: curl -fsSL https://onpremo.org/uninstall.sh | sh

The memory contract

Most local LLM tools let you pick a model and hope it fits. OnPremo treats memory as a control plane: estimate first, load only if the estimate passes, and never treat the OS OOM killer as normal control flow.

Required memory is estimated as:

Required memory =
  model residency
+ KV cache
+ compute buffers
+ tokenizer / runtime overhead
+ operating-system reserve
+ safety margin

The process budget (weights + KV + compute + overhead + margin) is what admission enforces. The OS reserve is reported for device-level capacity but sits outside that process ceiling.

Illustrative onpremo check output

Shape copied from the examples walk-through — numbers depend on host, profile, and model.

decision          admitted
context           512
batch             16
kv cache          F16
max output tokens 128
threads           4
weights           107374182 (102.4 MiB)
kv cache          ...
compute           ...
runtime overhead  ...
os reserve        ...
safety margin     ...
process total     ...  (budget 629145600 (600.0 MiB))
device total      ...

Downgrade order

When the profile default does not fit, admission applies downgrades in this fixed order:

  1. Start with the requested profile
  2. Reduce prompt batch size
  3. Reduce context length
  4. Select a lower-memory KV-cache type
  5. Reduce maximum output tokens
  6. Reject the model as incompatible if it still cannot fit

Rejected with a reason code beats an OOM kill. When nothing in the profile’s configuration space fits, OnPremo refuses the run with a machine-readable reason code instead of swapping the system to death or relying on the kernel OOM killer.

Models

Estimates from OnPremo's own admission math, pending on-device certification; OnPremo never downloads models — sources and licenses are recorded in the catalogue.

Model Weights KV per token micro-500mb pi-1gb-safe Status
TinyStories-33M Q4 ~25 MiB tiny fits fits high
all-MiniLM-L6-v2 F16 ~45 MiB n/a fits fits high
SmolLM2-135M-Instruct Q4_K_M ~100 MiB 22.5 KiB fits fits high
Gemma-3-270M-IT Q4 ~230 MiB ~20 KiB rejected fits medium
SmolLM2-360M-Instruct Q4_K_M ~258 MiB 40 KiB rejected fits high
Qwen2.5-0.5B-Instruct Q4_K_M ~380 MiB 12 KiB rejected tight high
Qwen3-0.6B Q4_K_M ~378 MiB 112 KiB rejected tight medium

Catalogue manifests: models/catalogue/ in the source repo. Hugging Face sources: TinyStories-33M, all-MiniLM-L6-v2, SmolLM2-135M, Gemma-3-270M-IT, SmolLM2-360M, Qwen2.5-0.5B, Qwen3-0.6B.

Quickstart

Get a small instruct model first — SmolLM2-135M-Instruct GGUF (Hugging Face). Download a Q4_K_M (or similar) file and point the CLI at it. OnPremo does not auto-download model weights. 

# 1. Inspect metadata without loading weights
onpremo inspect smollm2-135m-instruct-q4_k_m.gguf

# 2. Test compatibility with a device profile
onpremo check smollm2-135m-instruct-q4_k_m.gguf --profile pi-1gb-safe

# 3. Run one-shot streaming generation
onpremo run smollm2-135m-instruct-q4_k_m.gguf \
  --profile pi-1gb-safe \
  --prompt "Return the intent as JSON" \
  --max-tokens 32

Exit codes for check / run: 0 admitted or downgraded, 2 rejected, 1 for bad paths or invalid profiles. Full flag reference lives on the docs page.

Status

Pre-0.1. The core path is mock-verified: inspect, admit, check, run, benchmark, and the optional local HTTP API work with the deterministic mock backend used in development and CI.

The real llama.cpp backend compiles behind --features backend-llama, but hardware certification (Raspberry Pi 3 / Pi 5) is still pending. Treat published numbers as illustrative until certified benchmark reports land.

Read the docs for CLI flags, profiles, the memory model summary, HTTP API, and build-from-source notes.