Developer Reference

AlphaLlama — complete CLI and API reference.

Quick Start

# Run a model as an OpenAI-compatible server
alphallama -m model.gguf

# Download and run from HuggingFace
alphallama -hf Qwen/Qwen3-35B:Q4_K_M

# Specify GPU layers and context size
alphallama -m model.gguf -ngl auto -c 8192

# Serve with API key protection
alphallama -m model.gguf --api-key mysecretkey

Context Modes

AlphaLlama supports three context modes via the --context flag:

alphallama -m model.gguf --context native      # default
alphallama -m model.gguf --context extended     # for large knowledge bases
alphallama -m model.gguf --context sequential   # for very long documents

Native Mode

Default. Full prompt with native attention. Best for chat, coding, reasoning. Up to 262K tokens with Q4 KV cache on 24GB VRAM.

Extended Mode

For large knowledge bases. Index a corpus, then query with server-side retrieval + inference. Scales to 82B+ tokens with 92-99% accuracy. Client never touches raw chunk text.

# Step 1: Index chunks
curl http://localhost:18080/api/corpus/index -d '{
  "chunk_id": 1, "text": "Your document text..."
}'

# Step 2: Query — returns chunk IDs (not text)
curl http://localhost:18080/api/corpus/query -d '{
  "query": "search terms"
}'

# Step 3: Chat with extended context (server retrieves chunks internally)
curl http://localhost:18080/v1/chat/completions -d '{
  "messages": [{"role": "user", "content": "your question"}],
  "context_mode": "extended"
}'

Sequential Mode

For very long documents. Processes in overlapping windows. Tested up to 1.3M chars with 90.7% accuracy.

Choosing the Right Mode

Use Case	Mode
Chat, coding, reasoning	native
Search a knowledge base	extended
Query across millions of docs	extended
Long document (exceeds context)	sequential
Reproduce exact text from 1M+ doc	sequential

Model Loading

Flag	Description
-m, --model FNAME	Path to GGUF model file
-hf USER/MODEL[:QUANT]	Download and run from HuggingFace
-hff FILE	Specific file to download from HF repo
-hft TOKEN	HuggingFace access token for gated models
--lora FNAME	Apply LoRA adapter
--lora-scaled FNAME:SCALE	Apply LoRA with custom scale
-mm, --mmproj FILE	Multimodal projector file (vision models)

alphallama -m ~/models/qwen3-35b.gguf
alphallama -hf Qwen/Qwen3-35B-MoE:Q4_K_M
alphallama -m model.gguf --lora adapter.gguf
alphallama -m model.gguf -mm projector.gguf

GPU & Memory Management

Flag	Description
-ngl N	Layers to offload to GPU (auto, all, or number)
--fit [on\|off]	Auto-adjust parameters to fit in VRAM (default: on)
--fit-target MiB	Target VRAM margin per device
-c, --ctx-size N	Context size in tokens (default: 4096)
--cache-type-k TYPE	KV cache type for keys (f16, q8_0, q4_0)
--cache-type-v TYPE	KV cache type for values
-cmoe, --cpu-moe	Offload MoE expert weights to CPU
-ts, --tensor-split N,N,...	Split model across multiple GPUs
--numa distribute	NUMA-aware memory allocation

# Auto-detect best GPU offload
alphallama -m model.gguf -ngl auto

# Q4 KV cache (fits 262K context on 24GB)
alphallama -m model.gguf -ngl all --cache-type-k q4_0 --cache-type-v q4_0 -c 262144

# MoE offload (keep attention on GPU, experts on CPU)
alphallama -m model.gguf -ngl all --cpu-moe

# Multi-GPU split
alphallama -m model.gguf -ngl all -ts 60,40

Server Options

Flag	Description
--host HOST	Listen address (default: 127.0.0.1)
--port PORT	Listen port (default: 18080)
-np, --parallel N	Number of parallel request slots
--ssl-key-file FILE	TLS private key for HTTPS
--ssl-cert-file FILE	TLS certificate for HTTPS
--no-webui	Disable the built-in web UI
--metrics	Enable Prometheus metrics at /metrics
--no-license	Skip license check (cloud/RunPod deployment)

# Production server: bind all interfaces, 4 parallel slots
alphallama -m model.gguf --host 0.0.0.0 --port 18080 -ngl all --parallel 4

# HTTPS with SSL
alphallama -m model.gguf --ssl-key-file key.pem --ssl-cert-file cert.pem

# RunPod / cloud deployment (no license needed)
alphallama -m model.gguf --host 0.0.0.0 --port 18080 -ngl 99 --no-license --parallel 4

API Authentication

Two levels of API key protection:

Flag	Description
--api-key KEY	Server-side API key (local protection)
--api-key-file FILE	Load API keys from file (one per line)
ALPHACHAT_API_KEY	AlphaChat platform key (tier-based, see Pricing)

# Server-side key
alphallama -m model.gguf --api-key mysecret
curl -H "Authorization: Bearer mysecret" http://localhost:18080/v1/chat/completions

# Platform key (Business tier)
export ALPHACHAT_API_KEY=sk-alphachat-biz-abc123
alphallama -m model.gguf

Sampling Parameters

Flag	Description
--temp N	Temperature (default: 0.8, 0 = greedy)
--top-k N	Top-K sampling (default: 40, 0 = disabled)
--top-p N	Top-P / nucleus sampling (default: 0.95)
--min-p N	Min-P sampling (default: 0.05)
--repeat-penalty N	Repeat penalty (default: 1.0)
--dry-multiplier N	DRY repetition penalty multiplier
--seed N	Random seed (-1 = random)

Reasoning / Thinking

Enable chain-of-thought reasoning for supported models:

Flag	Description
--reasoning	Enable reasoning mode
--reasoning-budget N	Max tokens for thinking (0 = unlimited)
--reasoning-format FORMAT	Output format: deepseek \| none

alphallama -m model.gguf --reasoning --reasoning-budget 4096

Speculative Decoding

Speed up inference by drafting tokens with a smaller model:

Flag	Description
-md, --model-draft FNAME	Draft model for speculative decoding
-nd, --draft N	Number of draft tokens (default: 16)
--spec-layer-skip N	Self-speculative layer-skip decoding

# Draft model speculation
alphallama -m big-model.gguf -md small-model.gguf -nd 16

# Self-speculative (no draft model needed)
alphallama -m model.gguf --spec-layer-skip 4

API Endpoints

OpenAI-Compatible

Method	Endpoint	Description
POST	/v1/chat/completions	Chat completions (streaming supported)
POST	/v1/completions	Text completions
POST	/v1/embeddings	Generate embeddings
GET	/v1/models	List loaded models

AlphaLlama-Specific (Extended Context)

Method	Endpoint	Description
POST	/api/corpus/index	Index a chunk into the corpus
POST	/api/corpus/query	Query the corpus — returns chunk IDs only (no text)
DELETE	/api/corpus/clear	Clear all indexed chunks
GET	/api/corpus/stats	Corpus statistics (chunk count, size)

Ollama-Compatible

Also supports Ollama API endpoints:

/api/chat · /api/generate · /api/tags · /api/show · /api/ps

Environment Variables

Flag	Description
ALPHACHAT_API_KEY	Platform API key (sk-alphachat-* / sk-alphachat-biz-* / sk-alphachat-org-*)
CUDA_VISIBLE_DEVICES	Select specific GPU(s)
GGML_CUDA_NO_PINNED	Disable pinned memory (saves RAM)
LLAMA_CACHE	Model cache directory
LLAMA_ARG_*	Set any CLI flag via env var (e.g., LLAMA_ARG_N_GPU_LAYERS=99)

Common Configurations

16GB GPU (RTX 4080)

alphallama -m qwen3-35b-q4.gguf -ngl auto --cpu-moe -c 8192 --parallel 2

24GB GPU (RTX 3090) — Maximum Context

alphallama -m qwen3-35b-q4.gguf -ngl all --cache-type-k q4_0 --cache-type-v q4_0 -c 262144

Knowledge Base Server

alphallama -m qwen3-35b-q4.gguf -ngl all --context extended \
  --host 0.0.0.0 --port 18080 --parallel 4

Low-VRAM (8GB laptop)

alphallama -m qwen3-4b-q4.gguf -ngl auto --fit on -c 4096

Cloud / RunPod Deployment

alphallama -m model.gguf --host 0.0.0.0 --port 18080 \
  -ngl 99 --no-license --ctx-size 32768 --parallel 4