Hanzo Engine - Native Rust Inference & Embedding Engine

Hanzo Engine is Hanzo AI's high-performance, Rust-native inference and embedding engine that powers the entire Hanzo ecosystem.

Skill Level: Intermediate to Advanced Prerequisites: Rust knowledge helpful but not required for API usage

Overview

Hanzo Engine is Hanzo AI's high-performance, Rust-native inference and embedding engine that powers the entire Hanzo ecosystem. Built on mistral.rs with Hanzo-specific optimizations, it provides blazing-fast local inference for all ZenLM models and industry-standard LLMs.

Core Philosophy: Maximum performance, native Rust implementation, multimodal support, and seamless integration with Hanzo Node and Cloud.

Key Features

🚀 Blazingly Fast Performance

Native Rust: Zero-overhead implementation
PagedAttention: Memory-efficient attention mechanism
FlashAttention: 2-8x faster attention computation
ISQ (In-Situ Quantization): On-the-fly model quantization
MLX: Optimized for Apple Silicon (M1/M2/M3)

🔮 All-in-One Multimodal

Text ↔ Text: Standard LLM inference
Text + Vision ↔ Text: Vision-language models
Text + Vision + Audio ↔ Text: Multimodal understanding
Text → Speech: Text-to-speech generation
Text → Image: Image generation (coming soon)

🎯 Embeddings First-Class

Optimized for Qwen3-Embedding: #1 on MTEB multilingual benchmark
Multiple dimensions: 1024, 2048, 4096 dims
Reranking support: Qwen3-Reranker models
Production-ready: High-throughput batch processing

🌐 Multiple APIs

Rust API: Native high-performance integration
Python API: PyO3 bindings (via mistralrs-pyo3)
OpenAI HTTP: Compatible with OpenAI Chat Completions & Embeddings API
MCP Support: Model Context Protocol for agentic workflows

🔗 Embedded Everywhere

Hanzo Node: Local inference and mining
Cloud Nodes: Distributed inference clusters
Hanzo Desktop: Native macOS app with mining
Python/Go SDKs: Connect via OpenAI-compatible API

Architecture

Hanzo Engine in the Stack

┌─────────────────────────────────────────────────┐
│           ZenLM Models (zen-nano, zen-eco)      │
│  zen-agent, zen-musician, zen-thinking, etc.    │
└─────────────────────────────────────────────────┘
                    ↓
┌─────────────────────────────────────────────────┐
│         Hanzo Engine (Rust - port 36900)        │
│  ├─ Native inference & embeddings               │
│  ├─ Optimized for Qwen3 models (#1 MTEB)       │
│  ├─ Multimodal: text, vision, audio            │
│  ├─ PagedAttention, FlashAttention, ISQ, MLX   │
│  └─ MCP support built-in                        │
└─────────────────────────────────────────────────┘
                    ↓
┌─────────────────────────────────────────────────┐
│           Embedded in:                          │
│  ├─ Hanzo Node (local inference)                │
│  └─ Cloud Nodes (distributed inference)         │
└─────────────────────────────────────────────────┘
                    ↓
┌─────────────────────────────────────────────────┐
│           SDKs connect to Node/Cloud:           │
│  ├─ Python SDK (hanzoai package)                │
│  ├─ Go SDK (github.com/hanzoai/go-sdk)          │
│  └─ JavaScript/TypeScript SDK (@hanzo/sdk)      │
└─────────────────────────────────────────────────┘

Based on mistral.rs

Hanzo Engine extends mistral.rs with:

Custom CLI tool (hanzo-engine command)
Model management (pull, list, cache)
OpenAI-compatible server (default port: 36900)
Hanzo-specific optimizations for ZenLM models
Integration with Hanzo Node and Cloud infrastructure

Upstream Sync: Fully synchronized with mistral.rs (commit 530463af1)

Installation

From Source (Recommended)

# Clone Hanzo Engine
git clone https://github.com/hanzoai/engine.git
cd engine

# Build for macOS (Metal backend)
cargo build --package hanzo-engine --release --no-default-features --features metal

# Build for Linux (CUDA backend)
cargo build --package hanzo-engine --release --features cuda

# Install binary
cargo install --path hanzo-engine --no-default-features --features metal

Via Cargo

# Install from GitHub
cargo install --git https://github.com/hanzoai/engine hanzo-engine

# With CUDA support (Linux)
cargo install --git https://github.com/hanzoai/engine hanzo-engine --features cuda

Verify Installation

hanzo-engine --version
hanzo-engine --help

Quick Start

Pull a Model

# Pull ZenLM model (zen-eco-4B)
hanzo-engine pull qwen/qwen3-4b

# Pull embedding model (Qwen3-Embedding-8B)
hanzo-engine pull qwen/qwen3-embedding-8b

# Pull from Ollama
hanzo-engine pull ollama://llama3.2:3b

# Pull GGUF model from URL
hanzo-engine pull https://huggingface.co/TheBloke/Llama-2-7B-GGUF/resolve/main/llama-2-7b.Q4_K_M.gguf

Start the Server

# Start on default port (36900)
hanzo-engine serve

# With specific port
hanzo-engine serve --port 36900

# With specific model
hanzo-engine serve --model qwen3-4b

# With custom model directory
hanzo-engine serve --model-dir ~/.hanzo/models

Test Inference

# Chat completions (OpenAI-compatible)
curl -X POST http://localhost:36900/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3-4b",
    "messages": [{"role": "user", "content": "Explain Rust ownership"}],
    "temperature": 0.7
  }'

# Embeddings
curl -X POST http://localhost:36900/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3-embedding-8b",
    "input": "Hello, Hanzo Engine!"
  }'

ZenLM Native Support

All ZenLM models are natively supported in Hanzo Engine:

zen-nano (0.6B)

hanzo-engine pull qwen/qwen3-0.6b
hanzo-engine serve --model qwen3-0.6b

zen-eco (4B) - instruct/thinking/agent variants

# zen-eco-instruct
hanzo-engine pull qwen/qwen3-4b

# zen-eco-thinking (with chain-of-thought)
hanzo-engine pull qwen/qwen3-4b-cot

# zen-agent (tool calling & function execution)
hanzo-engine pull qwen/qwen3-4b-function-calling

zen-musician (7B) - Music generation with lyrics

hanzo-engine pull map/yue-s1-7b-anneal-en-cot

zen-vision (Vision-language)

# Qwen3-VL for vision understanding
hanzo-engine pull qwen/qwen3-vl-8b

Performance: All ZenLM models run at 50+ tokens/sec on consumer hardware (M1, RTX 3060).

Integration with Hanzo Node

Hanzo Engine is the core inference backend for Hanzo Node:

Configuration

# In Hanzo Node configuration (.env or environment)
export EMBEDDINGS_SERVER_URL="http://localhost:36900"  # Engine port
export EMBEDDING_MODEL_TYPE="qwen3-embedding-8b"
export USE_NATIVE_EMBEDDINGS="true"
export HANZO_ENGINE_ENABLED="true"

Automatic Connection

When you start Hanzo Node, it automatically connects to Hanzo Engine:

# Start Hanzo Engine
hanzo-engine serve --port 36900

# Start Hanzo Node (connects to Engine on port 36900)
hanzo-node start --mine --gpu

Hanzo Node will:

Use Engine for all inference requests
Generate embeddings via Engine's Qwen3-Embedding models
Leverage Engine's multimodal capabilities
Benefit from PagedAttention and FlashAttention optimizations

Python SDK Integration

from hanzo import Hanzo

# Connect to local Hanzo Engine (via Hanzo Node or directly)
hanzo = Hanzo(
    inference_mode='local',
    node_url='http://localhost:8080'  # Hanzo Node (which uses Engine)
)

# Or connect directly to Engine
hanzo_engine = Hanzo(
    base_url='http://localhost:36900',
    api_key='not-needed-for-local'
)

# Chat completion (uses zen-eco via Engine)
response = hanzo_engine.chat.completions.create(
    model='qwen3-4b',
    messages=[{'role': 'user', 'content': 'Explain Rust'}]
)

# Embeddings (uses Qwen3-Embedding via Engine)
embedding = hanzo_engine.embeddings.create(
    model='qwen3-embedding-8b',
    input='Hello, Hanzo Engine!'
)

print(f"Embedding dimensions: {len(embedding.data[0].embedding)}")  # 4096

Go SDK Integration

package main

import (
    "context"
    "fmt"
    "github.com/hanzoai/go-sdk"
    "github.com/hanzoai/go-sdk/option"
)

func main() {
    // Connect to Hanzo Engine via Hanzo Node
    client := hanzoai.NewClient(
        option.WithInferenceMode("local"),
        option.WithNodeURL("http://localhost:8080"),
    )

    // Chat completion
    response, err := client.Chat.Completions.Create(
        context.Background(),
        hanzoai.ChatCompletionCreateParams{
            Model: hanzoai.F("qwen3-4b"),
            Messages: hanzoai.F([]hanzoai.ChatCompletionMessageParam{
                hanzoai.UserMessage("Explain Rust ownership"),
            }),
        },
    )

    if err != nil {
        panic(err)
    }

    fmt.Println(response.Choices[0].Message.Content)
}

Rust Native API

For maximum performance, use the native Rust API:

use mistralrs::{
    ChatCompletionRequest, IsqType, Loader, MistralRs, ModelDType,
    NormalLoaderBuilder, NormalRequest, Request, RequestMessage, Response,
    SamplingParams, SchedulerConfig, TokenSource
};
use tokio::sync::mpsc::channel;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Load ZenLM model (zen-eco-4B)
    let loader = NormalLoaderBuilder::new(
        "qwen/qwen3-4b",
        None,
        None,
        Some(ModelDType::Auto),
    )
    .build();

    let model = loader.load_model().await?;
    let pipeline = model.build_pipeline(SchedulerConfig::default_config())?;

    let mistralrs = MistralRs::new(pipeline)?;

    // Create request
    let (tx, mut rx) = channel(10_000);
    let request = Request::Normal(NormalRequest {
        messages: RequestMessage::Chat(vec![
            ChatCompletionRequest {
                role: "user".to_string(),
                content: "Explain Rust ownership".to_string(),
            }
        ]),
        sampling_params: SamplingParams::default(),
        response: tx,
        ..Default::default()
    });

    // Send request
    mistralrs.get_sender().send(request).await?;

    // Receive response
    let response = rx.recv().await.unwrap();
    println!("{}", response.choices[0].text);

    Ok(())
}

Model Management

Pull Models

# From HuggingFace
hanzo-engine pull qwen/qwen3-4b

# From Ollama
hanzo-engine pull ollama://llama3.2:3b

# From MLX Community
hanzo-engine pull mlx://mlx-community/Llama-3.2-3B-Instruct-4bit

# From direct URL
hanzo-engine pull https://example.com/model.gguf

List Downloaded Models

hanzo-engine list

# Output:
# Downloaded models:
# - qwen3-4b (4.3 GB) - /Users/z/.hanzo/models/qwen3-4b
# - qwen3-embedding-8b (8.1 GB) - /Users/z/.hanzo/models/qwen3-embedding-8b
# - llama3.2-3b (3.2 GB) - /Users/z/.hanzo/models/llama3.2-3b

Model Storage

Default model directory: ~/.hanzo/models/

Custom directory:

export HANZO_MODELS_DIR=/path/to/models
hanzo-engine serve --model-dir /path/to/models

Performance Optimizations

PagedAttention

Memory-efficient attention with dynamic memory allocation:

# Enable PagedAttention (default for long contexts)
hanzo-engine serve --paged-attention

# Adjust memory usage
hanzo-engine serve --gpu-memory-fraction 0.9

Benefits:

2-4x higher throughput
Support for longer sequences
Efficient KV cache management

FlashAttention

Ultra-fast attention computation:

# FlashAttention is automatic with CUDA
cargo build --release --features "cuda flash-attn"

# Verify FlashAttention is active
hanzo-engine serve --log-level debug
# Look for: "Using FlashAttention for faster inference"

Benefits:

2-8x faster attention
10-20% less memory usage
No accuracy degradation

In-Situ Quantization (ISQ)

On-the-fly model quantization for lower memory:

# Quantize to 4-bit on load
hanzo-engine serve --isq Q4K

# Quantize to 8-bit
hanzo-engine serve --isq Q8_0

# Available formats: Q4K, Q4_0, Q5K, Q8_0, Q8_1

Benefits:

50-75% memory reduction
Minimal accuracy loss (<2% on benchmarks)
Faster loading times

MLX (Apple Silicon)

Optimized for M1/M2/M3 Macs:

# Build with Metal backend (macOS)
cargo build --package hanzo-engine --release --no-default-features --features metal

# Use MLX models for maximum performance
hanzo-engine pull mlx://mlx-community/Llama-3.2-3B-Instruct-4bit
hanzo-engine serve --model llama-3.2-3b-instruct-4bit

Benefits on M1 Max:

50+ tokens/sec for 4B models
30+ tokens/sec for 7B models
Unified memory = faster transfers

MCP Support

Hanzo Engine has built-in MCP support for agentic workflows:

# Start Engine with MCP enabled
hanzo-engine serve --mcp-enabled --mcp-port 3691

# MCP tools automatically exposed:
# - hanzo_infer: Run inference on any model
# - hanzo_embed: Generate embeddings
# - hanzo_list_models: List available models

MCP Integration Example

import { MCPClient } from '@hanzo/mcp'

const client = new MCPClient({
  host: 'localhost',
  port: 3691
})

// Use Hanzo Engine tools via MCP
const result = await client.callTool('hanzo_infer', {
  model: 'qwen3-4b',
  prompt: 'Explain MCP',
  temperature: 0.7
})

console.log(result.response)

Supported Models

Embedding Models (Optimized)

Qwen3-Embedding-8B: 4096 dims, #1 on MTEB multilingual
Qwen3-Embedding-4B: 2048 dims, balanced performance
Qwen3-Embedding-0.6B: 1024 dims, lightweight

Reranker Models

Qwen3-Reranker-4B: Superior reranking quality
Qwen3-Reranker-0.6B: Lightweight reranking

LLM Models

All models from mistral.rs:

Llama: 2, 3, 3.1, 3.2, 3.3
Mistral: v0.1, v0.2, v0.3, Nemo, Large
Phi: Phi-2, Phi-3, Phi-3.5
Gemma: Gemma, Gemma 2
Qwen: Qwen2, Qwen2.5, Qwen3 (ZenLM)
DeepSeek: DeepSeek-V2, DeepSeek-V3
Yi: Yi-1.5, Yi-Coder
And many more...

Vision-Language Models

Qwen3-VL: Vision understanding (ZenLM)
Llama-3.2-Vision: Vision + language
Pixtral: Mistral's vision model
Idefics2/3: Open vision-language models

Hardware Requirements

Minimum (zen-nano 0.6B)

CPU: Any modern CPU (x86_64, ARM64)
RAM: 2 GB
GPU: Optional (runs on CPU)

Recommended (zen-eco 4B)

CPU: 4+ cores
RAM: 8 GB
GPU: 8 GB VRAM (RTX 3060, M1 8GB)

Optimal (zen-musician 7B)

CPU: 8+ cores
RAM: 16 GB
GPU: 16 GB VRAM (RTX 3090, M1 Max 24GB)

Performance Benchmarks

Model	Hardware	Tokens/sec	Latency
zen-nano 0.6B	M1 8GB	100+	10ms
zen-eco 4B	M1 16GB	50+	20ms
zen-eco 4B	RTX 3060 12GB	45+	22ms
zen-musician 7B	M1 Max 32GB	35+	28ms
zen-musician 7B	RTX 3090 24GB	60+	16ms

Troubleshooting

Engine won't start

# Check port availability
lsof -i :36900

# Use different port
hanzo-engine serve --port 37000

# Check logs
hanzo-engine serve --log-level debug

Model not found

# Verify model is downloaded
hanzo-engine list

# Re-pull model
hanzo-engine pull qwen/qwen3-4b --force

# Check model directory
ls ~/.hanzo/models/

Out of memory

# Use quantized model
hanzo-engine serve --model qwen3-4b --isq Q4K

# Reduce GPU memory usage
hanzo-engine serve --gpu-memory-fraction 0.8

# Use smaller model
hanzo-engine pull qwen/qwen3-0.6b

Slow inference

# Enable FlashAttention (CUDA)
cargo build --release --features "cuda flash-attn"

# Enable PagedAttention
hanzo-engine serve --paged-attention

# Use quantized model for faster loading
hanzo-engine pull qwen/qwen3-4b-q4k

Hanzo Engine vs Alternatives

Feature	Hanzo Engine	llama.cpp	vLLM	Ollama
Language	Rust	C++	Python	Go
Performance	⭐⭐⭐⭐⭐	⭐⭐⭐⭐	⭐⭐⭐⭐⭐	⭐⭐⭐⭐
Multimodal	✅ All	✅ Limited	✅ Yes	✅ Yes
Embeddings	✅ Optimized	❌ No	✅ Yes	✅ Yes
MCP Support	✅ Native	❌ No	❌ No	❌ No
PagedAttention	✅ Yes	❌ No	✅ Yes	❌ No
FlashAttention	✅ Yes	✅ Limited	✅ Yes	❌ No
Apple Silicon	✅ MLX	✅ Metal	❌ No	✅ Metal
Model Management	✅ Built-in	Manual	Manual	✅ Built-in
OpenAI API	✅ Yes	✅ Yes	✅ Yes	✅ Yes
Hanzo Integration	✅ Native	❌ No	❌ No	❌ No

Why Hanzo Engine?

Native Hanzo integration: Seamless with Node, Cloud, SDKs
ZenLM optimized: Best performance for Zoo Labs models
Embeddings first-class: Qwen3-Embedding #1 on MTEB
MCP native: Built-in agentic workflow support
Rust performance: Maximum speed with memory safety
All-in-one: Text, vision, audio, embeddings, reranking

hanzo-node.md: Hanzo Node embeds Engine for local inference
python-sdk.md: Python SDK connects to Engine via Node
go-sdk.md: Go SDK connects to Engine via Node
zenlm.md: ZenLM models run natively in Engine
hanzo-mcp.md: MCP integration patterns with Engine

Additional Resources

GitHub: https://github.com/hanzoai/engine
Documentation: https://docs.hanzo.ai/engine
Hanzo AI: https://hanzo.ai
mistral.rs: https://github.com/EricLBuehler/mistral.rs (upstream)

Remember: Hanzo Engine is the native Rust inference and embedding engine powering the entire Hanzo ecosystem. All ZenLM models run natively with optimal performance through Engine's advanced features like PagedAttention, FlashAttention, and ISQ.

Hanzo Engine - Native Rust Inference & Embedding Engine

On this page