🇧🇷 Português · 🇺🇸 English · 🇨🇳 中文

---

“Tensors speak Gleam. Kernels burn silicon. The BEAM holds the soul.”

viva_tensor IS NOT A WRAPPER. It is a production-grade FP8 LLM inference engine written from scratch: hand-tuned CUDA kernels, blocked W8A16 GEMV, full-token CUDA Graphs, and a public ModelHandle API — all driven from Gleam on the BEAM.

It is faster than Ollama on the same hardware.

🎯 Overview

A tensor library for Gleam on the BEAM. Provides a pure-Gleam tensor API for portability, an inference API for FP8 / INT4-2:4 / INT8-2:4 sparse linear layers, and a public LLM ModelHandle API for Llama-family HuggingFace checkpoints.

The library works fully in pure BEAM (slow but portable) and transparently upgrades to the native CUDA path when the NIF is loaded.

⚡ Quick Start

git clone https://github.com/gabrielmaialva33/viva_tensor.git && cd viva_tensor
gleam deps download

# Optional: native CUDA backend (RTX 4090 / Ada SM89)
make cutlass-libs    # CUTLASS + cuSPARSELt static archives
make zig             # the NIF shared object

gleam test           # 792 tests, all pass with NIF loaded

Generate text in 4 lines of Gleam

import viva_tensor as t

let assert Ok(model) = t.load_model("tmp/tinyllama/model.safetensors")
let assert Ok(result) = t.generate(model, "Hello", t.default_generate_opts())
result.text

🏗️ Architecture

   ┌─────────────────────────────────────────────────────────┐
   │                  Gleam application code                 │
   │       viva_tensor.load_model / .generate / .Tensor      │
   └────────────────────────┬────────────────────────────────┘
                            │
   ┌────────────────────────▼────────────────────────────────┐
   │            Erlang public API (viva_tensor_llm)          │
   │  SafeTensors loader · BPE tokenizer · sampling · KV     │
   └────────────────────────┬────────────────────────────────┘
                            │
   ┌────────────────────────▼────────────────────────────────┐
   │          NIF dispatch (viva_tensor_zig.so + .erl)       │
   │   PackedWeight · EmbeddingTable · KvCache · ModelHandle │
   └─────────┬──────────────────────────────┬────────────────┘
             │                              │
   ┌─────────▼──────────┐         ┌─────────▼────────────────┐
   │ Pure-Gleam tensors │         │      CUDA kernels        │
   │   (no GPU needed)  │         │ W8A16 GEMV · FlashAttn   │
   │                    │         │ Full-token CUDA Graph    │
   │                    │         │ CUTLASS FP8/INT4 sparse  │
   └────────────────────┘         └──────────────────────────┘

📊 Performance

All numbers measured on RTX 4090 (Ada SM89) + Intel i9-13900K (32 threads @ 5.80 GHz). Reproducible via bench/ harness.

Text generation — TinyLlama-1.1B-Chat (FP8 W8A16)

Validated models

Quantized GEMM kernels (RTX 4090)

Full methodology + raw numbers in bench/results/matmul_showdown.md.

🧬 Design Principles

🛠️ Public API

High-level: LLM inference

import viva_tensor as t

pub fn main() {
  let assert Ok(model) = t.load_model("tmp/tinyllama/model.safetensors")

  let opts =
    t.GenerateOpts(
      max_new_tokens: 50,
      temperature: 0.0,           // argmax — deterministic
      top_k: t.TopKInfinity,
      top_p: 1.0,
      seed: 42,
      stop_on_eos: True,
    )

  let assert Ok(result) = t.generate(model, "Hello", opts)
  result.text
}

Reproducible sampling

let sampling_opts =
  t.GenerateOpts(
    max_new_tokens: 30,
    temperature: 0.8,
    top_k: t.TopK(40),
    top_p: 0.95,
    seed: 42,
    stop_on_eos: True,
  )

Same seed → same token sequence across machines.

Low-level: quantized linears

let assert Ok(packed) = t.prepack_fp8_weight_blocked(weight, 16)
let assert Ok(output) = t.linear_fp8_w8a16(input, packed, bias)

Prepack once, run linear forwards many times — the FP8 weight + scales live on the device for the lifetime of the PackedWeight resource.

🗺️ Roadmap

🤝 Contributing

git checkout -b feature/your-feature
make cutlass-libs && make zig
gleam test          # 792 should pass with NIF loaded
make test-no-nif    # 791 should pass without NIF

See CONTRIBUTING.md for guidelines.

📚 Documentation

Guides

• Getting started — install, first run.
• LLM inference end-to-end — load → tokenize → decode → sample.
• FFI architecture — Gleam ↔ Erlang ↔ C/CUDA boundaries.
• Project structure — repo layout.

API reference

• LLM ModelHandle — load_model, generate, tested models.
• Inference API — prepack + linear FP8 / INT-sparse.
• Tensor API — pure-Gleam tensor surface.

Technical paper

• Honest paper — what works, what doesn’t, why.

📜 What’s new in 2.2.102

• Public LLM API. viva_tensor.load_model(path) accepts a HuggingFace Llama-family checkpoint (single file, sharded, or folder) and returns an opaque ModelHandle. viva_tensor.generate(model, prompt, opts) drives deterministic argmax or seeded temperature/top-k/top-p sampling.
• Fast FP8 W8A16 decode. Hand-tuned vt_w8a16_mmv_blocked_k16 GEMV with uint4 vectorized loads, full-token CUDA Graph capture with cudaGraphExecUpdate, persistent device-resident KV caches, and a cuBLASLt plan cache.
• Multi-model validated. TinyLlama-1.1B and Llama-3.2-1B-Instruct pass byte-identical and through the same public API.
• Sharded SafeTensors. Loads single .safetensors, HF model.safetensors.index.json, or any folder containing either.
• Byte-level BPE. GPT-2 / Llama-3 byte-encoded vocabularies decode back to readable text; SentencePiece (▁) still works as before.
• Tied embeddings. Detects tie_word_embeddings from config.json and reuses embed_tokens as lm_head when set.

Full evolution from 63 tok/s baseline to 448 tok/s across 14 rounds of optimization is documented in CHANGELOG.md.

Star if you believe BEAM can do LLM inference ⭐

Created by Gabriel Maia · MIT License