<SYSTEM>This is the full developer documentation for tensorless</SYSTEM>

# Page not found

> This page is not here. It may have moved when the docs were reorganized.


# Start here

> What tensorless is, the one command it runs, what you need, and where each part of these docs takes you.

tensorless runs a model that was tuned for your GPU, on the engine it was tuned with, in one command. The first card is the NVIDIA Tesla P100 (16 GB), a server card that sells used for about $100. On it, tensorless runs a 27-billion-parameter model at 30 tokens a second.

```sh
curl -fsSL https://tensorless.dev/install.sh | sh
tensorless run iamMrDuncan/Qwen3.8-27B-PN3-GGUF
```

The launcher checks your card, picks the recipe for that model on that card, and pulls the engine image and the model file. It checks both by hash, then starts an OpenAI-compatible server on `127.0.0.1:8080`. Every number on these pages was measured, and each one says how.

## What you need

[Section titled “What you need”](#what-you-need)

* An x86-64 Linux machine (the engine image is linux/amd64) with an NVIDIA Tesla P100 16 GB, free for this model alone.
* An x86-64 CPU with AVX2, FMA and F16C.
* An NVIDIA driver from R525 to R580. R580 is the last branch that supports the P100.
* podman or docker, with NVIDIA container support.
* About 13 GB of disk space for the model, plus a few GB for the engine image.

`tensorless doctor` checks the card, the driver, the container runtime, NVIDIA container support and disk space, and names the fix for anything missing. It does not check the CPU: `grep -o -w -E 'avx2|fma|f16c' /proc/cpuinfo | sort -u` should print all three.

## Find your way

[Section titled “Find your way”](#find-your-way)

The docs follow [Diátaxis](https://diataxis.fr/): each page has one job.

[Tutorials**Learn by doing**Run your first model on a P100, then measure your card against the receipt.](/docs/tutorials/first-model/)[How-to guides**Get one task done**Install, prepare a host, run in the background, connect a client, benchmark, fix a failed start.](/docs/how-to/setup/install-launcher/)[Reference**Look it up**Every command and flag, the recipe file, the engine image and its switches, the PN format, the receipts.](/docs/reference/cli/)[Explanation**Understand why**How the pieces fit, why the P100, where the engine came from, and why you can trust its numbers.](/docs/explanation/how-tensorless-works/)

## What it runs today

[Section titled “What it runs today”](#what-it-runs-today)

| Launch command                                                   | Decode on one P100 |
| ---------------------------------------------------------------- | -----------------: |
| `tensorless run iamMrDuncan/Qwen3.8-27B-PN3-GGUF`                |    **30.03 tok/s** |
| `tensorless run iamMrDuncan/Qwen3.8-27B-UD-IQ3_XXS-Q3Floor-GGUF` |        26.18 tok/s |

Both were measured on the published engine image with `tensorless bench`: the median of 3 runs of 256 greedy tokens after a 429-token prompt. The [launch targets](/docs/reference/launch-targets/) page has the details, and the [receipts](/docs/reference/receipts/) have every measurement.

## Get help

[Section titled “Get help”](#get-help)

Ask on [Discord](https://discord.gg/3Qs2uejUf9), or open an [issue on GitHub](https://github.com/iammrduncan/tensorless/issues). Results from your own P100 are welcome in both places.

# How tensorless works

> What happens between typing tensorless run and a server answering, and why each piece is pinned.

This page explains what `tensorless run` does, piece by piece, and why it is built that way. You don’t need it to run a model; the [first tutorial](/docs/tutorials/first-model/) is enough for that. Read it when you want to know what you are trusting.

## Four pieces

[Section titled “Four pieces”](#four-pieces)

tensorless is four things that fit together:

* **The engine:** the inference code, a fork of llama.cpp with kernels written for the Tesla P100. Its source is in the repository’s [`engine/`](https://github.com/iammrduncan/tensorless/tree/main/engine) folder.
* **The image:** one container image per engine release and GPU architecture, such as `ghcr.io/iammrduncan/tensorless-engine:0.1.0-sm60`. It holds the built `llama-server` and nothing else that matters to speed.
* **The recipe:** one model on one card, in a small TOML file. It names the exact image (by digest), the exact model file (by revision and sha256), the settings that were measured, and what those settings measured.
* **The launcher:** the `tensorless` command. It reads the recipe and makes it happen on your machine.

The launcher holds no inference code. Everything numerical happens inside the image, the same bytes that were measured.

## What one command does

[Section titled “What one command does”](#what-one-command-does)

```sh
tensorless run iamMrDuncan/Qwen3.8-27B-PN3-GGUF
```

```
flowchart TD
    A["tensorless run ORG/REPO"] --> B["Detect the GPU<br/>(nvidia-smi: compute capability, memory, driver)"]
    B --> C["Find the recipe<br/>recipes/ORG/REPO/CARD.toml"]
    C --> D{"Released, published,<br/>and fits this card?"}
    D -- no --> X["Refuse, and say why"]
    D -- yes --> E["Pull the engine image<br/>by digest"]
    D -- yes --> F["Download the model<br/>at its pinned revision"]
    F --> G["Check the file's sha256"]
    E --> H["Start the container<br/>podman or docker, GPU passed through"]
    G --> H
    H --> I["llama-server with the recipe's<br/>switches and flags"]
    I --> J["Wait for /health"]
    J --> K["OpenAI-compatible API<br/>on 127.0.0.1:8080"]
```

1. **It finds the card.** The launcher asks `nvidia-smi` for each GPU’s compute capability, memory and driver version.
2. **It finds the recipe.** The model argument is a Hugging Face repo id. The launcher looks for that repository’s recipe for the detected card, such as `recipes/iamMrDuncan/Qwen3.8-27B-PN3-GGUF/p100.toml`. The recipes are built into the launcher, so a new model arrives with a new launcher release.
3. **It checks the recipe can run here.** The card’s compute capability must equal the recipe’s, its memory must be at least the recipe’s minimum, and the driver must be at least `min_driver`. A draft recipe, or one with placeholder values, is refused.
4. **It pulls the image by digest.** Not by tag: a tag can be moved, a digest cannot.
5. **It downloads the model** into the Hugging Face cache, at the commit the recipe pins, and checks its sha256. Other Hugging Face tools can read the same file.
6. **It starts the container** with the model mounted read-only, the recipe’s engine switches as environment variables, and the recipe’s `llama-server` flags.
7. **It waits for the server** to report healthy, then prints the address. From there it is a normal OpenAI-compatible server: `/v1/chat/completions`, `/v1/completions`, `/v1/models`.

The [command-line reference](/docs/reference/cli/) has each command, and `tensorless info` shows every one of these values for a model before anything is downloaded.

## Three rules behind it

[Section titled “Three rules behind it”](#three-rules-behind-it)

### Pin everything

[Section titled “Pin everything”](#pin-everything)

A recipe’s numbers are only true for the exact bytes they were measured on. So the recipe names the image by digest and the model by commit and sha256. If you change any of them, you are no longer running what was measured. That is also why recipes are reviewed like code: a change to a recipe’s engine or run settings needs a new measurement.

### Refuse rather than guess

[Section titled “Refuse rather than guess”](#refuse-rather-than-guess)

The launcher would rather stop with a clear reason than run something that might be slow, wrong or broken:

* no recipe for this card: it stops, instead of trying the nearest one;
* a hash that does not match: it stops, instead of loading the file;
* a driver too old for the image: it stops, with the version you need;
* a host where some GPUs don’t fit the recipe: it gives the container only the matched card, so the server cannot land on a card the image was not built for.

`tensorless doctor` runs the same checks up front and prints the command that fixes each one.

### Label what changed

[Section titled “Label what changed”](#label-what-changed)

Another port or address changes nothing that was measured. A shorter context or an extra server flag is allowed, but the launcher tells you the run is **untuned**, because the receipt’s numbers no longer apply. [Tuned and untuned runs](/docs/explanation/tuned-and-untuned/) explains where that line is.

## Why containers

[Section titled “Why containers”](#why-containers)

The engine needs a specific CUDA runtime (12.x for the P100, because CUDA 13 dropped Pascal) and a specific build of its libraries. Shipping that as a container image means:

* **the host only needs the NVIDIA driver,** not a CUDA toolkit;
* **what runs on your machine is byte for byte what was measured,** because the image is pinned by digest;
* **nothing is built on your machine,** so the result does not depend on your compiler or your CPU.

The cost is that you need podman or docker with NVIDIA container support. [Prepare a host](/docs/how-to/setup/prepare-host/) covers that.

## Why one image per architecture, not per model

[Section titled “Why one image per architecture, not per model”](#why-one-image-per-architecture-not-per-model)

One engine build serves every model on a card. What differs between models is configuration: which kernels to switch on, the context length, the KV-cache type, batch sizes. That is data, so it lives in the recipe. A fix to the engine is then one rebuild, not one per model.

Each GPU architecture gets its own image because the kernels are built and measured for one architecture. The P100 image holds `sm_60` code and nothing else. A single image for several cards would not be the build that was measured. [How a release is made](/docs/explanation/releases-and-receipts/) goes further.

# Models built on someone else's file

> Why the Q3Floor model is hosted whole under tensorless's name, how it is named, and how its original publisher is credited.

Some tuned models are a small change to a file someone else published. Qwen3.8-27B-UD-IQ3_XXS-Q3Floor is one: most of its bytes are Unsloth’s. This page explains how such a model is built, why tensorless hosts the whole file instead of patching the original on your machine, and how the name and the credit work.

## What Q3Floor is

[Section titled “What Q3Floor is”](#what-q3floor-is)

Unsloth publishes a ladder of quantizations of Qwen3.8-27B. Their **UD-IQ3_XXS** file (“UD” is Unsloth Dynamic) keeps most tensors at about 3 bits, but stores 91 of them below 3 bits, in types such as IQ2_S, IQ2_XXS and IQ1_S. That saves space. On the P100 it costs speed, because the lowest-bit types are the most expensive to unpack on `sm_60` ([Why the P100](/docs/explanation/why-the-p100/)).

Qwen3.8-27B-UD-IQ3_XXS-Q3Floor puts a **3-bit floor** under that file:

| Part                               | In Unsloth’s UD-IQ3_XXS                    | In Q3Floor                                                                                             |
| ---------------------------------- | ------------------------------------------ | ------------------------------------------------------------------------------------------------------ |
| 91 tensors below 3 bits            | IQ2_S, IQ2_XXS, IQ2_XS, Q2_K, IQ1_S, IQ1_M | IQ3_XXS, re-quantized with stock `llama-quantize` from Unsloth’s BF16 file and their importance matrix |
| The token embedding                | Q2_K                                       | Q3_K, copied byte for byte from Unsloth’s UD-IQ4_XS                                                    |
| Everything else, metadata included |                                            | unchanged, byte for byte                                                                               |

About 73% of the file’s bytes are Unsloth’s, untouched. Even the re-quantized tensors are made from their BF16 conversion with their importance matrix. It uses only standard llama.cpp types, so it also loads in stock llama.cpp.

On the P100 the floor made the model both faster and better: about 4.5% faster at 32K and 65K context than the original file on the same engine, and lower perplexity on a short check. The [model card](https://github.com/iammrduncan/tensorless/blob/main/docs/model-cards/Qwen3.8-27B-UD-IQ3_XXS-Q3Floor-GGUF.md) has the numbers.

## Why host the whole file

[Section titled “Why host the whole file”](#why-host-the-whole-file)

Another way would be to point the recipe at Unsloth’s repository and apply the change on your machine, at download time. tensorless does not do that, for three reasons.

* **The receipts measured one exact file.** Its sha256 is `4b4d6a39…`. A remix on your machine would have to produce those exact bytes every time, with your copy of `llama-quantize`, from Unsloth’s files at the same revision. Any difference, and the numbers no longer apply. Hosting the measured file means the launcher can check the hash and know.
* **Pinning works the same way for every model.** A recipe names a repository, a revision, a file and a sha256. A remix step would be a second kind of recipe, with its own failure modes.
* **It is less to download and less to run.** One 11 GB file, checked once. No BF16 source file, no quantization run.

So the file lives on Hugging Face beside Qwen3.8-27B-PN3, as `iamMrDuncan/Qwen3.8-27B-UD-IQ3_XXS-Q3Floor-GGUF`: renamed, and nothing else.

## How it is named

[Section titled “How it is named”](#how-it-is-named)

The name says what the file is:

* **It keeps the original label.** “UD-IQ3_XXS” is Unsloth’s own name for the quantization, so their users recognize what it is built on.
* **It adds what changed.** “Q3Floor”: no tensor below 3 bits.

The file’s internal metadata is still Unsloth’s, `general.name` included. Changing it would change the sha256 the receipts were measured on, so the file was renamed, not rewritten.

## How the original publisher is credited

[Section titled “How the original publisher is credited”](#how-the-original-publisher-is-credited)

A model built on someone else’s file says so where people look first:

* **the model card** names Unsloth, links their repository at the exact revision used (`4ca72078…`), lists every change, and carries their license (Apache-2.0, from Qwen and Unsloth);
* **the recipe** says the same in its comments, and names the license;
* **the [credits](/docs/project/credits/)** say that the Q3Floor model is mostly Unsloth’s work.

This is the rule for any future model of this kind: host the measured bytes whole, keep the original publisher’s name in the model’s name, and credit them by revision.

# Why a P100-native weight format

> The design choices behind PN, the weight format of Qwen3.8-27B-PN3, and the quality it trades for speed.

Qwen3.8-27B-PN3 stores most of its weights in **PN**, a weight format made for the Tesla P100. This page explains why a new format was worth making, what each of its design choices buys, and what it costs in quality. The exact layout, bit by bit, is in the [PN format reference](/docs/reference/pn-format/).

## The problem PN solves

[Section titled “The problem PN solves”](#the-problem-pn-solves)

On the P100, the cost of a decode GEMV is set by how much work it takes to unpack each weight, not only by how many bits each weight has. [Why the P100](/docs/explanation/why-the-p100/) explains why: `sm_60` lacks the integer dot-product instruction that llama.cpp’s low-bit kernels rely on. The existing 3-bit formats were designed for other hardware. Their blocks hold scales and codes in layouts that cost the P100 extra instructions to take apart.

PN starts from the other end: what is the cheapest thing for an `sm_60` lane to decode? The answer turned into these choices.

## The choices

[Section titled “The choices”](#the-choices)

### Every block is 34 bytes

[Section titled “Every block is 34 bytes”](#every-block-is-34-bytes)

Both PN types use the same block: a 2-byte scale and eight 32-bit words of codes.

| Type   | Weights per block | Bits per weight |     Codes per word |
| ------ | ----------------: | --------------: | -----------------: |
| `pn3t` |                80 |            3.40 | 10 three-bit codes |
| `pn4t` |                64 |            4.25 |   8 four-bit codes |

One block size for both types means the kernels share their addressing. A row must hold a whole number of blocks: 5120, the input width of most of Qwen3.8-27B’s weight matrices, is exactly 64 `pn3t` blocks. 6144 and 17408 are not multiples of 80, so those tensors use `pn4t`, whose 64 divides all three.

### One lane, one word

[Section titled “One lane, one word”](#one-lane-one-word)

In the decode GEMV, each GPU lane owns one packed 32-bit word, ten `pn3t` codes or eight `pn4t` codes, and the activations they multiply. The word is loaded once and decoded in registers. There is no per-weight division to find a block, and no scale stored per small group to look up.

### Codes never straddle the two halves of a word

[Section titled “Codes never straddle the two halves of a word”](#codes-never-straddle-the-two-halves-of-a-word)

Codes are packed in pairs: even codes fill the low 16 bits of a word, odd codes the high 16 bits, at the same offsets. No code crosses the middle.

That matters because of alignment. A block is 34 bytes, so a 32-bit word inside it is 4-byte aligned only in every other block. Each 16-bit half, though, is always 2-byte aligned. Because no code crosses the middle, a kernel can load the halves separately and decode each code from its own half, safely, in every block.

### Exact decode in fp32

[Section titled “Exact decode in fp32”](#exact-decode-in-fp32)

A weight is `w = d * (n - z)`: the block’s scale times the code minus a fixed zero point. The scale is an fp16 number (11 significant bits) and `n - z` is a small integer, so the product always fits exactly in an fp32 number. There is no rounding in decode.

That makes PN easy to check. The CPU reference decoder and every GPU kernel produce bit-identical weights, and the format’s golden tests compare them byte for byte. During the engine’s hill climb, any kernel change that altered a single decoded weight could be caught.

### Zero is a code, and the scale has a sign

[Section titled “Zero is a code, and the scale has a sign”](#zero-is-a-code-and-the-scale-has-a-sign)

The codes are **mid-tread**: one code means exactly zero, so small weights that round to zero stay zero. That leaves an uneven range, for example −4 to +3 steps for `pn3t`. The scale is **signed**, so the quantizer can point the extra step at whichever side of the block has the larger weight. The largest weight in each block gets the full range.

## How Qwen3.8-27B-PN3 uses it

[Section titled “How Qwen3.8-27B-PN3 uses it”](#how-qwen38-27b-pn3-uses-it)

A PN model is still a GGUF file. PN types sit next to standard ggml types, and each tensor gets the type that suits it. Qwen3.8-27B-PN3 uses the mix called **P0k+19**:

* `pn3t` on every decode GEMV tensor 5120 wide, except `attn_k` and `attn_v`;
* `pn4t` on `attn_k`, `attn_v` and the tensors 6144 and 17408 wide;
* 19 more tensors promoted from `pn3t` to `pn4t`;
* standard types for the rest: the token embedding Q3_K, the output head Q4_K, `ssm_alpha` and `ssm_beta` Q8_0, and the norms F32.

The model also applies a **Hadamard rotation** to the activations that reach the PN weights, and the engine performs it at run time. A rotation spreads a few very large values across many, so a block’s single scale fits its weights better. This follows the line of work in QuaRot and SpinQuant, and the weights were quantized with a GPTQ-style method; the [credits](/docs/project/credits/) list the papers.

The result is a 12.1 GiB file that decodes at about 30 tokens per second on one P100, against 26.3 for the 11.0 GiB Qwen3.8-27B-UD-IQ3_XXS-Q3Floor on the same card and engine.

## What it costs

[Section titled “What it costs”](#what-it-costs)

PN gives up some quality for its speed, and the receipts say how much. Before the decode hill climb started, Qwen3.8-27B-PN3 was checked against Q3Floor on one P100, with these bars:

| Check                                | Qwen3.8-27B-PN3 | Q3Floor | Bar                                          |
| ------------------------------------ | --------------: | ------: | -------------------------------------------- |
| Perplexity, context 2,048 (6 chunks) |          2.5494 |  2.5682 | at most 1.005 × Q3Floor: passes (0.993×)     |
| Decode-path perplexity, code at 32K  |         1.38147 | 1.37773 | at most 1.005 × Q3Floor: passes (1.003×)     |
| Decode-path perplexity, prose at 32K |         6.07793 | 5.65357 | at most 1.005 × Q3Floor: **misses (1.075×)** |
| Needle recall, up to 31K and 63K     |          8 of 8 |   exact | exact                                        |
| Task screen (26 cases)               |              23 |      23 | at most 1 fewer than Q3Floor                 |

On code, short text, long-context recall and the task screen, PN3 matches Q3Floor. On long prose it is measurably worse: 7.5% higher perplexity.

That miss was accepted, for a reason the measurements give. The format’s own fp32 reference path scores 6.07572 on the same check, so the fast kernels account for only 0.5% of the gap. The rest belongs to the quantization itself, three bits per weight on most tensors. The hill climb then held every later step to the starting point’s outputs, within tight bars, so the speed-up from 7.74 to 30 tokens per second did not trade away more quality.

If long prose quality matters more to you than speed, Q3Floor is the safer choice on this card today. The [launch targets](/docs/reference/launch-targets/) list both.

## Why the format is specific to this engine

[Section titled “Why the format is specific to this engine”](#why-the-format-is-specific-to-this-engine)

PN’s GGUF type ids, 124 and 125, are only known to the tensorless engine’s fork of ggml. Stock llama.cpp cannot load a PN file, and because the ids are not reserved upstream, another build could one day misread it. So PN files need the tensorless engine, and the format is written down in full, with a reference decoder and test vectors, for anyone who wants to implement it elsewhere.

# How a release is made, and why you can trust its numbers

> How an engine release goes from research commit to a pinned image, and what a receipt does and does not promise.

Every speed tensorless publishes comes with a receipt: what was measured, on what, by whom. This page explains how an engine release is made, why the image you pull is byte for byte the one that was measured, and how to read a receipt’s scope.

## One lineage, many images

[Section titled “One lineage, many images”](#one-lineage-many-images)

There is one engine, and it moves forward in releases: engine-v0.1.0, then the next. Each release is built once for each GPU architecture it supports, and each build is one container image:

```text
ghcr.io/iammrduncan/tensorless-engine:0.1.0-sm60
                                      └─┬─┘ └┬─┘
                              engine release  GPU architecture
```

Recipes never point at that tag. They point at the image’s **digest**, the sha256 of its manifest, which no one can move. A tag is a name for people; the digest is what runs.

Older images stay in the registry. A model whose recipe has not moved to a new release keeps running on the release it was measured with. So a new release never silently changes a model you already use: its recipe has to move, and a recipe moves only with a new receipt.

## From champion to release

[Section titled “From champion to release”](#from-champion-to-release)

```
flowchart TD
    A["Research campaigns<br/>each with its own metric and guards"] --> B["Champions merged<br/>into the next engine"]
    B --> C["Release candidate image<br/>built once by CI: 0.1.0-rc1-sm60"]
    C --> D["Release gate<br/>measures that exact digest"]
    D -- passes --> E["Same digest tagged<br/>0.1.0-sm60, no rebuild"]
    E --> F["Receipts written,<br/>recipes moved"]
    D -- fails --> G["Recipe stays on<br/>its previous release"]
```

This is the process for later releases. engine-v0.1.0 is a single climb’s champion, exported as is, and its published image went through steps 3 to 6.

1. **Campaigns find improvements.** Each hill-climb campaign starts from the latest engine release and keeps its best change, its **champion** ([Where the engine comes from](/docs/explanation/the-engine/)).
2. **Champions are merged** into the next engine, with a check that each one’s changes are really there. A champion only had to beat its own campaign’s previous best, so a change that was lost along the way is re-applied and measured again.
3. **CI builds a release candidate once.** A GitHub workflow builds the engine image from the repository’s Containerfile and publishes it as, for example, `0.1.0-rc1-sm60`. Its digest is printed for the gate.
4. **The release gate measures that digest** on a real card, for every recipe that will move to it.
5. **The measured digest becomes the release.** A second workflow copies the image to its release tag, `0.1.0-sm60`, without rebuilding it, then checks that the tag points at exactly the measured digest. If the digests differ, it fails.
6. **The receipts are written and the recipes move,** in the same change.

Step 5 is the important one. A rebuild, even from the same source, can produce a different binary. Promoting the measured digest instead means the release is not “built the same way as what was measured”: it *is* what was measured.

For engine-v0.1.0, `0.1.0-rc1-sm60` and `0.1.0-sm60` are the same image, `sha256:35409ff97c2f518e468790c28bd19206ae777a945fcffe5775585e28fd9939a7`.

## What the gate checks

[Section titled “What the gate checks”](#what-the-gate-checks)

The design asks the release gate for these, on each card, for each recipe:

* the build’s own checks (the P100 image runs the PN format’s golden tests while it builds);
* each model’s numerics against the previous release, within that model’s recorded bars;
* speed no worse than any merged champion, within measured noise;
* decode at long contexts (32K and 65K on the P100) and prefill;
* the model’s quality checks, once per release.

engine-v0.1.0 was the first release, so there was no previous release to compare against. Its numerics and quality were held by the hill climb that produced it. The published image was then measured on a P100 for both recipes. Long-context decode on this engine has not been measured yet, and the receipts say so. [Known limits](/docs/project/known-limits/) keeps the list.

If a change helps one model and fails another’s check, the design’s answer is not to fork the engine. The failing model’s recipe stays on the previous release, and the release notes record why.

## What “measured” means

[Section titled “What “measured” means”](#what-measured-means)

Two kinds of measurement appear in the receipts, and both time decode the same way: from outside the engine, by stamping each streamed token as it arrives.

* **The research judge** built the engine from source with the image’s CUDA configuration and timed it on one P100: 256 greedy tokens after a cold prompt of about 600 tokens, the median of confirmation blocks. This is where a recipe’s `decode_tps` comes from.
* **`tensorless bench`** measures the published image the same way: a warm-up, then greedy runs with a fixed seed and no prompt cache, decode speed as tokens over the time between the first and last streamed token. Its prompt is different (the start of the Apache-2.0 license text), so expect a few percent of difference from the research number.

For engine-v0.1.0, the published image measured within half a percent of the research numbers: 30.03 against 30.06 tok/s for Qwen3.8-27B-PN3, and 26.18 against 26.26 for Qwen3.8-27B-UD-IQ3_XXS-Q3Floor.

Timing between streamed tokens, rather than trusting the server’s own timer, means the number is what a client sees. It also means a slow network adds to the time to the first token but not to the decode rate. You can run `tensorless bench` on your own card and compare: [Benchmark a running server](/docs/how-to/use/benchmark/).

## How to read a receipt

[Section titled “How to read a receipt”](#how-to-read-a-receipt)

A number on its own says little. A receipt gives every number its **scope**:

* **which build:** a source build of a research commit, or the published image by digest;
* **which model file:** by sha256;
* **which card:** a Tesla P100-PCIE-16GB, its reported compute capability and memory;
* **what work:** how many tokens, after how long a prompt, code or prose, greedy or not, how many runs;
* **what was not measured.**

Read the last item as carefully as the others. “30 tok/s” is true at short context. It is not a claim about 65K context, about batched requests, or about a run with your own flags. The [receipts reference](/docs/reference/receipts/) has engine-v0.1.0’s in full.

# Where the engine comes from

> The engine's lineage from llama.cpp, what the tensorless fork adds, and the automated hill climb that made it fast.

The tensorless engine is a fork of llama.cpp, several forks deep, with kernels written for the Tesla P100. This page explains where it came from, what it adds, and how its PN kernels were tuned by an automated hill climb that took one model from 7.74 to about 30 tokens per second.

## The lineage

[Section titled “The lineage”](#the-lineage)

```
flowchart LR
    A["llama.cpp / ggml<br/>ggml-org"] --> B["llama-cpp-turboquant<br/>TheTom"]
    B --> C["llamAmpere<br/>JakeATX, 2cb16936"]
    C --> D["tensorless engine<br/>engine-v0.1.0"]
```

* **[llama.cpp](https://github.com/ggml-org/llama.cpp)** is the foundation: the GGUF format, the quantization types, `llama-server` and the CUDA backend.
* **[llama-cpp-turboquant](https://github.com/TheTom/llama-cpp-turboquant)** added TurboQuant KV-cache types and native multi-token prediction.
* **[llamAmpere](https://github.com/JakeATX/llamAmpere)** added an EXL3 path and Qwen3.8 work. The tensorless engine starts from its commit `2cb16936`.

All three are MIT-licensed, and their notices travel with the engine. tensorless’s own changes are Apache-2.0. The [credits](/docs/project/credits/) name everyone else whose work shaped it.

## What the fork adds

[Section titled “What the fork adds”](#what-the-fork-adds)

Compared with its upstream base, the engine at engine-v0.1.0 changes 55 files. The main additions, listed in [`engine/PROVENANCE.md`](https://github.com/iammrduncan/tensorless/blob/main/engine/PROVENANCE.md):

* **P100 decode kernels.** Split-KV flash-decoding attention over a q4_0 KV cache, quantized GEMVs for `sm_60` (including an IQ3_XXS kernel that decodes through an fp16 table), Q3_K, Q4_K and Q6_K kernels, an output-head kernel, prefill dequantization and chunked GEMM, and fused norms.
* **The PN weight types,** `pn3t` and `pn4t`, with a CPU reference decoder, CUDA dequantization and `sm_60` GEMV kernels. See [Why a P100-native weight format](/docs/explanation/pn-format/).
* **Tools for checking it:** a teacher-forced logits tool and golden tests for the PN format.

The P100 kernels for standard weight types are opt-in. An environment variable such as `GGML_CUDA_FATTN_SM60=1` switches one on, and the recipe sets exactly the switches that were measured. The PN kernels run whenever a tensor is `pn3t` or `pn4t`, and a few exact changes, such as the IQ2 shared-memory codebook, are on by default on sm_60. The [engine switches reference](/docs/reference/engine-switches/) lists them.

## Tuned by an automated hill climb

[Section titled “Tuned by an automated hill climb”](#tuned-by-an-automated-hill-climb)

Most of the engine’s speed on Qwen3.8-27B-PN3 came from a **decode hill climb**: a long run in which AI agents proposed changes to the engine, and a judge measured each one and kept it only if it was faster and still correct.

The climb ran on [Thermite](https://usethermite.com), an open-source autoresearch harness ([source](https://github.com/iammrduncan/thermite)). Thermite’s central idea is that the agents do not decide what counts as better:

* **Agents propose.** Each attempt is a change to the engine’s code, proposed by an AI model.
* **A judge measures.** The judge builds the change, runs it on a real P100 and times decode from outside the engine. The agents cannot see the judge’s held-out checks or change how it measures.
* **Guards hold the numerics.** An attempt that is faster but changes the model’s output beyond its bars is rejected, however fast it is.
* **Only confirmed wins are kept.** A faster attempt is measured again before it becomes the new best, the **champion**. Each new champion only has to beat the one before it.

The numbers for the climb that produced engine-v0.1.0:

|                                                           |                     |
| --------------------------------------------------------- | ------------------: |
| Judged attempts                                           |                 131 |
| Confirmed improvements                                    |                  28 |
| Qwen3.8-27B-PN3 decode at the start (code / prose prompt) |   7.74 / 7.73 tok/s |
| Qwen3.8-27B-PN3 decode at the end (code / prose prompt)   | 30.06 / 30.18 tok/s |
| Speed-up                                                  |               3.89× |

Short context means 256 greedy tokens after a cold prompt of about 600 tokens, on one P100.

### What the guards held

[Section titled “What the guards held”](#what-the-guards-held)

Speed alone is easy to fake: skip work, lose precision, return the wrong token faster. So every step of the climb was held to these bars, against the climb’s starting point:

* **Qwen3.8-27B-PN3’s outputs:** KL divergence within 2.593e-3 over 128 forced prose tokens, the same top token at 95% or more of positions, and decode-path perplexity within 0.25%.
* **Qwen3.8-27B-UD-IQ3_XXS-Q3Floor’s text:** bit-identical greedy output, every step. This model shares most of the engine’s code paths, so a change that broke it was caught even when it was aimed at PN.
* **The PN format itself:** the golden test blocks, and one real tensor of each PN class, decoded bit-exactly on CPU and GPU.

The [receipts](/docs/reference/receipts/) have the full measurements.

### What a climb does not guarantee

[Section titled “What a climb does not guarantee”](#what-a-climb-does-not-guarantee)

A hill climb finds changes that help the case it measures. engine-v0.1.0’s climb measured decode at short context. It did not re-measure 32K and 65K contexts, where attention costs more of each token, so those numbers are still open ([Known limits](/docs/project/known-limits/)). The release gate is meant to measure long contexts too; for engine-v0.1.0 that measurement is still outstanding. See [How a release is made](/docs/explanation/releases-and-receipts/).

## Earlier work

[Section titled “Earlier work”](#earlier-work)

The PN kernels came from the Thermite climb. Before that, earlier P100 kernel campaigns, run with [pi-autoresearch](https://github.com/davebcn87/pi-autoresearch), built kernels for the standard llama.cpp types, and Qwen3.8-27B-UD-IQ3_XXS-Q3Floor was the model they were first tuned on. That is why the Q3Floor model is also a launch target, and why it served as the comparison for every PN measurement.

## How the source gets here

[Section titled “How the source gets here”](#how-the-source-gets-here)

The engine was developed in a separate research repository. At each release, its `engine/` folder is exported into tensorless at the exact commit that was measured. The export:

* leaves out research-only files (the harness’s state and notes);
* rewrites a handful of comment lines that pointed into the research repository, one line for one line, so line numbers and compiled code do not change;
* refuses to finish if it finds a private name or path anywhere it should not be;
* writes `engine/PROVENANCE.md`: the upstream base, the research commit, a digest of the whole tree, and what was changed.

So the source in `engine/` is the code that was measured, and `PROVENANCE.md` says how to check that.

# Tuned and untuned runs

> What it means for a run to match its recipe, what makes a run untuned, and why the launcher labels it instead of forbidding it.

A recipe’s numbers are true for one exact configuration. This page explains what that configuration is, which changes take a run outside it, and why tensorless lets you make those changes but tells you when you have.

## What “tuned” means

[Section titled “What “tuned” means”](#what-tuned-means)

Each recipe has a `[run]` section: the context length, every `llama-server` flag, and the engine switches, exactly as the release gate measured them. For Qwen3.8-27B-PN3 on the P100 that is a 65,536-token context, a q4_0 KV cache, flash attention, batches of 512, and 21 `GGML_CUDA_*` switches that turn on the P100 kernels.

A **tuned** run is that configuration, on the image the recipe pins, with the model file whose sha256 the recipe names. The receipt’s numbers apply to it. `tensorless run MODEL`, with no extra settings, is always a tuned run.

## What makes a run untuned

[Section titled “What makes a run untuned”](#what-makes-a-run-untuned)

Three things change the measured configuration:

| You pass                                           | Why it matters                                                                                                     |
| -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| `--context N`, with N other than the recipe’s      | The KV cache changes size, and with it the memory left on the card and the speed of attention.                     |
| Extra `llama-server` arguments after `--`          | Any flag can change what the server does: sampling, batching, offload, cache types.                                |
| `MODEL@REVISION`, a commit other than the recipe’s | Another revision of the repository may hold other files. The downloaded file must still match the recipe’s sha256. |

When you do any of these, the launcher says so before it starts:

```text
UNTUNED: --context 32768 instead of 65536. This is not the configuration the receipt measured; its numbers do not apply.
```

`tensorless info --json` reports the same thing in its `untuned` field, `--dry-run` prints it in the plan, and `tensorless bench` adds a note under its results.

Some options do not make a run untuned: `--port`, `--host`, `--gpu`, `--runtime` and `--cache`. They change where the server listens, which card it uses, how the container starts and where the model is stored, but not what the engine computes.

## Why label, and not forbid

[Section titled “Why label, and not forbid”](#why-label-and-not-forbid)

The launcher’s rule is to [refuse rather than guess](/docs/explanation/how-tensorless-works/#refuse-rather-than-guess). An untuned run is not a guess, though. It is you choosing something on purpose: a shorter context to leave room for another process, a sampling flag your client needs, a newer revision to test. Refusing those would only push people to run the engine by hand, where nothing checks anything.

So the line is drawn by what the launcher can know:

* **Things it can check, it checks,** every time: the card and the driver (unless you pass `--card`, which skips GPU detection), the image digest, the model’s hash. Those are refused when wrong.
* **Things that are your choice, it allows,** and labels. The label is there so that nobody, you included, compares an untuned run with the receipt and draws the wrong conclusion.

A run that is slower than its receipt is worth knowing about only if it is tuned. That is why the label exists.

## Drafts and unpublished recipes

[Section titled “Drafts and unpublished recipes”](#drafts-and-unpublished-recipes)

Two more kinds of recipe sit outside the tuned path:

* **A draft** is a recipe that has not passed the release gate. Its settings may be right, but nothing has confirmed them on the image it names. The launcher refuses it unless you pass `--allow-draft`, and says it is using a draft when you do. `MODEL@REVISION` needs the same flag, for the same reason: no gate measured it.
* **An unpublished recipe** still has placeholder values: an all-zero revision or image digest, or a model size of 0. It cannot run at all, even with `--allow-draft`, because there is nothing real to pull. `tensorless info` and `--dry-run` still show it, so its plan can be reviewed before the upload exists.

Both released recipes in tensorless v0.1.0 need no flags. `tensorless models` shows each recipe’s status.

# Why the P100

> Why tensorless starts with a 2016 data-center card, what limits its speed, and what the engine does about it.

The first card tensorless supports is the NVIDIA Tesla P100 16 GB. This page explains why that card is worth the work, what limits how fast it can run a 27-billion-parameter model, and why it needed its own engine and its own weight format.

## A lot of fast memory for little money

[Section titled “A lot of fast memory for little money”](#a-lot-of-fast-memory-for-little-money)

The P100 is a 2016 data-center card from NVIDIA’s Pascal generation. Retired from servers, it sells used for around $100. For that price you get:

* **16 GB of HBM2 memory,** stacked next to the chip, at **732 GB/s** (the PCIe 16 GB model);
* **compute capability 6.0** (`sm_60`), the data-center member of the Pascal family.

16 GB is enough to hold a 27B model at about 3 bits per weight, with a long context beside it. 732 GB/s is more memory bandwidth than many much newer consumer cards have. For running a language model one token at a time, that bandwidth is what matters most.

## Why memory bandwidth sets the pace

[Section titled “Why memory bandwidth sets the pace”](#why-memory-bandwidth-sets-the-pace)

Generating text (decode) produces one token at a time. For each token, the model’s weights are read from memory once and each is used for only a little arithmetic. So decode is **memory-bound**: the card spends its time waiting for weights to arrive, not computing with them.

That gives a simple ceiling. Here is an estimate for Qwen3.8-27B-PN3:

```text
model file                  12,954,114,496 bytes   (12.1 GiB)
rated memory bandwidth      732 GB/s
time to read every byte     12.95 GB / 732 GB/s  ≈ 17.7 ms
ceiling                     ≈ 56 tokens per second
```

This is a rough upper bound, not a prediction. It assumes every byte of the file is read once per token at the full rated bandwidth. In practice the embedding table is only looked up, not read whole; the KV cache adds reads that grow with the context; and no real kernel reaches the rated bandwidth. The released engine decodes this model at about 30 tokens per second at short context, a bit over half the estimate.

The same arithmetic says why low-bit formats exist at all: fewer bytes per weight means fewer bytes to read per token.

## Why bytes are not the whole story on sm_60

[Section titled “Why bytes are not the whole story on sm_60”](#why-bytes-are-not-the-whole-story-on-sm_60)

On newer cards, fewer bits per weight almost always means faster decode. On the P100 it does not, and the reason is what `sm_60` lacks:

* **No DP4A.** The other Pascal chips (`sm_61`, such as the GeForce GTX 10 series and the Tesla P40) and every generation since have an instruction that multiplies and adds four 8-bit integers at once. llama.cpp’s quantized kernels lean on it. `sm_60` does not have it, so those kernels fall back to slower paths.
* **No tensor cores.** They arrived with Volta, the next generation.

Without those, unpacking each weight costs instructions, and the lowest-bit formats cost the most instructions to unpack. A GEMV (the matrix-vector product at the heart of decode) on the P100 can become limited by **instructions per weight** rather than bits per weight.

The evidence is in the two launch targets:

| Model                          | File size | Decode, short context |
| ------------------------------ | --------: | --------------------: |
| Qwen3.8-27B-UD-IQ3_XXS-Q3Floor |  11.0 GiB |            26.3 tok/s |
| Qwen3.8-27B-PN3                |  12.1 GiB |            30.1 tok/s |

The PN3 file is about 10% larger and decodes about 14% faster. Its weights are laid out so the P100 can unpack them cheaply. The Q3Floor model shows the same effect from the other side: raising its 91 sub-3-bit tensors to 3 bits made it both better and faster on this card ([its model card](https://github.com/iammrduncan/tensorless/blob/main/docs/model-cards/Qwen3.8-27B-UD-IQ3_XXS-Q3Floor-GGUF.md) has the numbers). [Why a P100-native weight format](/docs/explanation/pn-format/) explains the layout.

## What it takes to keep running Pascal

[Section titled “What it takes to keep running Pascal”](#what-it-takes-to-keep-running-pascal)

The P100 is an old card, and the software around it is moving on:

* **CUDA 13 dropped Pascal.** CUDA 12 is the last major version that compiles for `sm_60`, and the engine image is built with CUDA 12.6. The CUDA runtime is inside the image, so your host does not need CUDA at all.
* **The R580 driver branch is Pascal’s last.** The image works with drivers from R525 to R580. Newer branches will not support the card.

So the P100 engine is a fixed target: one architecture, one CUDA line, one driver range. That is part of why the work pays off. A kernel tuned for `sm_60` today will still be the right kernel for every P100 next year.

## Why a whole engine, and not settings for llama.cpp

[Section titled “Why a whole engine, and not settings for llama.cpp”](#why-a-whole-engine-and-not-settings-for-llamacpp)

Stock llama.cpp runs on the P100, but its fast paths assume features the card does not have. Getting from “runs” to “fast” took new kernels: decode attention, GEMVs for each weight type the models use, an output-head kernel, and prefill paths, each written for `sm_60` and switched on by the recipe. [Where the engine comes from](/docs/explanation/the-engine/) tells that story, and the [engine switches reference](/docs/reference/engine-switches/) lists the switches.

# Build the engine image yourself

> Build the sm_60 engine image from the repository's Containerfile with podman or docker, check it, and run it.

The engine image is built from the source in [`engine/`](https://github.com/iammrduncan/tensorless/tree/main/engine) by one Containerfile per GPU architecture. Build it yourself to read exactly what goes into it, to try a change to the engine, or to run without the published registry.

## Before you start

[Section titled “Before you start”](#before-you-start)

You need:

* an x86-64 Linux machine with podman or docker. Building needs no GPU;
* `git`;
* several GB of free disk space for the CUDA build image and the build.

The build compiles the CUDA kernels on every core of the machine, so it takes a while.

## 1. Get the source at a release

[Section titled “1. Get the source at a release”](#1-get-the-source-at-a-release)

```sh
git clone --branch engine-v0.1.0 --depth 1 https://github.com/iammrduncan/tensorless
cd tensorless
```

`engine-v0.1.0` is the tag of the engine source in the `0.1.0-sm60` image. To build your own change, check out your branch instead.

## 2. Build

[Section titled “2. Build”](#2-build)

From the repository root, which is the build context:

```sh
podman build -f containers/engine.sm60.Containerfile --build-arg VERSION=0.1.0-local \
  -t tensorless-engine:0.1.0-local-sm60 .
# or: docker build -f containers/engine.sm60.Containerfile --build-arg VERSION=0.1.0-local -t tensorless-engine:0.1.0-local-sm60 .
```

`VERSION` is written into the image as `/opt/tensorless/VERSION`. Pick a label that says the image is yours.

The build stage configures the engine with the measured CUDA settings (`sm_60` only, flash attention with the measured KV-cache types, no CUDA graphs, no VMM) and a portable x86-64 CPU baseline. The full list is in [Engine image](/docs/reference/engine-image/#cmake-configuration). It then builds `llama-server` and `test-pn-format`, and runs `test-pn-format`. If the PN format test fails, the build fails.

## 3. Check the image

[Section titled “3. Check the image”](#3-check-the-image)

The image lists the sha256 of every file it ships under `/opt/tensorless`. Check them, and read the version:

```sh
podman run --rm --entrypoint sh tensorless-engine:0.1.0-local-sm60 \
  -c 'cd /opt/tensorless && sha256sum -c --quiet SHA256SUMS && cat VERSION'
```

It prints `0.1.0-local` and nothing else. A mismatched file prints `FAILED`.

## 4. Run it

[Section titled “4. Run it”](#4-run-it)

Your image’s digest is not the release’s. Builds are not byte for byte reproducible: the base images and packages they install change over time, and so do the layers’ timestamps. The recipes pin the published digest, so the launcher does not use your build.

Run it as [Run the engine image without the launcher](/docs/how-to/advanced/without-launcher/) shows, with `tensorless-engine:0.1.0-local-sm60` in place of the published image. To use it through the launcher instead, push it to a registry you control and pin its digest in a recipe of your own: see [Try a recipe of your own](/docs/how-to/advanced/write-a-recipe/).

Note

A run on your own build is not what the receipts measured, even from the same source. To compare, measure it with `tensorless bench`: see [Benchmark a running server](/docs/how-to/use/benchmark/).

## How release images are built

[Section titled “How release images are built”](#how-release-images-are-built)

Release images are built by the [`engine-image`](https://github.com/iammrduncan/tensorless/blob/main/.github/workflows/engine-image.yml) workflow, from the same Containerfile, as a release candidate (`X.Y.Z-rcN-sm60`). After the release gate has measured that candidate on a P100, the [`engine-promote`](https://github.com/iammrduncan/tensorless/blob/main/.github/workflows/engine-promote.yml) workflow gives its digest the release tag without rebuilding it. Why is in [How a release is made](/docs/explanation/releases-and-receipts/).

# Run the engine image without the launcher

> Pull the engine image, download and check the model yourself, and start llama-server with podman or docker exactly as a recipe does.

The launcher only runs a container. You can run the same container yourself, for example from your own scripts, a compose file or a service manager. This guide starts Qwen3.8-27B-PN3 exactly as its recipe does.

## Before you start

[Section titled “Before you start”](#before-you-start)

You need:

* a Linux x86_64 host (the engine image is linux/amd64) with a Tesla P100 16 GB that no other process is using;
* an x86-64 CPU with AVX2, FMA and F16C;
* an NVIDIA driver from R525 to R580;
* podman with an NVIDIA CDI spec, or docker with the NVIDIA container toolkit (see [Prepare a Linux host for a P100](/docs/how-to/setup/prepare-host/));
* about 13 GB of free disk space for the model, and a few GB for the image;
* `curl` and `sha256sum`.

Tip

`tensorless info pn3 --card p100` prints the exact container command for your machine, on any computer, with no GPU: `tensorless info` works on macOS too. Use it to check the command below, or to get the one for another recipe.

## 1. Pull the engine image by digest

[Section titled “1. Pull the engine image by digest”](#1-pull-the-engine-image-by-digest)

```sh
podman pull ghcr.io/iammrduncan/tensorless-engine@sha256:35409ff97c2f518e468790c28bd19206ae777a945fcffe5775585e28fd9939a7
# or: docker pull ghcr.io/iammrduncan/tensorless-engine@sha256:35409ff97c2f518e468790c28bd19206ae777a945fcffe5775585e28fd9939a7
```

Pull by digest, not by tag: the digest is what the receipt measured.

## 2. Download the model and check it

[Section titled “2. Download the model and check it”](#2-download-the-model-and-check-it)

Download the file at the revision the recipe pins, then check its sha256:

```sh
mkdir -p ~/models && cd ~/models
curl -fL -C - -o Qwen3.8-27B-PN3-P0k19.gguf \
  https://huggingface.co/iamMrDuncan/Qwen3.8-27B-PN3-GGUF/resolve/c701b6690ec360b16d40f85df1cb76062e30cd38/Qwen3.8-27B-PN3-P0k19.gguf
echo "15ce2d4469673f1f1753c55de3780cfe04311dd6f05cb41633d73df128e91476  Qwen3.8-27B-PN3-P0k19.gguf" | sha256sum -c
```

`sha256sum` prints `Qwen3.8-27B-PN3-P0k19.gguf: OK`. If the download stops, run the `curl` line again: `-C -` resumes it. The values for each model are in [Launch targets](/docs/reference/launch-targets/).

## 3. Start the server

[Section titled “3. Start the server”](#3-start-the-server)

With podman:

```sh
podman run -d --rm --name tensorless-pn3 \
  --device nvidia.com/gpu=all --security-opt label=disable \
  -p 127.0.0.1:8080:8080 \
  -v ~/models/Qwen3.8-27B-PN3-P0k19.gguf:/models/Qwen3.8-27B-PN3-P0k19.gguf:ro \
  -e OMP_NUM_THREADS=8 \
  -e GGML_CUDA_FATTN_SM60=1 -e GGML_CUDA_FATTN_SM60_CTAS=4 -e GGML_CUDA_FATTN_TILE_OCC1=1 \
  -e GGML_CUDA_FUSE_PRE_ADD=1 -e GGML_CUDA_MMV_SM60=1 \
  -e GGML_CUDA_MMV_SM60_INPLACE_IQ3S=1 -e GGML_CUDA_MMV_SM60_INPLACE_XXS=1 \
  -e GGML_CUDA_MMV_SM60_IQ3S_STOCK=1 -e GGML_CUDA_MMV_SM60_Q3K=1 \
  -e GGML_CUDA_MMV_SM60_Q4K_DHEAD=1 -e GGML_CUDA_MMV_SM60_Q4K_HEAD=1 \
  -e GGML_CUDA_MMV_SM60_Q6K=1 -e GGML_CUDA_MMV_SM60_XXS_KB_WIDE=1 \
  -e GGML_CUDA_MMV_SM60_XXS_STOCK=1 -e GGML_CUDA_NORM_EMIT_Q8=1 \
  -e GGML_CUDA_PREFILL_CHUNK_GEMM=1 -e GGML_CUDA_PREFILL_CHUNK_STREAMS=3 \
  -e GGML_CUDA_PREFILL_DEQ_F4=1 -e GGML_CUDA_PREFILL_REP_DIRECT=1 \
  -e GGML_CUDA_SM86_GDN_COLS=8 -e GGML_CUDA_SM86_IQ3_SMEM_GRID=1 \
  ghcr.io/iammrduncan/tensorless-engine@sha256:35409ff97c2f518e468790c28bd19206ae777a945fcffe5775585e28fd9939a7 \
  -m /models/Qwen3.8-27B-PN3-P0k19.gguf --host 0.0.0.0 --port 8080 -c 65536 \
  --alias iamMrDuncan/Qwen3.8-27B-PN3-GGUF -t 8 -tb 8 \
  -np 1 -ngl 99 -ot token_embd.weight=CUDA0 -ctk q4_0 -ctv q4_0 -sm none -lm none -fit off \
  --threads-http 4 -b 512 -ub 512 -fa on --jinja
```

With docker, replace the second line with `--gpus all`, as the launcher does.

What the parts are:

| Part                              | From the recipe                                                                               |
| --------------------------------- | --------------------------------------------------------------------------------------------- |
| `-e GGML_CUDA_...`                | `[run.env]`: the 21 [engine switches](/docs/reference/engine-switches/), exactly as measured. |
| `-c 65536`                        | `run.context`.                                                                                |
| `--alias`                         | `run.alias`, the served model name.                                                           |
| `-t 8 -tb 8`, `OMP_NUM_THREADS=8` | `run.threads_max`. The launcher uses the smaller of this and your CPU count.                  |
| Everything after `-tb 8`          | `run.args`, in order.                                                                         |

The image’s entrypoint is `llama-server`, so the arguments after the image name go to it. To give the container only one GPU, use `--device nvidia.com/gpu=N` (podman) or `--gpus device=N` (docker).

## 4. Check that it works

[Section titled “4. Check that it works”](#4-check-that-it-works)

The model takes a little while to load. Follow the logs until the server listens:

```sh
podman logs -f tensorless-pn3
```

Then, from another terminal:

```sh
curl -s http://127.0.0.1:8080/health
curl -s http://127.0.0.1:8080/v1/models
```

`/health` answers `{"status":"ok"}` once the model is loaded, and `/v1/models` lists `iamMrDuncan/Qwen3.8-27B-PN3-GGUF`. To measure it the way the receipts did, run `tensorless bench pn3 --card p100`: `bench` needs only the server.

## 5. Stop it

[Section titled “5. Stop it”](#5-stop-it)

```sh
podman stop tensorless-pn3
```

The container was started with `--rm`, so stopping it also removes it.

## If it fails

[Section titled “If it fails”](#if-it-fails)

* `no kernel image is available` in the logs: the image does not match the card. The `sm60` image runs only on compute capability 6.0.
* An out-of-memory error: another process is using the GPU. The model and its context take about 14 GB of the card’s 16 GB.
* An error about the GPU device: podman has no CDI spec, or docker has no NVIDIA support. `tensorless doctor` checks both and prints the fix.

# Try a recipe of your own

> Write a draft recipe in a directory of your own, check it with info and --dry-run, and run it with --recipes and --allow-draft.

A recipe is a small TOML file: one model on one card. To try other settings, another model or your own engine build, write a draft recipe in a directory of your own and point the launcher at it. Nothing in the launcher needs to change.

## Before you start

[Section titled “Before you start”](#before-you-start)

You need:

* the `tensorless` launcher (see [Install the launcher](/docs/how-to/setup/install-launcher/));
* for a new model: its Hugging Face repository, the commit to pin, the GGUF file’s name, its sha256 and its exact size in bytes;
* the [Recipe file](/docs/reference/recipes/) reference open beside you.

## 1. Make a recipes directory

[Section titled “1. Make a recipes directory”](#1-make-a-recipes-directory)

A recipes directory holds `<org>/<repo>/<card>.toml`, where `<org>/<repo>` is the model’s Hugging Face repository. Start from a built-in recipe:

```sh
mkdir -p ~/my-recipes/iamMrDuncan/Qwen3.8-27B-PN3-GGUF
curl -fsSL -o ~/my-recipes/iamMrDuncan/Qwen3.8-27B-PN3-GGUF/p100.toml \
  https://raw.githubusercontent.com/iammrduncan/tensorless/main/recipes/iamMrDuncan/Qwen3.8-27B-PN3-GGUF/p100.toml
```

With `--recipes DIR`, the launcher reads only that directory, not its built-in recipes. Copy every recipe you want to use with it.

## 2. Make it a draft and change it

[Section titled “2. Make it a draft and change it”](#2-make-it-a-draft-and-change-it)

Open the file and set its status to draft:

```toml
status = "draft"
```

Then make your change. For example:

* **Other settings:** edit `[run]` (`context`, `args`) or `[run.env]` (the [engine switches](/docs/reference/engine-switches/)).
* **Another model:** put the file under that model’s `<org>/<repo>/` directory, and fill in `[model]`: `hf_repo` (the same as the directory), `revision` (a 40-character commit, not a branch), `file`, `sha256`, `size_bytes` and `license`. Change `run.alias` to the new repo id, and `model.name` to a short name of its own, or remove it.
* **Your own engine build:** push it to a registry you control, and set `[engine].image` to `<registry>/<name>@sha256:<digest>`. A tag is refused.

To get a file’s sha256 and size from a copy you downloaded:

```sh
sha256sum MODEL.gguf
stat -c %s MODEL.gguf
```

Leave out the `llama-server` flags the launcher sets itself: `-m`, `--host`, `--port`, `-c`, `--alias`, `-t` and `-tb` (and their long forms), and `OMP_NUM_THREADS` in `[run.env]`.

## 3. Check it

[Section titled “3. Check it”](#3-check-it)

`info` reads the recipe and shows it exactly as it would run. It needs no GPU:

```sh
tensorless info iamMrDuncan/Qwen3.8-27B-PN3-GGUF --recipes ~/my-recipes --card p100
```

A recipe with a mistake fails with every problem listed, for example:

```text
tensorless: /home/user/my-recipes/iamMrDuncan/Qwen3.8-27B-PN3-GGUF/p100.toml: invalid recipe:
  - engine.image must be pinned by digest (name@sha256:<64 hex>), not "ghcr.io/iammrduncan/tensorless-engine:0.1.0-sm60"
  - run.args must not set -c; the launcher sets it
```

A misspelled key fails with `unknown keys:` and its name. Fix the file and run `info` again until it shows the recipe.

Then see the whole plan, with nothing pulled or started:

```sh
tensorless run iamMrDuncan/Qwen3.8-27B-PN3-GGUF --recipes ~/my-recipes --allow-draft --dry-run
```

`tensorless models --recipes ~/my-recipes` lists it with the status `draft`.

## 4. Run it

[Section titled “4. Run it”](#4-run-it)

```sh
tensorless run iamMrDuncan/Qwen3.8-27B-PN3-GGUF --recipes ~/my-recipes --allow-draft
```

Without `--allow-draft`, the launcher refuses a draft:

```text
tensorless: the recipe for iamMrDuncan/Qwen3.8-27B-PN3-GGUF on p100 is a draft: it has not passed the release gate. Pass --allow-draft to use it anyway; 'tensorless info iamMrDuncan/Qwen3.8-27B-PN3-GGUF' shows it
```

With it, the launcher prints `note: using the draft recipe for iamMrDuncan/Qwen3.8-27B-PN3-GGUF on p100 (--allow-draft)`, then pulls, verifies and serves as usual. It has worked when it prints that the model `is ready at http://127.0.0.1:8080`.

A recipe whose revision, sha256 or image digest is still all zeros, or whose `size_bytes` is 0, is refused even with `--allow-draft`: those are placeholders, and nothing can be pulled for them.

## 5. Measure it

[Section titled “5. Measure it”](#5-measure-it)

With the server running, measure decode speed:

```sh
tensorless bench iamMrDuncan/Qwen3.8-27B-PN3-GGUF --recipes ~/my-recipes
```

`bench` compares the result with the recipe’s `[receipt]`. If you changed the settings, that receipt no longer describes them: remove or replace its numbers. See [Benchmark a running server](/docs/how-to/use/benchmark/).

## Make it a released recipe

[Section titled “Make it a released recipe”](#make-it-a-released-recipe)

A recipe becomes `released` in the repository, through a pull request, only with a receipt: the release gate’s measurements of that model on that card, with that image. A pull request that changes `[run]` or `[engine]` of a released recipe needs a new receipt too. How to propose one is in [Contributing](/docs/project/contributing/).

# Change the context length or server arguments

> Run a model with another context length, or pass extra llama-server arguments, and know what that does to its measured numbers.

Each recipe runs `llama-server` with exactly the settings that were measured, including a context of 65,536 tokens. You can change the context length, or add `llama-server` arguments of your own. The launcher then calls the run **untuned**: it is no longer the configuration the receipt measured, so the receipt’s numbers don’t apply to it.

You need the launcher installed and a host that passes `tensorless doctor`. See [Prepare a Linux host for a P100](/docs/how-to/setup/prepare-host/). Words such as *recipe* and *receipt* are in the [glossary](/docs/reference/glossary/).

## See the measured settings first

[Section titled “See the measured settings first”](#see-the-measured-settings-first)

```sh
tensorless info pn3
```

Under **Run**, `context` is the context length and `args` lists every other argument the recipe passes, in order. [See exactly what will run](/docs/how-to/run/preview-a-run/) explains the rest of the output.

## Change the context length

[Section titled “Change the context length”](#change-the-context-length)

Pass `--context` to `run` or `serve`:

```sh
tensorless run iamMrDuncan/Qwen3.8-27B-PN3-GGUF --context 32768
```

Before it starts, the launcher says what differs from the recipe:

```text
UNTUNED: --context 32768 instead of 65536. This is not the configuration the receipt measured; its numbers do not apply.
```

The value goes to `llama-server` as `-c`. With `-np 1` in the recipes, the server has one slot, and that slot gets the whole context.

Caution

The recipes’ 65,536-token context already takes nearly all of the P100’s 16 GB: with Qwen3.8-27B-PN3, the published image used 13,997 MiB once loaded and 14,577 MiB while generating ([the receipt](/docs/reference/receipts/)). A larger context needs more memory than the card has, and the server fails while it loads. A smaller one frees memory.

## Pass extra llama-server arguments

[Section titled “Pass extra llama-server arguments”](#pass-extra-llama-server-arguments)

Everything after `--` goes to `llama-server`, after the recipe’s own arguments:

```sh
tensorless run iamMrDuncan/Qwen3.8-27B-PN3-GGUF -- --temp 0.7
```

```text
UNTUNED: extra arguments --temp 0.7. This is not the configuration the receipt measured; its numbers do not apply.
```

Because they come last, your arguments can also override one of the recipe’s. The launcher sets the model, address, context, alias and threads itself, so it refuses `-m`, `--host`, `--port`, `-c`, `--alias`, `-t` and `-tb` (and their long forms) after `--`:

```text
tensorless: do not pass -c after --: the launcher sets it. Use --host, --port or --context instead; the model, alias and threads come from the recipe
Run 'tensorless help' for usage.
```

Only `run`, `serve` and `info` take arguments after `--`. The others refuse them.

Tip

Sampling settings such as temperature can also be sent with each request, which leaves the run tuned. See [Connect an OpenAI-compatible client](/docs/how-to/use/openai-clients/).

## Combine both

[Section titled “Combine both”](#combine-both)

```sh
tensorless run iamMrDuncan/Qwen3.8-27B-PN3-GGUF --context 32768 -- --temp 0.7
```

```text
UNTUNED: --context 32768 instead of 65536 and extra arguments --temp 0.7. This is not the configuration the receipt measured; its numbers do not apply.
```

## Check that it worked

[Section titled “Check that it worked”](#check-that-it-worked)

Preview the command before you run it, with the same flags and `--dry-run`. The plan’s header includes the `UNTUNED` line, and the `podman run` (or `docker run`) command shows your `-c` value and, at the end, your extra arguments:

```sh
tensorless run iamMrDuncan/Qwen3.8-27B-PN3-GGUF --context 32768 --dry-run -- --temp 0.7
```

To measure an untuned run, start the server, then run `tensorless bench` with the same `--context`. The result notes that the run is untuned, and its comparison with the receipt is only a rough guide. [Tuned and untuned runs](/docs/explanation/tuned-and-untuned/) explains why.

# Keep models on another disk

> Point the launcher's model downloads at a larger disk with --cache or the Hugging Face cache variables, and move models you already have.

Each launch target is one file of 11.8 to 13 GB. The launcher keeps it in the Hugging Face cache, by default in your home directory. This page puts the cache on another disk, and moves models you have already downloaded.

You need the launcher installed. See [Install the launcher](/docs/how-to/setup/install-launcher/).

## Where the cache is

[Section titled “Where the cache is”](#where-the-cache-is)

The launcher uses the first of these that is set:

1. `--cache DIR` on the command line;
2. the `HF_HUB_CACHE` environment variable;
3. `HF_HOME`, with `/hub` added;
4. `~/.cache/huggingface/hub`.

These are the same variables the Hugging Face tools read, so the launcher and those tools share one cache.

## 1. Choose the new place

[Section titled “1. Choose the new place”](#1-choose-the-new-place)

Pick a directory on a disk with room for every model you want, about 13 GB each. Its path must not contain `:`, because the container mount cannot express one; the launcher refuses such a path.

## 2. Point the launcher at it

[Section titled “2. Point the launcher at it”](#2-point-the-launcher-at-it)

For every command, set the variable in your shell’s profile:

```sh
export HF_HUB_CACHE=/data/hf-cache
```

Or for one command, pass `--cache`:

```sh
tensorless run iamMrDuncan/Qwen3.8-27B-PN3-GGUF --cache /data/hf-cache
```

Use the same cache for `pull`, `run`, `serve` and `models`: a command that looks in another cache does not find the file there.

Caution

`sudo` and service managers usually change `$HOME` and the environment, so a command run that way looks in another cache. The launcher itself does not need root: run it as the user who owns the cache.

## 3. Move models you already have (optional)

[Section titled “3. Move models you already have (optional)”](#3-move-models-you-already-have-optional)

Copy the model’s whole folder, keeping its symlinks, then remove the old one:

```sh
rsync -a ~/.cache/huggingface/hub/models--iamMrDuncan--Qwen3.8-27B-PN3-GGUF /data/hf-cache/
rm -rf ~/.cache/huggingface/hub/models--iamMrDuncan--Qwen3.8-27B-PN3-GGUF
```

The next command that uses the file checks its sha256 once more, because the launcher remembers a verified file by its path. That reads the whole file, so it takes a while; after it, the check is skipped again.

## What is in the cache

[Section titled “What is in the cache”](#what-is-in-the-cache)

For each model, the launcher writes the layout the Hugging Face tools use:

```text
models--iamMrDuncan--Qwen3.8-27B-PN3-GGUF/
  blobs/15ce2d4469673f1f1753c55de3780cfe04311dd6f05cb41633d73df128e91476
  snapshots/c701b6690ec360b16d40f85df1cb76062e30cd38/Qwen3.8-27B-PN3-P0k19.gguf -> ../../blobs/15ce2d44...
```

* **The blob** is the file itself, named by its sha256. The server’s container mounts it read-only.
* **The snapshot** is a symlink named after the recipe’s revision and file, so other Hugging Face tools that ask for that revision find the file.
* **A partial download** is the blob’s name with `.incomplete` added. Run the same command again to resume it where it stopped. `tensorless models` shows such a model as `partial`.

Before a download, the launcher checks that the disk has room for it and refuses with `not enough disk space` if not.

The launcher remembers which files it has verified in `$XDG_STATE_HOME/tensorless/verified/`, or `~/.local/state/tensorless/verified/` when `XDG_STATE_HOME` is not set: one small file per model, named by its sha256. While the model file keeps the same path, size and modification time, the launcher does not hash it again. Deleting these markers is safe; the next run verifies again.

## Check that it worked

[Section titled “Check that it worked”](#check-that-it-worked)

```sh
tensorless models
```

The **CACHED** column says `yes` for a model in the new cache. `tensorless info pn3` shows the full path under **Model**, on the `cache` line, and `tensorless doctor` reports the free space where the cache now is.

# Pick a GPU on a host with several

> Choose which GPU a model runs on with --gpu, run one model per card, and know what --card skips.

On a host with more than one GPU, the launcher picks the first one that matches the recipe. This page chooses the GPU yourself, and runs a different model on each card.

You need the launcher installed and a host that passes `tensorless doctor`. See [Prepare a Linux host for a P100](/docs/how-to/setup/prepare-host/).

## 1. Find the GPU’s index

[Section titled “1. Find the GPU’s index”](#1-find-the-gpus-index)

The launcher numbers GPUs the way `nvidia-smi` does. List them:

```sh
nvidia-smi --query-gpu=index,name,memory.total --format=csv
```

`tensorless doctor` lists them too, with the recipes each one fits.

## 2. Run the model on that GPU

[Section titled “2. Run the model on that GPU”](#2-run-the-model-on-that-gpu)

Pass the index with `--gpu`:

```sh
tensorless run iamMrDuncan/Qwen3.8-27B-PN3-GGUF --gpu 1
```

The launcher checks that GPU 1 exists and matches the recipe (its compute capability, its memory and the driver), and the container then gets only that GPU: `--device nvidia.com/gpu=1` with podman, `--gpus device=1` with docker. If there is no GPU 1, it says so and lists the ones `nvidia-smi` sees. If GPU 1 does not fit the recipe, it says what the recipe needs.

## How the launcher chooses without —gpu

[Section titled “How the launcher chooses without —gpu”](#how-the-launcher-chooses-without-gpu)

* It takes the first GPU, by index, that matches the recipe.
* If every GPU on the host matches, the container gets all of them. The recipes run `llama-server` with `-sm none`, which keeps the model on one card.
* If some GPUs don’t match, the container gets only the matched one, so `llama-server` cannot land on a card the engine image was not built for.

## Run one model per card

[Section titled “Run one model per card”](#run-one-model-per-card)

Each model gets its own container, named after its repo id, so two different models can run side by side on two cards. Give each its own GPU and port:

```sh
tensorless run iamMrDuncan/Qwen3.8-27B-PN3-GGUF --detach --gpu 0
tensorless run iamMrDuncan/Qwen3.8-27B-UD-IQ3_XXS-Q3Floor-GGUF --detach --gpu 1 --port 8081
```

The same model cannot run twice: its container name is taken, and the second `run` says it `is already running`.

## Skip detection with —card

[Section titled “Skip detection with —card”](#skip-detection-with-card)

`--card p100` chooses the P100 recipe without asking `nvidia-smi`. It is meant for showing a recipe (`info`, `--dry-run`) on a machine without the card. When you start a server with it:

* the launcher checks nothing about the GPU: not its compute capability, its memory or the driver version;
* the container gets every GPU, unless you also pass `--gpu`.

So on a real P100 host, leave `--card` out and let the launcher check the card.

## Check that it worked

[Section titled “Check that it worked”](#check-that-it-worked)

While the model runs, `nvidia-smi` shows its memory in use on the GPU you chose (about 14,000 MiB for Qwen3.8-27B-PN3), and the other cards stay free for other work:

```sh
nvidia-smi --query-gpu=index,memory.used --format=csv
```

# See exactly what will run

> Show a recipe, its download and its full container command with tensorless info and --dry-run, before anything is downloaded or started.

Before you download 13 GB or start a server, you can see exactly what the launcher would do: which file it downloads from where, which image it pulls by digest, and the full `podman run` or `docker run` command. `tensorless info` shows one recipe; `--dry-run` shows the plan of a command.

You need the launcher installed. See [Install the launcher](/docs/how-to/setup/install-launcher/). Neither needs a GPU, so you can do this on a laptop.

## Show a recipe

[Section titled “Show a recipe”](#show-a-recipe)

```sh
tensorless info iamMrDuncan/Qwen3.8-27B-PN3-GGUF
```

```text
iamMrDuncan/Qwen3.8-27B-PN3-GGUF on p100: Qwen3.8-27B-PN3 (P100-native 3-bit, mix P0k+19)
  short name      pn3
  status          released
  recipe          recipes/iamMrDuncan/Qwen3.8-27B-PN3-GGUF/p100.toml
  fits this host  unknown (nvidia-smi not found)


Model
  repository  iamMrDuncan/Qwen3.8-27B-PN3-GGUF
  revision    c701b6690ec360b16d40f85df1cb76062e30cd38
  file        Qwen3.8-27B-PN3-P0k19.gguf
  sha256      15ce2d4469673f1f1753c55de3780cfe04311dd6f05cb41633d73df128e91476
  size        12.95 GB (12954114496 bytes)
  license     Apache-2.0 (from Qwen/Qwen3.8-27B)
  url         https://huggingface.co/iamMrDuncan/Qwen3.8-27B-PN3-GGUF/resolve/c701b6690ec360b16d40f85df1cb76062e30cd38/Qwen3.8-27B-PN3-P0k19.gguf
  cache       /home/you/.cache/huggingface/hub/models--iamMrDuncan--Qwen3.8-27B-PN3-GGUF/blobs/15ce2d4469673f1f1753c55de3780cfe04311dd6f05cb41633d73df128e91476
  cached      no


Card
  title    NVIDIA Tesla P100 16 GB
  compute  6.0
  memory   at least 16000 MiB
  driver   525.60.13 or newer


Engine
  release  engine-v0.1.0
  arch     sm60
  image    ghcr.io/iammrduncan/tensorless-engine@sha256:35409ff97c2f518e468790c28bd19206ae777a945fcffe5775585e28fd9939a7
```

The output goes on with **Run** (the context, the arguments, the engine switches and the full container command) and **Receipt** (what was measured). Each part:

* **The header** says which recipe file this is, whether it is released, and whether a GPU on this host fits it.
* **Model** is the exact file: its repository and pinned revision, its sha256 and size, the download address, and where it goes in your cache.
* **Card** is what the GPU needs: compute capability, memory and driver.
* **Engine** is the image, by digest, and the engine release it was measured on.
* **Run** is how the server is started, with the command your host would run.
* **Receipt** is what was measured with exactly these settings. The [Receipts](/docs/reference/receipts/) page has them all.

`info` takes most of the flags `run` takes. Add `--context`, `--port`, `--host`, `--gpu` or arguments after `--`, and it shows the command with them, with an `UNTUNED` line in the header when they change the measured configuration. Add `--runtime docker` to see the docker command.

## Get it as JSON

[Section titled “Get it as JSON”](#get-it-as-json)

```sh
tensorless info pn3 --json
```

The JSON holds the whole recipe under `recipe`, and what the launcher worked out for this host: `published`, `fits`, `download` (the URL, the blob and snapshot paths, the size and whether it is cached), `runtime`, `gpu`, `threads`, `command` (as a list of arguments) and `endpoint`.

`tensorless models --json` and `tensorless capabilities --json` list every recipe in short form: model, card, status and engine release.

## Show the plan of a command

[Section titled “Show the plan of a command”](#show-the-plan-of-a-command)

Add `--dry-run` to `run`, `pull`, `serve` or `stop`. Nothing is pulled, downloaded or started:

```sh
tensorless run iamMrDuncan/Qwen3.8-27B-PN3-GGUF --card p100 --dry-run
```

```text
dry run of 'tensorless run iamMrDuncan/Qwen3.8-27B-PN3-GGUF': nothing is pulled, downloaded or started.
recipe:   iamMrDuncan/Qwen3.8-27B-PN3-GGUF on p100 (released), from recipes/iamMrDuncan/Qwen3.8-27B-PN3-GGUF/p100.toml
gpu:      chosen with --card; GPU detection skipped; the container gets GPU all
runtime:  podman


# 1. pull the engine image, unless 'podman image inspect' finds it
podman pull \
    ghcr.io/iammrduncan/tensorless-engine@sha256:35409ff97c2f518e468790c28bd19206ae777a945fcffe5775585e28fd9939a7


# 2. download the model (12.95 GB), resuming a partial download, and check its sha256 15ce2d4469673f1f1753c55de3780cfe04311dd6f05cb41633d73df128e91476
GET https://huggingface.co/iamMrDuncan/Qwen3.8-27B-PN3-GGUF/resolve/c701b6690ec360b16d40f85df1cb76062e30cd38/Qwen3.8-27B-PN3-P0k19.gguf
    into /home/you/.cache/huggingface/hub/models--iamMrDuncan--Qwen3.8-27B-PN3-GGUF/blobs/15ce2d4469673f1f1753c55de3780cfe04311dd6f05cb41633d73df128e91476
```

It goes on with the snapshot link, the full `podman run` command, and how it waits for the server. This example ran on a machine without the card, so `--card p100` chose the recipe. On a P100 host, leave `--card` out: the `gpu:` line then names the GPU that matched.

A dry run still asks `nvidia-smi` for the GPU list, so it shows what would happen on this host. Without a GPU, `--dry-run` and `info` fall back to the model’s only recipe; pass `--card p100` to choose it yourself.

## Check that it worked

[Section titled “Check that it worked”](#check-that-it-worked)

Compare the command with the recipe file named on the `recipe` line, in [recipes/](https://github.com/iammrduncan/tensorless/tree/main/recipes): the image digest, the model’s revision and sha256, the arguments and the switches are the same. When you then run the command without `--dry-run`, the launcher downloads and starts exactly that.

# Run a model in the background, and stop it

> Start a model's server with --detach so it keeps running after the command returns, read its logs, and stop it.

By default, `tensorless run` stays in the foreground: it shows the server’s logs, and Ctrl-C stops the server. With `--detach` it returns as soon as the model has loaded and leaves the server running, so you can close the terminal.

You need the launcher installed and a host that passes `tensorless doctor`. See [Prepare a Linux host for a P100](/docs/how-to/setup/prepare-host/).

## 1. Start the server in the background

[Section titled “1. Start the server in the background”](#1-start-the-server-in-the-background)

```sh
tensorless run iamMrDuncan/Qwen3.8-27B-PN3-GGUF --detach
```

`run` downloads and checks whatever is missing first, as it always does. Then it starts the server’s container, waits for the model to load, and returns. Loading takes a few seconds once the file is in the page cache, and longer the first time. While it waits it prints:

```text
waiting for http://127.0.0.1:8080/health while the model loads (up to 20m)
```

When the server answers, it prints the endpoint and how to stop it:

```text
iamMrDuncan/Qwen3.8-27B-PN3-GGUF is ready at http://127.0.0.1:8080 (OpenAI-compatible API under /v1)
try it:
  curl http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' \
    -d '{"model": "iamMrDuncan/Qwen3.8-27B-PN3-GGUF", "messages": [{"role": "user", "content": "Hello"}]}'
stop it with: tensorless stop iamMrDuncan/Qwen3.8-27B-PN3-GGUF
```

If the image and the model are already pulled, `tensorless serve iamMrDuncan/Qwen3.8-27B-PN3-GGUF --detach` starts the server without pulling or downloading anything. It still checks both, and fails if either is missing.

Note

If the server stops before it is ready, `--detach` prints the last 60 lines of its log and fails. [Fix a failed check or a failed start](/docs/how-to/use/troubleshoot/) says what the usual lines mean.

## 2. Read the logs

[Section titled “2. Read the logs”](#2-read-the-logs)

The server runs in a container named `tensorless-` followed by the model’s repo id in lower case, with `/` replaced by `-`:

| Model                                             | Container                                                    |
| ------------------------------------------------- | ------------------------------------------------------------ |
| `iamMrDuncan/Qwen3.8-27B-PN3-GGUF`                | `tensorless-iammrduncan-qwen3.8-27b-pn3-gguf`                |
| `iamMrDuncan/Qwen3.8-27B-UD-IQ3_XXS-Q3Floor-GGUF` | `tensorless-iammrduncan-qwen3.8-27b-ud-iq3_xxs-q3floor-gguf` |

Follow its log with the container tool that started it, podman or docker:

```sh
podman logs -f tensorless-iammrduncan-qwen3.8-27b-pn3-gguf
# or:
docker logs -f tensorless-iammrduncan-qwen3.8-27b-pn3-gguf
```

Ctrl-C stops following the log. The server keeps running.

## 3. Stop the server

[Section titled “3. Stop the server”](#3-stop-the-server)

```sh
tensorless stop iamMrDuncan/Qwen3.8-27B-PN3-GGUF
```

```text
stopped tensorless-iammrduncan-qwen3.8-27b-pn3-gguf
```

`stop` looks for the container in both podman and docker, whichever are installed, so you don’t need to remember which one started it. The container was started with `--rm`, so stopping it also removes it. The model file and the image stay where they are, and the next `run` starts again in seconds.

## Check that it worked

[Section titled “Check that it worked”](#check-that-it-worked)

While the server runs, it answers on its health address:

```sh
curl -s http://127.0.0.1:8080/health
```

After `tensorless stop`, the same command fails to connect, and `tensorless stop` again says:

```text
tensorless-iammrduncan-qwen3.8-27b-pn3-gguf is not running
```

The server does not start again by itself after a reboot. Run `tensorless serve MODEL --detach` again, or start it from your own service manager.

# Serve on another port or address

> Move the server off port 8080, or make it reachable from other machines through an SSH tunnel, a private network or every interface.

The server listens on `127.0.0.1:8080` by default, so only programs on the same machine can reach it. This page moves it to another port, and shows three ways to reach it from other machines, safest first.

You need the launcher installed and a host that passes `tensorless doctor`. See [Prepare a Linux host for a P100](/docs/how-to/setup/prepare-host/).

Caution

The recipes start `llama-server` with no API key and no login. Anyone who can reach the port can use the model. Prefer an SSH tunnel or a private network to publishing the port on a network you don’t control.

## Use another port

[Section titled “Use another port”](#use-another-port)

Pass `--port` to `run` or `serve`:

```sh
tensorless run iamMrDuncan/Qwen3.8-27B-PN3-GGUF --port 9000
```

The server is then at `http://127.0.0.1:9000`, with the API under `http://127.0.0.1:9000/v1`. Inside the container it still listens on 8080; `--port` changes only the host’s side.

If something else already uses the port, the launcher refuses before it starts anything, with `cannot use port 8080 on 127.0.0.1` and the reason. Choose another port with `--port`.

`tensorless bench` takes the same `--port`, so pass it there too: `tensorless bench pn3 --port 9000`.

## Reach it from another machine through an SSH tunnel

[Section titled “Reach it from another machine through an SSH tunnel”](#reach-it-from-another-machine-through-an-ssh-tunnel)

This needs no change on the GPU host. On the machine you want to use it from, forward a local port to the server:

```sh
ssh -N -L 8080:127.0.0.1:8080 gpu-host-1
```

While that command runs, `http://127.0.0.1:8080/v1` on your machine reaches the server on `gpu-host-1`. Only people who can log in to `gpu-host-1` over SSH can use it.

## Serve on a private network

[Section titled “Serve on a private network”](#serve-on-a-private-network)

To publish the server on one address of the host only, give that address to `--host`. With [Tailscale](https://tailscale.com), for example, use the host’s Tailscale address:

```sh
tensorless run iamMrDuncan/Qwen3.8-27B-PN3-GGUF --detach --host "$(tailscale ip -4)"
```

The port is published on that address only, so devices on your tailnet reach the server at that address and port 8080, and other networks don’t. The launcher waits for the server and prints the endpoint at the same address.

## Serve on every interface

[Section titled “Serve on every interface”](#serve-on-every-interface)

`--host 0.0.0.0` publishes the port on every IPv4 address of the host (`--host ::` for IPv6):

```sh
tensorless run iamMrDuncan/Qwen3.8-27B-PN3-GGUF --detach --host 0.0.0.0
```

When it is ready, the launcher prints the local endpoint and a reminder:

```text
iamMrDuncan/Qwen3.8-27B-PN3-GGUF is ready at http://127.0.0.1:8080 (OpenAI-compatible API under /v1)
it listens on every interface of this host (--host 0.0.0.0)
```

Use this only on a network you trust, or behind a firewall that limits who can connect.

Note

`llama-server` can check an API key, with its `--api-key` option, which you can pass after `--`: `tensorless run pn3 -- --api-key "$KEY"`. The launcher labels any extra argument as untuned, because it cannot tell which arguments change the measured configuration (see [Change the context length or server arguments](/docs/how-to/run/change-context-or-args/)). `tensorless bench` sends no key, so it cannot measure a server started with one.

## Check that it worked

[Section titled “Check that it worked”](#check-that-it-worked)

From the other machine, ask the server which model it serves, at the address you chose:

```sh
curl -s http://127.0.0.1:8080/v1/models              # through the SSH tunnel
curl -s http://gpu-host-1:8080/v1/models             # --host 0.0.0.0, on your network
```

The answer lists the model’s repo id. Then point your client at that address: [Connect an OpenAI-compatible client](/docs/how-to/use/openai-clients/).

# Switch to another model

> Stop the model that holds the P100, then run another launch target, by its repo id or its short name.

A Tesla P100 has 16 GB, and one launch target with its 65,536-token context takes nearly all of it. To use another model, stop the one that is running, then run the other. This page switches from Qwen3.8-27B-PN3 to Qwen3.8-27B-UD-IQ3_XXS-Q3Floor (Q3Floor); the steps are the same in the other direction.

You need the launcher installed and a host that passes `tensorless doctor`. See [Prepare a Linux host for a P100](/docs/how-to/setup/prepare-host/).

## 1. See which models your launcher knows

[Section titled “1. See which models your launcher knows”](#1-see-which-models-your-launcher-knows)

```sh
tensorless models
```

```text
MODEL                                            CARD  STATUS    ENGINE         FITS  CACHED
iamMrDuncan/Qwen3.8-27B-PN3-GGUF                 p100  released  engine-v0.1.0  yes   yes
iamMrDuncan/Qwen3.8-27B-UD-IQ3_XXS-Q3Floor-GGUF  p100  released  engine-v0.1.0  yes   no
```

* **FITS** says whether a GPU on this host matches the recipe (`yes`, `no`, or `unknown` without a GPU).
* **CACHED** says whether the model file is already downloaded: `yes`, `partial` (an interrupted download) or `no`.

The launch targets, with their sizes and measured speeds, are in [Launch targets](/docs/reference/launch-targets/).

## 2. Stop the running model

[Section titled “2. Stop the running model”](#2-stop-the-running-model)

```sh
tensorless stop iamMrDuncan/Qwen3.8-27B-PN3-GGUF
```

If you started it in the foreground, Ctrl-C in its terminal does the same.

Caution

Don’t start a second model while the first holds the card. On the same port, the launcher refuses with `cannot use port 8080`. On another port, the second server runs out of GPU memory while it loads. On a host with several GPUs, you can run one model per card instead: see [Pick a GPU on a host with several](/docs/how-to/run/pick-a-gpu/).

## 3. Run the other model

[Section titled “3. Run the other model”](#3-run-the-other-model)

```sh
tensorless run iamMrDuncan/Qwen3.8-27B-UD-IQ3_XXS-Q3Floor-GGUF
```

The first time, this downloads the 11.76 GB model file and checks its sha256. The engine image is the same one both recipes use, so it is not pulled again.

Each recipe also defines a short name, which works anywhere a repo id does:

| Short name | Repo id                                           |
| ---------- | ------------------------------------------------- |
| `pn3`      | `iamMrDuncan/Qwen3.8-27B-PN3-GGUF`                |
| `q3floor`  | `iamMrDuncan/Qwen3.8-27B-UD-IQ3_XXS-Q3Floor-GGUF` |

```sh
tensorless stop pn3
tensorless run q3floor --detach
```

The repo id is not case-sensitive, as on the Hugging Face Hub.

## Check that it worked

[Section titled “Check that it worked”](#check-that-it-worked)

Ask the server which model it serves:

```sh
curl -s http://127.0.0.1:8080/v1/models
```

The answer lists `iamMrDuncan/Qwen3.8-27B-UD-IQ3_XXS-Q3Floor-GGUF`. Clients that name the model in their requests need the new repo id: see [Connect an OpenAI-compatible client](/docs/how-to/use/openai-clients/).

# Build the launcher from source

> Build the tensorless command with Go, for a platform without a release archive or to try a change, with the version stamped in.

Build the launcher yourself when your platform has no release archive, or to try a change to the launcher or its recipes. The launcher is plain Go with one dependency, and builds into a static binary.

## What you need

[Section titled “What you need”](#what-you-need)

* Go 1.23 or newer.
* `git`, to build from a checkout.

## Install with `go install`

[Section titled “Install with go install”](#install-with-go-install)

```sh
go install github.com/iammrduncan/tensorless/cmd/tensorless@v0.1.0
```

Go puts the binary in `$(go env GOPATH)/bin`, or in `$GOBIN` if you set it. `@latest` installs the newest release, and `@main` the newest commit of `main`.

A binary built this way has no version stamped in:

```sh
tensorless version
```

```text
tensorless dev, linux/amd64
```

## Build from a checkout

[Section titled “Build from a checkout”](#build-from-a-checkout)

1. Get the source:

   ```sh
   git clone https://github.com/iammrduncan/tensorless
   cd tensorless
   ```

2. Build it, with the version and commit stamped in:

   ```sh
   go build -ldflags "-X main.version=$(git describe --tags --always --match 'v*') -X main.commit=$(git rev-parse --short HEAD)" ./cmd/tensorless
   ```

   `--match 'v*'` names the launcher’s own tags (`v0.1.0`). The repository also has engine tags (`engine-v0.1.0`), which `git describe` would otherwise pick.

3. Check it:

   ```sh
   ./tensorless version
   ```

   At the tag `v0.1.0` this prints `tensorless v0.1.0 (commit` and the short commit. After the tag, `git describe` adds the number of commits since it, such as `v0.1.0-3-g147bf17`.

To build the way the release does, static and with the full commit, for any platform Go supports:

```sh
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath \
  -ldflags "-s -w -X main.version=0.1.0 -X main.commit=$(git rev-parse HEAD)" \
  -o tensorless ./cmd/tensorless
```

## Recipes are built in

[Section titled “Recipes are built in”](#recipes-are-built-in)

The launcher embeds the `recipes/` directory when it is built ([recipes_embed.go](https://github.com/iammrduncan/tensorless/blob/main/recipes_embed.go)). A recipe you add or change in your checkout is part of the binary you build next.

To try recipes without rebuilding, point the launcher at a directory with `--recipes DIR`. See [Try a recipe of your own](/docs/how-to/advanced/write-a-recipe/).

## Run the checks

[Section titled “Run the checks”](#run-the-checks)

The project’s CI runs these on every change. Run them before you send a pull request:

```sh
gofmt -l cmd internal *.go
go vet ./...
go test ./...
```

`gofmt -l` prints nothing when every file is formatted.

## How to tell it worked

[Section titled “How to tell it worked”](#how-to-tell-it-worked)

`./tensorless version` prints your version and commit, and `./tensorless models` lists the recipes from your checkout’s `recipes/` directory.

# Install the launcher

> Install the tensorless command on Linux or macOS, with the install script or from a release archive, checked by sha256.

The launcher, `tensorless`, is one static binary with no runtime dependencies. Install it with the install script, or download a release archive and check it yourself.

```sh
curl -fsSL https://tensorless.dev/install.sh | sh
```

## Where it runs

[Section titled “Where it runs”](#where-it-runs)

Each release has an archive for three platforms:

| Platform             | Archive         | What it can do                                                                                                                                 |
| -------------------- | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| Linux x86_64         | `linux-x86_64`  | Everything: pull, serve and benchmark models.                                                                                                  |
| Linux aarch64        | `linux-aarch64` | The launcher runs, but the 0.1.0 engine image is published for `linux/amd64` only, so it cannot serve a model on an Arm host.                  |
| macOS, Apple silicon | `macos-arm64`   | Read recipes and see exactly what would run: `models`, `info`, `capabilities` and `--dry-run`. A Mac has no NVIDIA GPU, so it serves no model. |

Serving a model also needs an NVIDIA driver, podman or docker, and NVIDIA container support. [Prepare a Linux host for a P100](/docs/how-to/setup/prepare-host/) sets those up.

## Install with the script

[Section titled “Install with the script”](#install-with-the-script)

You need `curl` or `wget`, and `sha256sum` or `shasum`.

1. Run the script:

   ```sh
   curl -fsSL https://tensorless.dev/install.sh | sh
   ```

   It finds your platform, downloads that archive and its `.sha256` file from the [GitHub release](https://github.com/iammrduncan/tensorless/releases), checks the archive’s sha256, and copies the one `tensorless` binary to `~/.local/bin`. It changes nothing else. You see:

   ```text
   Downloading tensorless 0.1.0 for linux-x86_64
   Checked sha256 cb0c796fd83a334eb67f163fffb30016f51513f8f6ca4a30e7144875487415eb
   Installed /home/you/.local/bin/tensorless
   ```

2. If the directory is not on your `PATH`, the script says so and prints the line to add, for example:

   ```text
   /home/you/.local/bin is not on your PATH. Add it, for example:
     export PATH="/home/you/.local/bin:$PATH"
   ```

   Run that line, and add it to your shell’s startup file (such as `~/.bashrc`) to keep it.

Two environment variables change what the script does:

| Variable                 | Default            | Meaning                                                            |
| ------------------------ | ------------------ | ------------------------------------------------------------------ |
| `TENSORLESS_VERSION`     | `0.1.0`            | The release to install.                                            |
| `TENSORLESS_INSTALL_DIR` | `$HOME/.local/bin` | Where the binary goes. The script creates the directory if needed. |

Set them for the `sh` that runs the script:

```sh
curl -fsSL https://tensorless.dev/install.sh | TENSORLESS_INSTALL_DIR="$HOME/bin" sh
```

Tip

To read the script before it runs, download it first: `curl -fsSL https://tensorless.dev/install.sh -o install.sh`, read `install.sh`, then run `sh install.sh`.

On any other platform, the script stops with `no release archive for` your system, and suggests building from source. See [Build the launcher from source](/docs/how-to/setup/build-launcher/).

## Install from a release archive

[Section titled “Install from a release archive”](#install-from-a-release-archive)

1. Download the archive for your platform and its `.sha256` file. Set `P` to `linux-x86_64`, `linux-aarch64` or `macos-arm64`:

   ```sh
   V=0.1.0
   P=linux-x86_64
   curl -fLO https://github.com/iammrduncan/tensorless/releases/download/v$V/tensorless-$V-$P.tar.gz
   curl -fLO https://github.com/iammrduncan/tensorless/releases/download/v$V/tensorless-$V-$P.tar.gz.sha256
   ```

2. Check the archive, then unpack it. On Linux:

   ```sh
   sha256sum -c tensorless-$V-$P.tar.gz.sha256 && tar -xzf tensorless-$V-$P.tar.gz
   ```

   On macOS, use `shasum -a 256 -c` in place of `sha256sum -c`. The check prints the archive’s name followed by `OK`.

3. The archive’s folder, `tensorless-0.1.0-linux-x86_64/`, holds the `tensorless` binary with `LICENSE`, `LICENSES/`, `NOTICE`, `THIRD_PARTY_NOTICES.md` and `README.md`. Put the binary on your `PATH`:

   ```sh
   mkdir -p ~/.local/bin
   install -m 755 tensorless-$V-$P/tensorless ~/.local/bin/
   ```

## Check that it worked

[Section titled “Check that it worked”](#check-that-it-worked)

```sh
tensorless version
```

```text
tensorless 0.1.0 (commit d104660be24427080948eafb0522265e1450d642), linux/amd64
```

`tensorless models` lists the launch targets this launcher knows. On a machine without an NVIDIA GPU, it cannot tell whether they fit:

```text
MODEL                                            CARD  STATUS    ENGINE         FITS     CACHED
iamMrDuncan/Qwen3.8-27B-PN3-GGUF                 p100  released  engine-v0.1.0  unknown  no
iamMrDuncan/Qwen3.8-27B-UD-IQ3_XXS-Q3Floor-GGUF  p100  released  engine-v0.1.0  unknown  no
fits is unknown: nvidia-smi not found
```

Next: [Prepare a Linux host for a P100](/docs/how-to/setup/prepare-host/), or go straight to [Run your first model on a P100](/docs/tutorials/first-model/) if the host is ready.

# Prepare a Linux host for a P100

> Install a driver that supports Pascal, give podman or docker the GPU, make room on the disk and the card, and get every tensorless doctor check to pass.

tensorless runs its engine in a container. So the host needs an NVIDIA driver that supports the P100, a container runtime (podman or docker), and NVIDIA container support, so that the runtime can hand the GPU to the container. The host does not need CUDA: the CUDA runtime is inside the engine image.

When every check of `tensorless doctor` passes, the host is ready.

## What you need

[Section titled “What you need”](#what-you-need)

* A Linux x86_64 machine with an NVIDIA Tesla P100 **16 GB** installed. The 12 GB P100 does not fit the recipes.
* An x86-64 CPU with AVX2, FMA and F16C. The engine image’s CPU code is built for that baseline, and some older Xeons that hosted P100s lack AVX2.
* `sudo` on that machine.
* The launcher: see [Install the launcher](/docs/how-to/setup/install-launcher/).

## 1. Install the NVIDIA driver

[Section titled “1. Install the NVIDIA driver”](#1-install-the-nvidia-driver)

The P100 recipes need a driver from **525.60.13** up to the **R580** branch:

* **525.60.13** is the oldest driver that runs programs built with CUDA 12. The engine image is built with CUDA 12.6, because CUDA 13 dropped Pascal cards.
* **R580** is the last driver branch that supports Pascal. Newer branches do not drive the P100.

Install the driver from your distribution’s packages, or from [NVIDIA’s driver downloads](https://www.nvidia.com/en-us/drivers/), then reboot.

Check it with a query like the one the launcher uses:

```sh
nvidia-smi --query-gpu=index,name,compute_cap,memory.total,driver_version --format=csv
```

The P100 shows compute capability `6.0` and about 16 GB of memory, with the driver version in the last column.

## 2. Install a container runtime

[Section titled “2. Install a container runtime”](#2-install-a-container-runtime)

Install podman or docker. If both are installed, tensorless uses podman, unless you pass `--runtime docker`.

```sh
sudo apt install podman   # Debian, Ubuntu
sudo dnf install podman   # Fedora, RHEL
```

If you use docker, your user must be able to run it without `sudo`, usually by being in the `docker` group.

## 3. Install the NVIDIA Container Toolkit

[Section titled “3. Install the NVIDIA Container Toolkit”](#3-install-the-nvidia-container-toolkit)

The toolkit connects the container runtime to the driver, and provides the `nvidia-ctk` command. Install it with [NVIDIA’s installation guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) for your distribution.

## 4. Give the runtime the GPU

[Section titled “4. Give the runtime the GPU”](#4-give-the-runtime-the-gpu)

**With podman,** generate a CDI spec, the file that tells podman which device files and driver libraries to give a container:

```sh
sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
```

The spec names the driver’s files by version. **Generate it again after every driver upgrade,** or containers fail to start.

**With docker,** register the NVIDIA runtime and restart docker:

```sh
sudo nvidia-ctk runtime configure --runtime=docker && sudo systemctl restart docker
```

## 5. Make room on the disk

[Section titled “5. Make room on the disk”](#5-make-room-on-the-disk)

The model goes into the Hugging Face cache, `~/.cache/huggingface/hub` unless you choose another place:

| Model                          | File size |
| ------------------------------ | --------: |
| Qwen3.8-27B-PN3                |  12.95 GB |
| Qwen3.8-27B-UD-IQ3_XXS-Q3Floor |  11.76 GB |

The engine image goes into the container runtime’s own storage. It is about 1.6 GB to download, and takes a few GB once unpacked.

To keep models on a larger disk, see [Keep models on another disk](/docs/how-to/run/model-cache/).

## 6. Free the GPU

[Section titled “6. Free the GPU”](#6-free-the-gpu)

A model and its 65,536-token context take about 14 GB of the card’s 16 GB: the release measured 13,997 MiB with the server idle and 14,577 MiB while it generated. Anything else on the card can make the server fail to start, or run slower.

`nvidia-smi` lists the processes that use the card, at the bottom of its output. Stop them before you start a model.

## 7. Check the host

[Section titled “7. Check the host”](#7-check-the-host)

```sh
tensorless doctor
```

`doctor` checks, in order:

1. that `nvidia-smi` is present and lists the GPUs;
2. that a GPU matches a recipe: compute capability 6.0 and at least 16,000 MiB;
3. that the driver is at least the recipe’s `min_driver`, 525.60.13;
4. that podman or docker is installed;
5. for podman, that a CDI spec exists at `/etc/cdi/nvidia.yaml` or `/var/run/cdi/nvidia.yaml`; for docker, that it has the nvidia runtime, a CDI spec or the toolkit’s hook;
6. that the model cache has room for the largest model that is not downloaded yet.

Each check prints `PASS`, `WARN` or `FAIL`. A `FAIL` comes with the command that fixes it:

```text
FAIL  no NVIDIA CDI spec for podman in /etc/cdi/nvidia.yaml or /var/run/cdi/nvidia.yaml
      fix: install nvidia-container-toolkit, then: sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
```

A `WARN` does not fail the check. For example, a second GPU that fits no recipe gets a `WARN`, and tensorless uses the card that fits.

## How to tell it worked

[Section titled “How to tell it worked”](#how-to-tell-it-worked)

`tensorless doctor` ends with:

```text
all checks passed.
```

and exits with status 0. If any check fails, it ends with the number of failed checks and exits with status 1. [Fix a failed check or start](/docs/how-to/use/troubleshoot/) covers each failure.

Next: [Run your first model on a P100](/docs/tutorials/first-model/).

# Update tensorless

> Move to a newer launcher release, and know what an update changes: the recipes the launcher knows, and never a model's engine without a new receipt.

Update the launcher to get new launch targets and recipes that moved to a newer engine release. An update replaces one binary. Running servers, downloaded models and pulled images are not touched.

## What you need

[Section titled “What you need”](#what-you-need)

* The version you have: `tensorless version`.
* The version you want, from the [releases page](https://github.com/iammrduncan/tensorless/releases).

## 1. Install the new launcher

[Section titled “1. Install the new launcher”](#1-install-the-new-launcher)

Run the install script again, with the release you want:

```sh
curl -fsSL https://tensorless.dev/install.sh | TENSORLESS_VERSION=0.1.0 sh
```

Replace `0.1.0` with the version you want. Without `TENSORLESS_VERSION`, the script installs the release it names as its default. If you installed somewhere other than `~/.local/bin`, set `TENSORLESS_INSTALL_DIR` to the same place, so that the new binary replaces the old one.

If you installed from an archive, download and check the new archive the same way, and copy its binary over the old one. See [Install the launcher](/docs/how-to/setup/install-launcher/#install-from-a-release-archive).

To go back to an older release, do the same with the older version.

## 2. See what the new launcher knows

[Section titled “2. See what the new launcher knows”](#2-see-what-the-new-launcher-knows)

```sh
tensorless models
```

The list shows every launch target in this launcher, with its engine release.

## 3. Restart a running model

[Section titled “3. Restart a running model”](#3-restart-a-running-model)

A server that was running before the update keeps running on what it started with. To move it to the new launcher’s recipe, stop it and run it again:

```sh
tensorless stop iamMrDuncan/Qwen3.8-27B-PN3-GGUF
tensorless run iamMrDuncan/Qwen3.8-27B-PN3-GGUF
```

## What an update changes

[Section titled “What an update changes”](#what-an-update-changes)

* **The recipes.** Recipes are built into the launcher, so a new launch target, or a recipe that moved to a new engine release, comes with a new launcher.
* **The engine, only when a recipe moves.** A recipe pins its engine image by digest. A model’s engine changes only when its recipe moves to a new engine release, and a recipe moves only with a new receipt for that model on that card. If it moved, the next `run` or `pull` pulls the new image. Older images stay in the container runtime’s storage until you remove them, with `podman rmi` or `docker rmi`.
* **The model, only when a recipe pins a new file.** Models stay in the Hugging Face cache. If the recipe’s revision and sha256 did not change, nothing is downloaded again. If they changed, the new file is downloaded next to the old one, which stays until you delete it.

[How a release is made](/docs/explanation/releases-and-receipts/) explains why recipes only move with a receipt.

## How to tell it worked

[Section titled “How to tell it worked”](#how-to-tell-it-worked)

`tensorless version` prints the new version.

# Benchmark a running server

> Measure a server's decode speed with tensorless bench, the way the receipts were measured, and compare it with the recipe's receipt.

`tensorless bench` measures how fast a running server generates text, and compares the result with the number the recipe’s receipt promises. Use it to check that your card performs as measured, or to compare settings.

You need a host that passes `tensorless doctor` ([Prepare a Linux host for a P100](/docs/how-to/setup/prepare-host/)) and nothing else using the GPU. For a guided first run, follow [Measure your card against the receipt](/docs/tutorials/measure-your-card/).

## 1. Start the server in the background

[Section titled “1. Start the server in the background”](#1-start-the-server-in-the-background)

```sh
tensorless serve iamMrDuncan/Qwen3.8-27B-PN3-GGUF --detach
```

`bench` does not start a server: it measures the one that answers on `--host` and `--port`, by default `127.0.0.1:8080`. Without one, it stops with:

```text
tensorless: no server answers http://127.0.0.1:8080/health. Start one with: tensorless serve iamMrDuncan/Qwen3.8-27B-PN3-GGUF --detach
```

## 2. Run the benchmark

[Section titled “2. Run the benchmark”](#2-run-the-benchmark)

```sh
tensorless bench iamMrDuncan/Qwen3.8-27B-PN3-GGUF
```

It sends one short warm-up request, then three measured ones, and prints each run as it ends. Then it prints the medians:

```text
benchmarking iamMrDuncan/Qwen3.8-27B-PN3-GGUF at http://127.0.0.1:8080: warm-up, then 3 runs of 256 tokens
run 1: <decode> tok/s decode, TTFT <ttft> ms
run 2: ...
run 3: ...
iamMrDuncan/Qwen3.8-27B-PN3-GGUF on p100, 3 runs of 256 tokens (median):
  decode         <decode> tok/s
  TTFT           <ttft> ms
  prompt tokens  <prompt>
  receipt        30.06 tok/s decode (short); this run is <difference>%
The receipt was measured with a different prompt, so expect a few percent of difference.
```

On the P100 that measured the release, the published image gave a median of 30.03 tok/s for this model, with a 429-token prompt: −0.1% against the receipt. Qwen3.8-27B-UD-IQ3_XXS-Q3Floor gave 26.18 tok/s against its receipt of 26.26. The [receipts](/docs/reference/receipts/) have the details.

## What it measures

[Section titled “What it measures”](#what-it-measures)

`bench` mirrors the hill climb’s judge, which timed every change to the engine:

1. **Warm-up:** one request for 8 tokens, not counted.
2. **Measured runs:** `--runs` requests (3 by default) to `/completion`, each generating `--tokens` tokens (256 by default). They are greedy (temperature 0, top_k 1, seed 424242), ignore the end-of-text token so every run generates the full count, and don’t reuse the prompt cache.
3. **Timing:** every streamed token is time-stamped as it arrives. **Decode** is (tokens − 1) divided by the time from the first token to the last. **TTFT** (time to first token) is the time from sending the request to the first token, so it includes reading the prompt.

The prompt is built in: the first 2,000 characters of the Apache-2.0 license, followed by “Continue the text above.” The result is the median of the runs.

## Options

[Section titled “Options”](#options)

| Flag               | Default             | Use it to                                 |
| ------------------ | ------------------- | ----------------------------------------- |
| `--runs N`         | 3                   | take more runs, for a steadier median     |
| `--tokens N`       | 256                 | generate more tokens per run (at least 2) |
| `--json`           | off                 | print the result as JSON                  |
| `--host`, `--port` | `127.0.0.1`, `8080` | reach a server on another address or port |

```sh
tensorless bench pn3 --runs 5 --tokens 256 --json
```

The JSON has the `model`, the `card` and the `endpoint`; each run’s `decode_tps`, `ttft_ms`, `prompt_tokens` and `tokens`; the `median_decode_tps`, `median_ttft_ms` and `prompt_tokens`; and, when the recipe has a receipt, `receipt_decode_tps` and `vs_receipt_percent`.

## Benchmark a server on another machine

[Section titled “Benchmark a server on another machine”](#benchmark-a-server-on-another-machine)

`bench` can run on your laptop against the GPU host. Decode speed is timed between tokens, so the network does not change it; TTFT includes the network’s delay. Forward the port over SSH:

```sh
ssh -N -L 8080:127.0.0.1:8080 gpu-host-1
```

Then, in another terminal on your laptop:

```sh
tensorless bench iamMrDuncan/Qwen3.8-27B-PN3-GGUF
```

`bench` needs no GPU on the machine it runs on: it only reads the recipe to know which receipt to compare with. On a machine that has another NVIDIA GPU, add `--card p100`, or the launcher looks for a P100 there and finds none.

## Read the result

[Section titled “Read the result”](#read-the-result)

* **Within a few percent of the receipt:** your card and host perform as measured. The receipt used a different prompt (about 600 tokens), so a small difference is expected.
* **Much slower:** check that nothing else uses the GPU (`nvidia-smi`), that the server was started by tensorless with no `--context` or extra arguments, and that the card is not held back by heat or a low power limit (`nvidia-smi -q -d PERFORMANCE` lists the reasons a card is slowed).
* **A warning that the server serves another model:** `bench` asks the server which model it runs, and warns when it is not the one you named. The comparison with the receipt then does not apply.

If you started the server with another `--context`, pass the same `--context` to `bench`. It then notes that the run is untuned, and that the receipt’s number is only a rough guide.

## Check that it worked

[Section titled “Check that it worked”](#check-that-it-worked)

The `receipt` line compares your median with the receipt, or the output says the recipe has no decode receipt to compare with. Stop the server when you are done:

```sh
tensorless stop iamMrDuncan/Qwen3.8-27B-PN3-GGUF
```

To share what you measured, see [Share your results](/docs/how-to/use/share-results/).

# Connect an OpenAI-compatible client

> Point curl, the Python openai library or any client that takes a base URL at the server tensorless starts.

The server tensorless starts is `llama-server`, which speaks the OpenAI API. Any client that lets you set a base URL can use it. This page connects curl and the Python `openai` library, and lists what any other client needs.

You need a model running: `tensorless run iamMrDuncan/Qwen3.8-27B-PN3-GGUF`, or the [first tutorial](/docs/tutorials/first-model/).

## What every client needs

[Section titled “What every client needs”](#what-every-client-needs)

| Setting  | Value                                                                                                  |
| -------- | ------------------------------------------------------------------------------------------------------ |
| Base URL | `http://127.0.0.1:8080/v1` (another address or port if you passed `--host` or `--port`)                |
| Model    | the repo id, `iamMrDuncan/Qwen3.8-27B-PN3-GGUF` (or `iamMrDuncan/Qwen3.8-27B-UD-IQ3_XXS-Q3Floor-GGUF`) |
| API key  | none: the recipes start the server without one. A client that insists on a key can send any text.      |

The model name is the recipe’s alias, which is the repo id. `GET /v1/models` lists it:

```sh
curl -s http://127.0.0.1:8080/v1/models
```

## curl

[Section titled “curl”](#curl)

```sh
curl -s http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"iamMrDuncan/Qwen3.8-27B-PN3-GGUF","messages":[{"role":"user","content":"Say hello in one sentence."}],"max_tokens":64}'
```

The answer is the usual chat completion JSON; the text is in `choices[0].message.content`. The server applies the model’s own chat template to `/v1/chat/completions`, because the recipes pass `--jinja`.

## Python

[Section titled “Python”](#python)

Install the `openai` library, then point it at the server:

```sh
pip install openai
```

```python
from openai import OpenAI


client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="none")


reply = client.chat.completions.create(
    model="iamMrDuncan/Qwen3.8-27B-PN3-GGUF",
    messages=[{"role": "user", "content": "Say hello in one sentence."}],
    max_tokens=64,
)
print(reply.choices[0].message.content)
```

The library needs some `api_key`; the server does not check it.

To print the answer as it is generated, ask for a stream:

```python
stream = client.chat.completions.create(
    model="iamMrDuncan/Qwen3.8-27B-PN3-GGUF",
    messages=[{"role": "user", "content": "Write a haiku about an old GPU."}],
    stream=True,
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
print()
```

## Sampling

[Section titled “Sampling”](#sampling)

Qwen3.8-27B-PN3’s [model card](https://huggingface.co/iamMrDuncan/Qwen3.8-27B-PN3-GGUF) recommends temperature 1.0, top_p 0.95 and top_k 20. Send them with each request; that leaves the run as it was measured. `top_k` is not part of the OpenAI API, so the Python library sends it in `extra_body`:

```python
reply = client.chat.completions.create(
    model="iamMrDuncan/Qwen3.8-27B-PN3-GGUF",
    messages=[{"role": "user", "content": "Name three uses of a 16 GB GPU."}],
    temperature=1.0,
    top_p=0.95,
    extra_body={"top_k": 20},
)
```

## Other clients

[Section titled “Other clients”](#other-clients)

Chat front ends, editors and agent tools that support “OpenAI-compatible” servers need the same three settings: the base URL, the model name and an empty or dummy key. Their names for these vary, often *API base*, *endpoint* or *base URL*.

* **A client on another machine** needs to reach the server first. See [Serve on another port or address](/docs/how-to/run/serve-on-your-network/).
* **A client in a container** on the same host cannot reach `127.0.0.1` of the host. Use the host’s address, and start the server with `--host` on an address that container can reach.

Note

The recipes run the server with one slot (`-np 1`): it works on one request at a time, and others wait their turn. Each request can use the whole 65,536-token context.

## Check that it worked

[Section titled “Check that it worked”](#check-that-it-worked)

The curl command above returns a JSON answer with the model’s text in `choices[0].message.content`, and the server’s log (in the terminal running `tensorless run`, or `podman logs`) shows the request. If the client says it cannot connect, check the server with `curl -s http://127.0.0.1:8080/health` on the GPU host, then the address and port the client uses.

# Share your results

> Collect a benchmark, the launcher's version and your card's details, and post them on Discord or in a GitHub issue.

Numbers from other people’s P100s show whether a recipe holds up on cards, drivers and hosts it was not measured on. This page collects what makes a result useful, and says where to post it.

You need a model running and benchmarked: see [Benchmark a running server](/docs/how-to/use/benchmark/).

## 1. Benchmark as JSON

[Section titled “1. Benchmark as JSON”](#1-benchmark-as-json)

Run the benchmark with its defaults, so your numbers compare with everyone else’s, and save the JSON:

```sh
tensorless bench iamMrDuncan/Qwen3.8-27B-PN3-GGUF --json > bench.json
```

It holds every run, the medians, and the difference from the recipe’s receipt.

## 2. Record the launcher and the card

[Section titled “2. Record the launcher and the card”](#2-record-the-launcher-and-the-card)

```sh
tensorless version
nvidia-smi --query-gpu=name,memory.total,driver_version,power.limit --format=csv
```

`tensorless version` names the release and commit. The `nvidia-smi` line gives the card’s exact model, its memory, the driver version and its power limit, which can all change the speed. `tensorless capabilities --json` adds the recipes your launcher knows, if someone asks.

## 3. Note how you ran it

[Section titled “3. Note how you ran it”](#3-note-how-you-ran-it)

Say, in a sentence or two:

* whether you used podman or docker;
* whether you passed `--context` or arguments after `--` (an untuned run measures something else);
* whether anything else used the GPU while it ran;
* anything unusual about the host: a PCIe riser, a card in an x4 slot, a cooling mod.

The JSON’s `endpoint` shows the address you benchmarked; check that you are happy to share it.

## 4. Post it

[Section titled “4. Post it”](#4-post-it)

* **Discord:** [join the tensorless server](https://discord.gg/3Qs2uejUf9) and post your results with the files above. It is the place for questions, results and ideas.
* **GitHub:** if your numbers are far from the receipt, or something failed, [open an issue](https://github.com/iammrduncan/tensorless/issues) with the same details, so it can be traced.

## Check that it worked

[Section titled “Check that it worked”](#check-that-it-worked)

Your post includes `bench.json` (or its median and `vs_receipt_percent`), the output of `tensorless version`, the `nvidia-smi` line, and how you ran it. That is enough for someone else to repeat your measurement.

# Fix a failed check or a failed start

> Read tensorless doctor's checks and the launcher's error messages, and do what each one asks.

When something is missing on the host, or a server does not start, the launcher says what failed and, almost always, what to run next. This page lists those checks and messages, and what to do about each.

## 1. Run the doctor

[Section titled “1. Run the doctor”](#1-run-the-doctor)

```sh
tensorless doctor
```

It checks the host in order and prints `PASS`, `WARN` or `FAIL` for each item, with a `fix:` line under each failure. On a machine without an NVIDIA driver and without the container toolkit, for example:

```text
FAIL  nvidia-smi not found: no NVIDIA driver is installed
      fix: install the NVIDIA driver, 525.60.13 or newer, and reboot
PASS  container runtime: podman (/usr/bin/podman)
FAIL  no NVIDIA CDI spec for podman in /etc/cdi/nvidia.yaml or /var/run/cdi/nvidia.yaml
      fix: install nvidia-container-toolkit, then: sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
PASS  model cache /home/you/.cache/huggingface/hub: 231.19 GB free, the model needs 12.95 GB


2 checks failed.
```

It exits with status 1 when a check fails, and ends with `all checks passed.` when none does. Do the fixes in order, then run it again.

## 2. Fix what the doctor found

[Section titled “2. Fix what the doctor found”](#2-fix-what-the-doctor-found)

| Check                                                                      | If it fails                                                                                                                                                                                                                    |
| -------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `nvidia-smi` is present and lists the GPUs                                 | Install the NVIDIA driver, 525.60.13 or newer, and reboot. If `nvidia-smi` is installed but fails, the driver is not loaded: reboot, then run `nvidia-smi`. For Pascal cards such as the P100, R580 is the last driver branch. |
| A GPU fits a recipe                                                        | The card’s compute capability or memory matches no recipe. Every recipe today needs a 16 GB Tesla P100 (compute capability 6.0); a 12 GB P100 does not fit. `WARN ... fits no recipe` names a GPU that is skipped.             |
| The driver is new enough                                                   | Install the driver the line names, or newer, and reboot.                                                                                                                                                                       |
| podman or docker is installed                                              | `sudo apt install podman` (Debian, Ubuntu) or `sudo dnf install podman` (Fedora, RHEL), or install docker.                                                                                                                     |
| podman: a CDI spec at `/etc/cdi/nvidia.yaml` or `/var/run/cdi/nvidia.yaml` | Install `nvidia-container-toolkit`, then `sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml`. Generate it again after every driver upgrade.                                                                           |
| docker: the nvidia runtime, a CDI spec or the toolkit’s hook               | Install `nvidia-container-toolkit`, then `sudo nvidia-ctk runtime configure --runtime=docker && sudo systemctl restart docker`. If `docker info failed`, start docker and make sure your user may use it (the `docker` group). |
| Free space in the model cache                                              | Free some space, or move the cache to a larger disk: [Keep models on another disk](/docs/how-to/run/model-cache/).                                                                                                             |

The full setup is in [Prepare a Linux host for a P100](/docs/how-to/setup/prepare-host/).

## 3. Fix a failed command

[Section titled “3. Fix a failed command”](#3-fix-a-failed-command)

Find the start of the message, then do what the right-hand column says. Each message starts with `tensorless:`.

| Message                                                           | What to do                                                                                                                                                            |
| ----------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `cannot detect the GPU: nvidia-smi not found`                     | Install the driver (step 2). To only look at a recipe on a machine without the card, pass `--card p100`.                                                              |
| `no recipe for ... fits this host's GPUs`                         | No GPU here matches the model’s recipe. The message lists your GPUs and what the recipe needs.                                                                        |
| `the NVIDIA driver ... is older than ...`                         | Install the driver version it names, or newer, then run `tensorless doctor`.                                                                                          |
| `there is no GPU N`                                               | `--gpu` named an index `nvidia-smi` does not list. Use one of the indexes it prints.                                                                                  |
| `no recipe for "..."`                                             | The launcher does not know that model. It lists the models it knows; so does `tensorless models`. A model added after your launcher’s release needs a newer launcher. |
| `... is a draft: it has not passed the release gate`              | The recipe has not been measured for release yet. Pass `--allow-draft` to try it anyway.                                                                              |
| `... is not published yet`                                        | The recipe’s model upload or engine image does not exist yet. Wait for a release.                                                                                     |
| `revision ... is not the one the recipe ... pins`                 | You gave `MODEL@REVISION`. Pass `--allow-draft` to use it anyway; the run is then untuned.                                                                            |
| `neither podman nor docker is installed`                          | Install podman or docker and the NVIDIA container toolkit (step 2).                                                                                                   |
| `could not pull the engine image`                                 | Check that this host can reach `ghcr.io`, then run the command again.                                                                                                 |
| `HTTP 401` or `HTTP 403` for a Hugging Face address               | The repository is private or gated. Accept its terms on the Hub, and set `HF_TOKEN`.                                                                                  |
| `HTTP 404` for a Hugging Face address                             | The recipe’s repository, revision or file does not exist on the Hub. Please report it.                                                                                |
| `not enough disk space in ...`                                    | Free some space, or pass `--cache DIR` on a larger disk.                                                                                                              |
| `the download of ... stopped at ...` or `download interrupted`    | Run the same command again: it resumes where it stopped.                                                                                                              |
| `sha256 mismatch`                                                 | The download was deleted. Run the command again. If it happens again, the upload or the recipe is wrong; please report it.                                            |
| `the engine image is not pulled` or `the model is not downloaded` | `serve` only starts what `pull` fetched. Run `tensorless pull MODEL`, or use `tensorless run MODEL`, which pulls first.                                               |
| `... is already running in podman`                                | The model’s server is already up. Use it, or stop it first with `tensorless stop MODEL`.                                                                              |
| `cannot use port 8080 on 127.0.0.1`                               | Something else listens on that port, often another model’s server. Pass `--port`, or stop the other server.                                                           |
| `could not start the container`                                   | The container tool refused the command, usually because it cannot reach the GPU. Run `tensorless doctor`.                                                             |
| `the server exited before it was ready`                           | Read the server’s log lines above the message (with `--detach`, the last 60 are printed). See below.                                                                  |
| `the server was not ready after 20m, so it was stopped`           | The model did not finish loading in 20 minutes. Check the log lines above it, and that the disk holding the cache is not very slow.                                   |
| `do not pass -c after --` (or another flag)                       | The launcher sets that flag itself. Use `--context`, `--port` or `--host` instead.                                                                                    |
| `no server answers http://.../health`                             | From `bench`: start the server first with `tensorless serve MODEL --detach`, or pass the `--host` and `--port` it runs on.                                            |

### When the server exits before it is ready

[Section titled “When the server exits before it is ready”](#when-the-server-exits-before-it-is-ready)

Read the log lines the launcher printed:

* **`no kernel image is available`**: the engine image was built for another GPU architecture than this card’s. The P100 image is built for sm_60 only.
* **An out-of-memory error**: another process is using the GPU. `nvidia-smi` lists the processes on each card; stop them, or another model’s server, and try again. A `--context` larger than the recipe’s can also run out of memory.

## Exit codes

[Section titled “Exit codes”](#exit-codes)

| Code | Meaning                                                 |
| ---- | ------------------------------------------------------- |
| 0    | Success.                                                |
| 1    | The command failed, or `doctor` found a failure.        |
| 2    | Bad usage: an unknown command or flag, a missing MODEL. |
| 130  | Interrupted, for example with Ctrl-C.                   |

[The command line reference](/docs/reference/cli/) describes every command and flag.

## Check that it worked

[Section titled “Check that it worked”](#check-that-it-worked)

`tensorless doctor` ends with `all checks passed.`, and `tensorless run iamMrDuncan/Qwen3.8-27B-PN3-GGUF` prints `is ready at http://127.0.0.1:8080`. If a message here does not help, ask on [Discord](https://discord.gg/3Qs2uejUf9) or [open an issue](https://github.com/iammrduncan/tensorless/issues) with the full output: see [Share your results](/docs/how-to/use/share-results/).

# Community

> Where to ask questions, share results from your own P100, and report problems.

tensorless is built in the open, and the people running it on their own cards are a big part of it. Here is where to find everyone.

## Discord

[Section titled “Discord”](#discord)

**[Join the tensorless Discord](https://discord.gg/3Qs2uejUf9).** It is the place for:

* questions about setting up a host, a driver or a container runtime;
* results from your own P100, good or bad;
* ideas for models, cards and kernels.

## GitHub

[Section titled “GitHub”](#github)

The repository is [iammrduncan/tensorless](https://github.com/iammrduncan/tensorless).

* **[Issues](https://github.com/iammrduncan/tensorless/issues)** are for problems and concrete requests: a command that fails, a number that does not match its receipt, a wrong page in these docs, a missing credit.
* **[Releases](https://github.com/iammrduncan/tensorless/releases)** have the launcher archives and each release’s notes.

When you report a problem, include what `tensorless version` and `tensorless doctor` print, and the full command you ran. [Fix a failed check or start](/docs/how-to/use/troubleshoot/) covers the common ones first.

## Share your numbers

[Section titled “Share your numbers”](#share-your-numbers)

A speed measured on someone else’s P100 is the best check there is on a receipt. If you run `tensorless bench`, post the result on Discord or in an issue, with your card, driver and launcher version. [Share your results](/docs/how-to/use/share-results/) says what to include.

## Models

[Section titled “Models”](#models)

The models live on Hugging Face:

* [iamMrDuncan/Qwen3.8-27B-PN3-GGUF](https://huggingface.co/iamMrDuncan/Qwen3.8-27B-PN3-GGUF)
* [iamMrDuncan/Qwen3.8-27B-UD-IQ3_XXS-Q3Floor-GGUF](https://huggingface.co/iamMrDuncan/Qwen3.8-27B-UD-IQ3_XXS-Q3Floor-GGUF)

# Contributing

> How to help: report problems, share benchmarks, propose recipes, and change the launcher, the engine or these docs.

There are many ways to help tensorless, and most of them don’t need any code. This page says what each kind of contribution needs, so that it can be used.

Contributions are made under the project’s license, Apache-2.0 ([License](/docs/project/license/)).

## Report a problem

[Section titled “Report a problem”](#report-a-problem)

Open an [issue](https://github.com/iammrduncan/tensorless/issues) with:

* what you ran, the whole command line;
* what happened, with the error message as printed;
* what `tensorless version` and `tensorless doctor` print;
* your card, driver version, distribution, and podman or docker.

A wrong number, a broken link and a confusing page are all problems worth reporting.

## Share a benchmark

[Section titled “Share a benchmark”](#share-a-benchmark)

Results from other people’s cards are how a receipt gets checked. Run `tensorless bench` on a tuned run (no `--context`, no extra arguments), and post the output on [Discord](https://discord.gg/3Qs2uejUf9) or in an issue. [Share your results](/docs/how-to/use/share-results/) lists what to include.

A result that is far from the receipt is the most useful kind. Please share those too.

## Propose a recipe

[Section titled “Propose a recipe”](#propose-a-recipe)

A recipe is a promise: this model, on this card, with these settings, measured this fast. So a new recipe, or a change to an existing one, needs evidence:

* **A new recipe** needs the model on Hugging Face at a fixed revision, its sha256, and a receipt: what you measured, on which card, with which image digest, and how.
* **A change to a recipe’s `[run]` or `[engine]` section** needs a new receipt, because the old numbers no longer apply.
* **A change to other fields,** such as the title or the license text, needs no measurement.

You can try a recipe on your own machine before proposing it: [Try a recipe of your own](/docs/how-to/advanced/write-a-recipe/). Recipes are reviewed like code. Open an issue or say hello on Discord first if you are planning a new model or card, so the work fits what is already under way.

## Change the launcher

[Section titled “Change the launcher”](#change-the-launcher)

The launcher is Go, in `cmd/` and `internal/`, with the recipes embedded from `recipes/`. Every pull request that touches it runs the same checks as the `launcher` workflow:

```sh
test -z "$(gofmt -l cmd internal *.go)"
go vet ./...
go test ./...
```

Run them before you open the pull request. Keep the launcher’s rules: it holds no inference code, it pins what it downloads, and it refuses with a reason rather than guessing. [How tensorless works](/docs/explanation/how-tensorless-works/) explains them.

## Change the engine

[Section titled “Change the engine”](#change-the-engine)

The engine in `engine/` is not edited in place. It is developed in a separate research repository and exported into tensorless at each release, as one snapshot of the exact commit that was measured ([Where the engine comes from](/docs/explanation/the-engine/)). A pull request against `engine/` would be replaced by the next export.

So for an engine change, open an issue first, or bring it to Discord: a bug, a kernel idea, a measurement that disagrees with a receipt. Engine changes reach tensorless through the research line and a release, measured under the same guards as everything else.

## Improve these docs

[Section titled “Improve these docs”](#improve-these-docs)

The pages are Markdown files in the repository’s `site/` folder, and every page has an edit link at the bottom. Each page is one of four types: a tutorial, a how-to guide, reference or explanation. A change keeps a page to its type, and describes only what exists in the code today.

# Credits

> The projects, papers, models, datasets and people tensorless is built on.

This page is [CREDITS.md](https://github.com/iammrduncan/tensorless/blob/main/CREDITS.md) from the repository, which is the source: a change goes there first.

tensorless stands on a great deal of other people’s work: an inference engine several forks deep, open models and quantizations, P100 owners who published their patches and numbers, and the papers behind the format, the kernels and the way we measured them. This page thanks them.

License obligations for the code tensorless ships are in [NOTICE](https://github.com/iammrduncan/tensorless/blob/main/NOTICE), [THIRD_PARTY_NOTICES.md](https://github.com/iammrduncan/tensorless/blob/main/THIRD_PARTY_NOTICES.md) and [LICENSES/](https://github.com/iammrduncan/tensorless/tree/main/LICENSES). Model licenses are in each model’s card. If we missed you, or got something wrong, open an issue or tell us on [Discord](https://discord.gg/3Qs2uejUf9).

## The code tensorless ships

[Section titled “The code tensorless ships”](#the-code-tensorless-ships)

* **[llama.cpp / ggml](https://github.com/ggml-org/llama.cpp)**, Georgi Gerganov and contributors. The foundation of the engine, plus its quantization formats (K-quants and i-quants), `llama-quantize`, the importance matrix, `llama-server` and the test tools. MIT.
* **[llama-cpp-turboquant](https://github.com/TheTom/llama-cpp-turboquant)**, TheTom: TurboQuant KV-cache types and native MTP, the base of llamAmpere. MIT.
* **[llamAmpere](https://github.com/JakeATX/llamAmpere)**, JakeATX: the fork our engine starts from (`2cb16936`), with its EXL3 GGUF path and Qwen3.8 work. MIT.
* **[exllamav3](https://github.com/turboderp-org/exllamav3)**, turboderp: the EXL3 format, whose code the engine uses for its EXL3 weight types. MIT.
* **Vendored libraries:** cpp-httplib, nlohmann/json, miniaudio, stb_image, subprocess.h, xxHash, SHA-1, SHA-256 and rotate-bits. See [THIRD_PARTY_NOTICES.md](https://github.com/iammrduncan/tensorless/blob/main/THIRD_PARTY_NOTICES.md).
* **[BurntSushi/toml](https://github.com/BurntSushi/toml)**, used by the launcher. MIT.

## Models and quantizations

[Section titled “Models and quantizations”](#models-and-quantizations)

* **The Qwen team (Alibaba): [Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B).** The base model of every launch target. Apache-2.0.

* **[Unsloth](https://huggingface.co/unsloth): [unsloth/Qwen3.8-27B-GGUF](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF).** Qwen3.8-27B-UD-IQ3_XXS-Q3Floor is mostly their work:

  * about 73% of its bytes are their UD-IQ3_XXS, unchanged;
  * its `token_embd` is from their UD-IQ4_XS;
  * its re-quantized tensors come from their BF16 GGUF, quantized with their importance matrix. Their quantization ladder was also our main comparison, and their Dynamic 3.0 notes gave us the trajectory-divergence check we used to judge quantization quality.

* **[turboderp/Qwen3.8-27B-exl3](https://huggingface.co/turboderp/Qwen3.8-27B-exl3):** the EXL3 quantizations we measured on the P100.

* **bartowski and AtomicChat:** Qwen3.8-27B GGUF ladders we compared against.

* **Drafters we measured for speculative decoding on the P100:** [z-lab/Qwen3.8-27B-DFlash2](https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2) (with [DFlash](https://github.com/z-lab/dflash)), incoai’s DFlash2 builds, and the RadixArk and Anbeeld DSpark drafters.

## P100 and Pascal know-how from the community

[Section titled “P100 and Pascal know-how from the community”](#p100-and-pascal-know-how-from-the-community)

* **[shinbunbun/llama-cpp-p100-patches](https://github.com/shinbunbun/llama-cpp-p100-patches):** 29 patches for llama.cpp on the P100, each measured on our card. Two shaped our engine directly:

  * the half2 Q4_1 GEMV with LOP3 unpacking (patch 12);
  * the MTP draft-vocabulary subset (patch 15), which led to our draft-vocab maps.

* **[qr0n/p100-llama-cpp](https://github.com/qr0n/p100-llama-cpp):** a P100 fork whose fp16-subnormal Q4_K GEMV (patch 0007) is behind our output-head kernel.

* **[Joe11221/p100-llama-cpp](https://github.com/Joe11221/p100-llama-cpp):** P100 issue-rate measurements (packed half2, fp32, emulated int8).

* **[Kmic-68/llama.cpp](https://github.com/Kmic-68/llama.cpp):** P100 optimization patches and a detailed optimization log, including the blocked fp16 cuBLAS algorithm for prefill, GEMM-based flash attention and head-256 GQA attention.

* **[hma02/cublasHgemm-P100](https://github.com/hma02/cublasHgemm-P100):** HGEMM and SGEMM rates on the P100.

* **P100 owners who posted numbers:** on r/LocalLLM, a [thread of P100 owners’ token rates](https://www.reddit.com/r/LocalLLM/comments/1vwvng2/tesla_p100_folks_whats_your_tps/) and a [P40 vs P100 vs V100 comparison on Qwen3.8](https://www.reddit.com/r/LocalLLM/comments/1w41178/the_lost_world_p40_vs_p100_vs_v100_in_qwen_38/). Also the llama.cpp issue on [SM60 accumulation](https://github.com/ggml-org/llama.cpp/issues/25593).

## Engines and projects we learned from

[Section titled “Engines and projects we learned from”](#engines-and-projects-we-learned-from)

* **[TensorFold](https://github.com/ashhart/TensorFold)**, ashhart:

  * Its exact speculative decoding, in-place regrouping and chain kernels shaped our kernel and speculation work.
  * Its recipe harness was the evaluation baseline for the PN quantization.
  * Its `pull` / `serve` / `info` / `capabilities` command line shaped the tensorless launcher.

* **[Lucebox](https://github.com/Luce-Org/lucebox-hub)**, Luce-Org: DFlash with DDTree on the RTX 3090, and a hybrid DeltaNet megakernel.

* **[0xSero/qwen38-b70](https://github.com/0xSero/qwen38-b70)** (Qwen3.8-27B on Intel Arc Pro B70) and **[0xSero/trellis-serve](https://github.com/0xSero/trellis-serve)** (EXL3 trellis kernels on the RTX 3090).

* **[MiaAI-Lab’s one-click Qwen3.8-27B installer](https://github.com/MiaAI-Lab/Qwen3.8-27B-16gb-NVIDIA-GPUs-one-click-install):** an EXL3 configuration for 16 GB cards.

* **[exllamav2](https://github.com/turboderp-org/exllamav2)**, for reconstructing to fp16 for wide batches. **[SGLang](https://github.com/sgl-project/sglang)** and **[vLLM](https://github.com/vllm-project/vllm)**, for their Qwen3.8 and hybrid-model implementations; vLLM also ran our structured-output evaluation.

* **llama.cpp pull requests and discussions we built on:**

  * the fused gated-delta-net kernel ([#19504](https://github.com/ggml-org/llama.cpp/pull/19504));
  * Hadamard rotation for the quantized KV cache ([#21038](https://github.com/ggml-org/llama.cpp/pull/21038));
  * deterministic mode ([#16016](https://github.com/ggml-org/llama.cpp/pull/16016));
  * DFlash2 support ([#27342](https://github.com/ggml-org/llama.cpp/pull/27342));
  * the CUDA graphs work ([#6766](https://github.com/ggml-org/llama.cpp/pull/6766));
  * the KV-cache quantization KLD tables ([discussion #23470](https://github.com/ggml-org/llama.cpp/discussions/23470)).

## Papers

[Section titled “Papers”](#papers)

### Quantization: the PN format and its quantizer

[Section titled “Quantization: the PN format and its quantizer”](#quantization-the-pn-format-and-its-quantizer)

* **GPTQ / OPTQ** (Frantar et al.), [arXiv 2210.17323](https://arxiv.org/abs/2210.17323): the quantizer behind PN3.

* **QuIP** (Chee et al.), [2307.13304](https://arxiv.org/abs/2307.13304), and **QuIP#** (Tseng et al.), [2402.04396](https://arxiv.org/abs/2402.04396): incoherence processing. No QuIP# code is used.

* **QuaRot** (Ashkboos et al.), [2404.00456](https://arxiv.org/abs/2404.00456), and **SpinQuant** (Liu et al.), [2405.16406](https://arxiv.org/abs/2405.16406): the rotation line our Hadamard-rotated basis follows.

* **QTIP** (Tseng, Sun, Hou, De Sa), [2406.11235](https://arxiv.org/abs/2406.11235): trellis-coded quantization, the basis of EXL3.

* **Other methods studied while designing the format:**

  * HIGGS (Malinovskii et al.), [2411.17525](https://arxiv.org/abs/2411.17525);
  * GPTAQ, [2504.02692](https://arxiv.org/abs/2504.02692);
  * Qronos, [2505.11695](https://arxiv.org/abs/2505.11695);
  * GAR, [2505.14638](https://arxiv.org/abs/2505.14638);
  * FlatQuant, [2410.09426](https://arxiv.org/abs/2410.09426);
  * AutoRound / SignRound, [2309.05516](https://arxiv.org/abs/2309.05516);
  * GSQ, [2604.18556](https://arxiv.org/abs/2604.18556);
  * RCO, [2605.00649](https://arxiv.org/abs/2605.00649);
  * ReQuant, [2608.07019](https://arxiv.org/abs/2608.07019).

* **Outliers and sensitive weights:**

  * Massive Activations (Sun et al.), [2402.17762](https://arxiv.org/abs/2402.17762);
  * Super Weights (Yu, Wang, Shan, Reed, Wan), [2411.07191](https://arxiv.org/abs/2411.07191);
  * Bondarenko, Nagel and Blankevoort, [2306.12929](https://arxiv.org/abs/2306.12929);
  * LLM.int8() (Dettmers et al.), [2208.07339](https://arxiv.org/abs/2208.07339);
  * Kozyrev and Maiboroda, [2609.04098](https://arxiv.org/abs/2609.04098), on which Qwen3.8-27B components tolerate low precision.

### Measuring quality and exactness

[Section titled “Measuring quality and exactness”](#measuring-quality-and-exactness)

* **Answer flips** (Dutta et al.), [2407.09141](https://arxiv.org/abs/2407.09141), and **MarginGate**, [2605.30218](https://arxiv.org/abs/2605.30218): why KL divergence and top-token agreement are the right gates.
* **The Curious Case of Neural Text Degeneration** (Holtzman et al.), [1904.09751](https://arxiv.org/abs/1904.09751).
* **Run-to-run variance:** Yuan et al., [2506.09501](https://arxiv.org/abs/2506.09501), and [2510.26788](https://arxiv.org/abs/2510.26788), on numerical nondeterminism in inference.
* **Thinking Machines, [“Defeating Nondeterminism in LLM Inference”](https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/):** batch invariance. **RepOps / Verde**, [2502.19405](https://arxiv.org/abs/2502.19405): reproducible bitwise inference.

### Speculative decoding

[Section titled “Speculative decoding”](#speculative-decoding)

* **Speculative sampling:** Leviathan et al., [2211.17192](https://arxiv.org/abs/2211.17192); Chen et al., [2302.01318](https://arxiv.org/abs/2302.01318).

* **Drafters and verifiers:**

  * DFlash, [2602.06036](https://arxiv.org/abs/2602.06036);
  * DSpark, [2607.05147](https://arxiv.org/abs/2607.05147);
  * DFlare, [2606.02091](https://arxiv.org/abs/2606.02091);
  * xPress, [2608.02438](https://arxiv.org/abs/2608.02438);
  * DBloom, [2608.30427](https://arxiv.org/abs/2608.30427);
  * Judge Decoding, [2501.19309](https://arxiv.org/abs/2501.19309);
  * lossy verification, [2607.26627](https://arxiv.org/abs/2607.26627);
  * speculative decoding meets quantization, [2505.22179](https://arxiv.org/abs/2505.22179).

* **The RTX 3090 Qwen3.8-27B [speculative decoding study](https://hackmd.io/@thc1006/qwen3-8-27b-speculative-decoding-rtx-3090)** by thc1006.

### Kernels and systems

[Section titled “Kernels and systems”](#kernels-and-systems)

* **Flash-Decoding** (Dao, Haziza, Massa, Sizov): the split-KV idea behind our P100 decode-attention kernel. **FlashAttention** (Dao et al.), [2205.14135](https://arxiv.org/abs/2205.14135).

* **DeepSeek-V3**, [2412.19437](https://arxiv.org/abs/2412.19437): promoting partial sums to fp32 at fixed intervals.

* **GPU synchronization microbenchmarks** (Zhang et al.), [2004.05371](https://arxiv.org/abs/2004.05371): the cost of a grid barrier on the P100.

* **Ideas we evaluated:**

  * TEAL, [2408.14690](https://arxiv.org/abs/2408.14690), activation sparsity;
  * Strassen on GPU, [1808.07984](https://arxiv.org/abs/1808.07984);
  * DistServe, [2401.09670](https://arxiv.org/abs/2401.09670), and Mooncake, [2407.00079](https://arxiv.org/abs/2407.00079), prefill/decode disaggregation.

## Data

[Section titled “Data”](#data)

The PN quantization was calibrated and evaluated on these datasets; tensorless does not redistribute any of them.

* **Calibration:**

  * [mlabonne/open-perfectblend](https://huggingface.co/datasets/mlabonne/open-perfectblend);
  * [open-thoughts/OpenThoughts-114k](https://huggingface.co/datasets/open-thoughts/OpenThoughts-114k);
  * [AI-MO/NuminaMath-1.5](https://huggingface.co/datasets/AI-MO/NuminaMath-1.5);
  * [glaiveai/glaive-function-calling-v2](https://huggingface.co/datasets/glaiveai/glaive-function-calling-v2);
  * [HuggingFaceFW/fineweb-edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu);
  * [allenai/c4](https://huggingface.co/datasets/allenai/c4);
  * [HuggingFaceH4/ultrachat_200k](https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k).

* **Evaluation:**

  * WikiText-2 (Merity et al.; [Salesforce/wikitext](https://huggingface.co/datasets/Salesforce/wikitext));
  * JSONSchemaBench (Geng et al., [2501.10868](https://arxiv.org/abs/2501.10868));
  * BFCL v3 ([gorilla-llm](https://huggingface.co/datasets/gorilla-llm/Berkeley-Function-Calling-Leaderboard));
  * StructEval ([2505.20139](https://arxiv.org/abs/2505.20139));
  * IFEval ([2311.07911](https://arxiv.org/abs/2311.07911));
  * RULER ([2404.06654](https://arxiv.org/abs/2404.06654)).

## Hardware documentation

[Section titled “Hardware documentation”](#hardware-documentation)

**NVIDIA:**

* the Pascal Tuning Guide;
* the Tesla P100 whitepaper and datasheet;
* the CUDA C Programming Guide, Math API and cuBLAS documentation;
* the CUDA 13 release notes, which set our CUDA 12.x floor for Pascal;
* the Container Toolkit’s CDI guide.

## Tools, harnesses and the agents that did the work

[Section titled “Tools, harnesses and the agents that did the work”](#tools-harnesses-and-the-agents-that-did-the-work)

* **[Thermite](https://github.com/iammrduncan/thermite):** the automated research harness that ran the PN campaigns: a judge the agents can’t see, held-out checks, A/A noise calibration, an auditor, and the decode hill climb (131 judged attempts, 28 confirmed wins).

* **[pi-autoresearch](https://github.com/davebcn87/pi-autoresearch)** (davebcn87) and the **[Pi](https://pi.dev/)** coding agent: the experiment loop of the earlier P100 kernel campaigns. **[pod-pi](https://github.com/Hackers-in-the-Loop/pod-pi):** their rootless podman workspace.

* **NVIDIA’s nvprof, compute-sanitizer, cuobjdump and nvdisasm**, which still support sm_60.

* **PyTorch, transformers, vLLM, llm-compressor, compressed-tensors and flash-linear-attention**, for the quantization pipeline. **LiteLLM**, for model routing. **Hugging Face**, for hosting the models.

* **The models that worked as researchers:**

  * GLM-5.3 Flash, DeepSeek V4.1 Flash, Ornith-1.5-122B and Claude Haiku 5.5 as workers;
  * GPT 6.1 Sol and Claude Opus 5.5 as teachers and auditors;
  * Claude Sonnet 5.5 as planner;
  * Claude Code and Codex for supervision, research and this repository.

# Known limits

> What tensorless v0.1.0 does not do yet, what has not been measured, and what to avoid.

tensorless v0.1.0 is a first release. This page lists what it does not do yet, what has not been measured, and the one thing to avoid. Each item says what is true today; none of them is a promise of when it will change.

## Long context is not measured on this engine

[Section titled “Long context is not measured on this engine”](#long-context-is-not-measured-on-this-engine)

The receipts for engine-v0.1.0 measure decode at short context: 256 tokens after a prompt of a few hundred. Decode at 32K and 65K context has not been measured on this engine yet.

* **Qwen3.8-27B-PN3:** only the hill climb’s starting point was measured at long context, at 7.55 and 7.39 tok/s for 32K and 65K. The climb then made short-context decode almost four times faster. Attention takes a larger share of each token at long context, so expect this engine’s long-context decode to be below its 30 tok/s, by an amount nobody has measured yet.
* **Qwen3.8-27B-UD-IQ3_XXS-Q3Floor:** 24.5 and 22.9 tok/s at 32K and 65K, measured on the climb’s starting commit. This model’s code path changed little after that: its text stayed bit-identical, and its speed was guarded within 3%.

Both recipes still allocate the full 65,536-token context.

## Don’t run PN models on the CPU

[Section titled “Don’t run PN models on the CPU”](#dont-run-pn-models-on-the-cpu)

The engine’s CPU backend decodes PN weights wrong in its dot product: it shifts by element index instead of pair index, and most PN weights come out wrong. The GPU paths are correct, and every recipe runs the whole model on the GPU, so `tensorless run` is not affected.

It matters only if you run the engine by hand. Don’t run Qwen3.8-27B-PN3 on the CPU backend, or with only some layers on the GPU, until this is fixed. [`engine/PROVENANCE.md`](https://github.com/iammrduncan/tensorless/blob/main/engine/PROVENANCE.md) records it.

## No speculative decoding yet

[Section titled “No speculative decoding yet”](#no-speculative-decoding-yet)

Neither recipe uses speculative decoding. An earlier kernel campaign ran Q3Floor with multi-token prediction (MTP) at about 28 tok/s at 32K and 65K context, on its own engine version, but that mode is not in any recipe.

## One request at a time

[Section titled “One request at a time”](#one-request-at-a-time)

Both recipes run `llama-server` with one slot (`-np 1`), the configuration that was measured. It serves one request at a time; others wait their turn. Batched or concurrent throughput has not been measured.

## One card family, on x86_64 Linux

[Section titled “One card family, on x86_64 Linux”](#one-card-family-on-x86_64-linux)

* **Only the Tesla P100 16 GB** (`sm_60`) has recipes. A 12 GB P100 does not fit them, and other cards have no recipe.
* **The engine image is built for Linux on x86_64 only.** The launcher also comes for Linux aarch64 and macOS on Apple silicon, where it can list and show recipes and benchmark a server running on another machine, but cannot serve a model.
* **The driver must be from R525 to R580.** R580 is the last driver branch that supports Pascal.

## Not yet confirmed

[Section titled “Not yet confirmed”](#not-yet-confirmed)

The project’s design notes list these as still to confirm:

* the image’s supported driver range, on a real P100 host;
* the launcher’s container support, on rootless podman with CDI and on docker with the NVIDIA container toolkit.

If you run tensorless on either setup, your result helps close these. Tell us on [Discord](https://discord.gg/3Qs2uejUf9) or in an [issue](https://github.com/iammrduncan/tensorless/issues).

## Quality on long prose

[Section titled “Quality on long prose”](#quality-on-long-prose)

Qwen3.8-27B-PN3 matches Q3Floor on code, short text, long-context recall and a task screen, but has 7.5% higher perplexity on long prose at 32K. The cause is the 3-bit quantization, not the kernels. [Why a P100-native weight format](/docs/explanation/pn-format/#what-it-costs) has the measurements.

# License

> tensorless is Apache-2.0; its engine keeps the MIT notices of the projects it derives from; model weights keep their own licenses.

tensorless is open source under the **Apache License, Version 2.0**. This page says what that covers, what keeps another license, and where each notice is.

## What is Apache-2.0

[Section titled “What is Apache-2.0”](#what-is-apache-20)

The launcher, the recipes, the container files, these docs and every change made for tensorless are licensed under [Apache-2.0](https://github.com/iammrduncan/tensorless/blob/main/LICENSE). The project’s [NOTICE](https://github.com/iammrduncan/tensorless/blob/main/NOTICE) file carries the attribution that Apache-2.0 asks you to keep.

## What keeps its own license

[Section titled “What keeps its own license”](#what-keeps-its-own-license)

**The engine’s upstreams.** The engine in `engine/` is derived from MIT-licensed projects, and their copyright and permission notices stay with it:

| Project                                                                                                  | License              |
| -------------------------------------------------------------------------------------------------------- | -------------------- |
| [llama.cpp / ggml](https://github.com/ggml-org/llama.cpp), the ggml authors                              | MIT                  |
| [llama-cpp-turboquant](https://github.com/TheTom/llama-cpp-turboquant), TheTom                           | MIT, the same notice |
| [llamAmpere](https://github.com/JakeATX/llamAmpere), JakeATX                                             | MIT, the same notice |
| [exllamav3](https://github.com/turboderp-org/exllamav3), turboderp (portions, for the EXL3 weight types) | MIT                  |

**Libraries vendored into the engine,** such as cpp-httplib, nlohmann/json and xxHash, each under its own license (MIT, BSD-2-Clause, public domain and others). **The launcher’s one dependency,** [BurntSushi/toml](https://github.com/BurntSushi/toml), is MIT. **The CUDA libraries** in the engine images come from NVIDIA’s CUDA base images, under the NVIDIA CUDA Toolkit license.

[THIRD_PARTY_NOTICES.md](https://github.com/iammrduncan/tensorless/blob/main/THIRD_PARTY_NOTICES.md) lists every one of them and where its notice is. The license texts are in [LICENSES/](https://github.com/iammrduncan/tensorless/tree/main/LICENSES) and in the engine’s own license files. The release archives of the launcher carry `LICENSE`, `NOTICE`, `THIRD_PARTY_NOTICES.md` and `LICENSES/`, and each engine image carries the same files under `/opt/tensorless/`.

## Model weights

[Section titled “Model weights”](#model-weights)

Model weights are not part of tensorless. They are downloaded from Hugging Face and keep their own licenses. Each recipe names its model’s license, and `tensorless info MODEL` shows it before anything is downloaded.

| Model                          | License                                                        |
| ------------------------------ | -------------------------------------------------------------- |
| Qwen3.8-27B-PN3                | Apache-2.0, from Qwen/Qwen3.8-27B                              |
| Qwen3.8-27B-UD-IQ3_XXS-Q3Floor | Apache-2.0, from Qwen/Qwen3.8-27B and unsloth/Qwen3.8-27B-GGUF |

Read a model’s card on Hugging Face for its full terms.

# Command line

> Every tensorless command, its flags and defaults, the MODEL argument, untuned runs and exit codes.

`tensorless` is one static binary. It runs a model’s [recipe](/docs/reference/recipes/) on your GPU: it pulls the engine image by digest, downloads the model into the Hugging Face cache, checks its sha256, and starts `llama-server` in podman or docker. It holds no inference code.

```text
tensorless COMMAND [MODEL] [flags] [-- llama-server args]
```

Run `tensorless help` for the list of commands, and `tensorless COMMAND -h` for a command’s flags.

## Commands

[Section titled “Commands”](#commands)

| Command                         | Usage                                                   | Summary                                                                               |
| ------------------------------- | ------------------------------------------------------- | ------------------------------------------------------------------------------------- |
| [`run`](#run)                   | `tensorless run MODEL [flags] [-- llama-server args]`   | Pull, then serve.                                                                     |
| [`pull`](#pull)                 | `tensorless pull MODEL [flags]`                         | Pull the engine image by digest and download the model into the Hugging Face cache.   |
| [`serve`](#serve)               | `tensorless serve MODEL [flags] [-- llama-server args]` | Start the server, wait until it is healthy, and print the endpoint.                   |
| [`stop`](#stop)                 | `tensorless stop MODEL [flags]`                         | Stop the model’s server.                                                              |
| [`bench`](#bench)               | `tensorless bench MODEL [flags]`                        | Measure decode speed against the running server and compare it with the receipt.      |
| [`models`](#models)             | `tensorless models [flags]`                             | List the recipes, whether they fit this GPU, and whether the model is cached.         |
| [`info`](#info)                 | `tensorless info MODEL [flags]`                         | Show a recipe exactly as it would run. Needs no GPU.                                  |
| [`doctor`](#doctor)             | `tensorless doctor [flags]`                             | Check the GPU, driver, container runtime, NVIDIA container support and disk space.    |
| [`capabilities`](#capabilities) | `tensorless capabilities [flags]`                       | Describe this launcher: version, recipes, runtimes. Needs no GPU, runtime or network. |
| [`version`](#version)           | `tensorless version`                                    | Print the version.                                                                    |
| [`help`](#help)                 | `tensorless help`                                       | Print the list of commands.                                                           |

## The MODEL argument

[Section titled “The MODEL argument”](#the-model-argument)

`run`, `pull`, `serve`, `stop`, `bench` and `info` take exactly one MODEL. `models`, `doctor`, `capabilities` and `version` take none.

| Form                    | Example                                            | Meaning                                                                                                                                                                                                                                                   |
| ----------------------- | -------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Repo id                 | `iamMrDuncan/Qwen3.8-27B-PN3-GGUF`                 | The model’s Hugging Face repository. Case does not matter, as on the Hub.                                                                                                                                                                                 |
| Short name              | `pn3`                                              | A recipe’s optional `model.name`. It works anywhere a repo id does.                                                                                                                                                                                       |
| Repo id with a revision | `iamMrDuncan/Qwen3.8-27B-PN3-GGUF@<40-hex commit>` | Another commit of the repository. The revision must be a 40-character commit, not a branch or a tag. A revision other than the recipe’s needs `--allow-draft`, makes the run [untuned](#untuned-runs), and the file must still match the recipe’s sha256. |

When a model has recipes for several cards, the launcher picks the one whose card matches a GPU that `nvidia-smi` reports, or the one `--card` names. [Launch targets](/docs/reference/launch-targets/) lists the models of this release, and `tensorless models` lists the ones your launcher knows.

## Flags

[Section titled “Flags”](#flags)

Every command except `version` accepts the common flags. Flags may come before or after MODEL. A flag is written with one dash or two (`-port` or `--port`), and its value after a space or `=`.

### Common flags

[Section titled “Common flags”](#common-flags)

| Flag             | Type                         | Default                                                               | Meaning                                                                                                                                                                                                  |
| ---------------- | ---------------------------- | --------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--recipes DIR`  | path                         | built-in recipes                                                      | Read recipes from `DIR/<org>/<repo>/<card>.toml` instead of the ones built into the binary.                                                                                                              |
| `--card ID`      | string                       | detected                                                              | Use this card’s recipe, for example `p100`, and skip GPU detection, with its memory and driver checks.                                                                                                   |
| `--gpu N`        | integer ≥ 0                  | the first GPU that matches the recipe                                 | Use GPU index `N`, as `nvidia-smi` numbers them. The container gets only that GPU.                                                                                                                       |
| `--runtime NAME` | `auto`, `podman` or `docker` | `auto`                                                                | The container runtime. `auto` prefers podman, then docker.                                                                                                                                               |
| `--cache DIR`    | path                         | `$HF_HUB_CACHE`, else `$HF_HOME/hub`, else `~/.cache/huggingface/hub` | The Hugging Face cache directory for model files.                                                                                                                                                        |
| `--allow-draft`  | switch                       | off                                                                   | Allow a draft recipe, or a `MODEL@REVISION` other than the recipe’s. The run is then untuned. It never allows a recipe that is [not published](/docs/reference/recipes/#draft-released-and-unpublished). |
| `--dry-run`      | switch                       | off                                                                   | Print every command and download URL instead of running it. It still reads the GPU list with `nvidia-smi`, unless `--card` is given.                                                                     |
| `--host ADDR`    | string                       | `127.0.0.1`                                                           | The host address the server is published on. Must not be empty.                                                                                                                                          |
| `--port N`       | integer 1–65535              | `8080`                                                                | The host port the server is published on.                                                                                                                                                                |
| `--context N`    | integer > 0                  | the recipe’s `run.context`                                            | The context length. Any other value than the recipe’s makes the run untuned.                                                                                                                             |

### Flags of some commands

[Section titled “Flags of some commands”](#flags-of-some-commands)

| Flag         | Commands                                  | Default | Meaning                                                          |
| ------------ | ----------------------------------------- | ------- | ---------------------------------------------------------------- |
| `--detach`   | `run`, `serve`                            | off     | Return once the server is healthy instead of following its logs. |
| `--json`     | `bench`, `models`, `info`, `capabilities` | off     | Print JSON.                                                      |
| `--runs N`   | `bench`                                   | `3`     | The number of measured runs. At least 1.                         |
| `--tokens N` | `bench`                                   | `256`   | Tokens to generate per run. At least 2.                          |

### Extra llama-server arguments

[Section titled “Extra llama-server arguments”](#extra-llama-server-arguments)

`run`, `serve` and `info` accept arguments after `--`. They are passed to `llama-server` after the recipe’s own arguments, and they make the run untuned. Other commands refuse them.

The launcher sets these `llama-server` flags itself, so they are refused after `--` (and in a recipe’s `run.args`): `-m`, `--model`, `--host`, `--port`, `-c`, `--ctx-size`, `-a`, `--alias`, `-t`, `--threads`, `-tb`, `--threads-batch`. Use `--host`, `--port` and `--context` instead.

## Untuned runs

[Section titled “Untuned runs”](#untuned-runs)

A run is untuned when the command line changes the configuration a recipe measured:

* `--context` with another value than the recipe’s;
* extra arguments after `--`;
* a `MODEL@REVISION` other than the recipe’s.

`run` and `serve` print `UNTUNED:` and the reason before they start, and say that the receipt’s numbers do not apply. `info` shows the reason in an `UNTUNED` line, and `--dry-run` in its header. `--host`, `--port`, `--gpu`, `--card`, `--cache` and `--runtime` do not make a run untuned.

## Refused recipes

[Section titled “Refused recipes”](#refused-recipes)

| Recipe                                                                           | `pull`, `run`, `serve`              | `info`                         | `--dry-run`                     |
| -------------------------------------------------------------------------------- | ----------------------------------- | ------------------------------ | ------------------------------- |
| Draft (`status = "draft"`)                                                       | Refused unless `--allow-draft`.     | Shown.                         | Refused unless `--allow-draft`. |
| Not published: an all-zero revision, sha256 or image digest, or `size_bytes = 0` | Refused, even with `--allow-draft`. | Shown, marked `NOT PUBLISHED`. | Shown.                          |

## run

[Section titled “run”](#run)

```sh
tensorless run iamMrDuncan/Qwen3.8-27B-PN3-GGUF
tensorless run iamMrDuncan/Qwen3.8-27B-PN3-GGUF --detach --port 9000
tensorless run iamMrDuncan/Qwen3.8-27B-PN3-GGUF -- --temp 0.7
```

Runs [`pull`](#pull), then [`serve`](#serve), in one command. Before it pulls, it checks that the port is free and that the model’s container is not already running.

## pull

[Section titled “pull”](#pull)

```sh
tensorless pull iamMrDuncan/Qwen3.8-27B-PN3-GGUF
```

1. Pulls the engine image by digest, unless `podman image inspect` (or `docker image inspect`) already finds it.
2. Downloads the model file at the recipe’s revision into the Hugging Face cache, as `<cache>/models--<org>--<repo>/blobs/<sha256>`, and checks its sha256.
3. Links it into the snapshot, `<cache>/models--<org>--<repo>/snapshots/<revision>/<file>`, so other Hugging Face tools find it.

| Behavior        | Detail                                                                                                                                       |
| --------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| Resume          | An interrupted download is kept as `<blob>.incomplete`. The next `pull` hashes what is there and resumes with an HTTP range request.         |
| Disk space      | Checked before the download. Too little space fails with the space needed and the space free.                                                |
| Stall           | A download that receives nothing for 2 minutes fails; run the command again to resume.                                                       |
| Already cached  | A cached file is hashed once. A marker in the state directory then skips the hash while the file’s size and modification time stay the same. |
| sha256 mismatch | The download is deleted and the command fails.                                                                                               |

## serve

[Section titled “serve”](#serve)

```sh
tensorless serve iamMrDuncan/Qwen3.8-27B-PN3-GGUF
tensorless serve iamMrDuncan/Qwen3.8-27B-PN3-GGUF --detach
```

Starts the model’s container from an image and model that `pull` already fetched and verified. It fails if the image is not pulled or the model is not downloaded, and names the `pull` command to run.

| Behavior        | Detail                                                                                                                                                                                    |
| --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Container name  | `tensorless-` and the repo id, lowercased, with every character other than `a`–`z`, `0`–`9`, `_`, `.` and `-` replaced by `-`. For example `tensorless-iammrduncan-qwen3.8-27b-pn3-gguf`. |
| Already running | Fails with `is already running` and points to `tensorless stop`. A stopped container of the same name is removed first.                                                                   |
| Port in use     | Fails with `cannot use port` before anything starts.                                                                                                                                      |
| Waiting         | Polls `http://<host>:<port>/health` every second, for up to 20 minutes, while the model loads.                                                                                            |
| Ready           | Prints the endpoint (the OpenAI-compatible API is under `/v1`) and a `curl` example. The served model name is the recipe’s `run.alias`, the repo id.                                      |
| Foreground      | Follows the container’s logs. Ctrl-C or SIGTERM stops the container.                                                                                                                      |
| `--detach`      | Returns once the server is healthy, and prints the `tensorless stop` command.                                                                                                             |
| Exits early     | If the container exits before it is healthy, prints its last log lines (up to 60) and fails with `the server exited before it was ready`.                                                 |
| Timeout         | If the server is not healthy after 20 minutes, stops it and fails.                                                                                                                        |

The container command (podman shown):

```text
podman run -d --rm --name <container name> \
    --device nvidia.com/gpu=<all or N> --security-opt label=disable \
    -p <host>:<port>:8080 -v <blob>:/models/<file>:ro \
    -e OMP_NUM_THREADS=<T> -e <each run.env variable, sorted> \
    <engine.image> \
    -m /models/<file> --host 0.0.0.0 --port 8080 -c <context> \
    --alias <run.alias> -t <T> -tb <T> <run.args> <extra args>
```

* With docker, the GPU is `--gpus all` or `--gpus device=N`, and there is no `--security-opt`.
* `T` is the smaller of the host’s CPU count and the recipe’s `run.threads_max`.
* The GPU is `all` when every GPU on the host matches the recipe’s card. When some do not, the container gets only the matched GPU, so `llama-server` cannot land on a card the image was not built for. `--gpu N` always gives only GPU `N`.
* `--host` with an IPv6 address is published as `[ADDR]:<port>:8080`.

## stop

[Section titled “stop”](#stop)

```sh
tensorless stop iamMrDuncan/Qwen3.8-27B-PN3-GGUF
```

Stops the model’s container. With `--runtime auto`, it looks in every installed runtime, podman and docker. It prints `stopped <name>`, or `<name> is not running`. With `--dry-run`, it prints the stop commands.

## bench

[Section titled “bench”](#bench)

```sh
tensorless bench iamMrDuncan/Qwen3.8-27B-PN3-GGUF
tensorless bench iamMrDuncan/Qwen3.8-27B-PN3-GGUF --runs 5 --tokens 256 --json
```

Measures decode speed against a server that is already running on `--host` and `--port`. It fails if nothing answers `/health` there. It warns if the server’s `/v1/models` does not list the recipe’s alias.

| Step          | Detail                                                                                                                                                                               |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Prompt        | The first 2,000 characters of the Apache-2.0 license text, then a blank line and `Continue the text above.`                                                                          |
| Warm-up       | One request for 8 tokens, not timed.                                                                                                                                                 |
| Measured runs | `--runs` streamed `POST /completion` requests, each for `--tokens` tokens: temperature 0, `top_k` 1, seed 424242, `ignore_eos`, no prompt cache. Each request uses a new connection. |
| Decode speed  | (tokens − 1) / (time of the last streamed token − time of the first).                                                                                                                |
| TTFT          | The time of the first streamed token minus the start of the request.                                                                                                                 |
| Result        | The median decode speed and TTFT of the runs, the prompt tokens, and the comparison with the recipe’s `receipt.decode_tps.short`.                                                    |

The JSON output has `model`, `card`, `endpoint`, `runs` (each with `decode_tps`, `ttft_ms`, `prompt_tokens` and `tokens`), `median_decode_tps`, `median_ttft_ms`, `prompt_tokens`, and, when the recipe has a short-context receipt, `receipt_decode_tps` and `vs_receipt_percent`.

## models

[Section titled “models”](#models)

```sh
tensorless models
tensorless models --json
```

Lists every recipe the launcher knows, one row each:

| Column   | Values                                                                                                 |
| -------- | ------------------------------------------------------------------------------------------------------ |
| `MODEL`  | The repo id.                                                                                           |
| `CARD`   | The card id.                                                                                           |
| `STATUS` | `draft` or `released`, with `, unpublished` while a pinned value is a placeholder.                     |
| `ENGINE` | The engine release.                                                                                    |
| `FITS`   | `yes`, `no`, or `unknown` when no GPU can be read. `yes` needs a matching GPU and a new enough driver. |
| `CACHED` | `yes` (the file is there at its full size), `partial` (an interrupted download) or `no`.               |

The JSON rows have `model`, `name`, `card`, `title`, `status`, `published`, `release`, `arch`, `fits` and `cached`.

## info

[Section titled “info”](#info)

```sh
tensorless info iamMrDuncan/Qwen3.8-27B-PN3-GGUF
tensorless info pn3 --card p100 --json
```

Shows a recipe exactly as it would run. It needs no GPU, downloads nothing and starts nothing. It shows:

* the model’s repository, revision, file, sha256, size, license and download URL, and its place in the cache;
* the card requirements and the engine image;
* the run settings, the thread count, and the full container command with the endpoint;
* whether the run is untuned, and why;
* the receipt: summary, source, decode and prefill numbers, and notes.

`--context` and arguments after `--` show in the command and the `UNTUNED` line. The JSON output has `recipe`, `published`, `unpublished`, `fits`, `untuned`, `download` (`url`, `blob`, `snapshot`, `size_bytes`, `cached`), `runtime`, `gpu`, `threads`, `command` and `endpoint`.

## doctor

[Section titled “doctor”](#doctor)

```sh
tensorless doctor
```

Checks the host. Each line starts with `PASS`, `WARN` or `FAIL`; a failed check is followed by a `fix:` line with the command or step that fixes it. It ends with `all checks passed.` or `N checks failed.`, and exits 1 if any check failed.

| Check                    | Passes when                                                                                                                                                                  |
| ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| NVIDIA driver            | `nvidia-smi` is installed and lists the GPUs.                                                                                                                                |
| GPU                      | A GPU matches a recipe’s compute capability and memory. A GPU that fits no recipe is a `WARN`; no fitting GPU at all is a `FAIL`.                                            |
| Driver version           | The driver is at least the highest `min_driver` of the recipes that fit.                                                                                                     |
| Container runtime        | podman or docker is installed (`--runtime` picks one).                                                                                                                       |
| NVIDIA container support | podman: a CDI spec at `/etc/cdi/nvidia.yaml` or `/var/run/cdi/nvidia.yaml`. docker: the nvidia runtime in `docker info`, a CDI spec, or the NVIDIA container toolkit’s hook. |
| Disk space               | The model cache has room for the largest model that is not cached yet, among the recipes that fit (or all recipes, when no GPU can be read).                                 |

## capabilities

[Section titled “capabilities”](#capabilities)

```sh
tensorless capabilities --json
```

Describes the launcher. It runs no external command and needs no GPU, driver, runtime or network.

| JSON field          | Meaning                                                                                       |
| ------------------- | --------------------------------------------------------------------------------------------- |
| `name`              | `tensorless`                                                                                  |
| `version`, `commit` | The launcher’s version and the commit it was built from.                                      |
| `platform`          | Operating system and architecture, such as `linux/amd64`.                                     |
| `recipe_schema`     | The recipe schema this launcher reads: `1`.                                                   |
| `runtimes`          | `podman`, `docker`.                                                                           |
| `cards`             | Each known card: `id`, `title`, `compute_capability`, `min_memory_mib`, `min_driver`, `arch`. |
| `recipes`           | Each recipe: `model`, `name`, `card`, `status`, `published`, `release`, `arch`.               |

## version

[Section titled “version”](#version)

```sh
tensorless version
```

Prints the version, the commit and the platform, for example `tensorless 0.1.0 (commit d104660be24427080948eafb0522265e1450d642), linux/amd64`. `tensorless --version` does the same. It takes no flags.

## help

[Section titled “help”](#help)

`tensorless help`, `tensorless -h` and `tensorless --help` print the list of commands. `tensorless` with no command prints it too, and exits 2.

## Exit codes

[Section titled “Exit codes”](#exit-codes)

| Code  | Meaning                                                                           |
| ----- | --------------------------------------------------------------------------------- |
| `0`   | Success, including `help` and `-h`.                                               |
| `1`   | The command failed, or `doctor` found a failure.                                  |
| `2`   | Bad usage: an unknown command or flag, a missing or extra MODEL, a refused value. |
| `130` | Interrupted by Ctrl-C or SIGTERM.                                                 |

## Environment

[Section titled “Environment”](#environment)

The launcher reads `HF_HUB_CACHE`, `HF_HOME`, `HF_ENDPOINT`, `HF_TOKEN` and `XDG_STATE_HOME`. See [Files and environment variables](/docs/reference/files-and-environment/).

# Engine image

> The tensorless engine container image: its tags and digest, base images, contents, entrypoint, build configuration and host requirements.

The engine image holds `llama-server` built from the [engine source](https://github.com/iammrduncan/tensorless/tree/main/engine) for one GPU architecture. There is one image per engine release and architecture. Recipes pin it by digest, never by tag.

## Published images

[Section titled “Published images”](#published-images)

Registry: `ghcr.io/iammrduncan/tensorless-engine`. The images are public.

| Tag              | Digest                                                                    | Engine release  | Architecture        | Meaning                                                                                     |
| ---------------- | ------------------------------------------------------------------------- | --------------- | ------------------- | ------------------------------------------------------------------------------------------- |
| `0.1.0-sm60`     | `sha256:35409ff97c2f518e468790c28bd19206ae777a945fcffe5775585e28fd9939a7` | `engine-v0.1.0` | `sm60` (Tesla P100) | The release.                                                                                |
| `0.1.0-rc1-sm60` | `sha256:35409ff97c2f518e468790c28bd19206ae777a945fcffe5775585e28fd9939a7` | `engine-v0.1.0` | `sm60`              | The release candidate the release gate measured. The release tag points at the same digest. |

```text
ghcr.io/iammrduncan/tensorless-engine@sha256:35409ff97c2f518e468790c28bd19206ae777a945fcffe5775585e28fd9939a7
```

## Tags

[Section titled “Tags”](#tags)

| Pattern            | Example          | Made by                                                                                                                                                                                                                                               |
| ------------------ | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `X.Y.Z-rcN-<arch>` | `0.1.0-rc1-sm60` | A release-candidate build ([`engine-image.yml`](https://github.com/iammrduncan/tensorless/blob/main/.github/workflows/engine-image.yml)).                                                                                                             |
| `X.Y.Z-<arch>`     | `0.1.0-sm60`     | Promotion of a measured candidate’s digest, without a rebuild ([`engine-promote.yml`](https://github.com/iammrduncan/tensorless/blob/main/.github/workflows/engine-promote.yml)). The workflow checks that the new tag points at the measured digest. |

The source of an engine release is tagged `engine-vX.Y.Z` in the repository. The launcher is versioned separately, `vX.Y.Z`.

## Host requirements

[Section titled “Host requirements”](#host-requirements)

| Requirement       | Value                                                                                                                            |
| ----------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| GPU               | Compute capability 6.0 (`sm60` images): the NVIDIA Tesla P100.                                                                   |
| NVIDIA driver     | R525 or newer, on the R580 branch or older. R580 is the last branch with Pascal support. The recipes’ `min_driver` is 525.60.13. |
| CUDA on the host  | Not needed: the CUDA 12.6 runtime is in the image. CUDA 13 dropped Pascal, so `sm60` images stay on CUDA 12.x.                   |
| Container runtime | podman with an NVIDIA CDI spec, or docker with the NVIDIA container toolkit.                                                     |
| CPU               | x86-64 with AVX2, FMA and F16C. The image is built for `linux/amd64`.                                                            |

## Contents

[Section titled “Contents”](#contents)

| Path                                                               | Contents                                                                                                                                             |
| ------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| `/opt/tensorless/bin/llama-server`                                 | The server.                                                                                                                                          |
| `/opt/tensorless/bin/*.so`                                         | Its shared libraries, built with it.                                                                                                                 |
| `/opt/tensorless/LICENSE`                                          | Apache-2.0.                                                                                                                                          |
| `/opt/tensorless/NOTICE`, `/opt/tensorless/THIRD_PARTY_NOTICES.md` | Attribution, including the MIT-licensed upstreams.                                                                                                   |
| `/opt/tensorless/LICENSES/`                                        | The upstream license texts, the engine’s `LICENSE` (as `engine-LICENSE.txt`) and the licenses of code vendored into the engine (`LICENSES/engine/`). |
| `/opt/tensorless/PROVENANCE.md`                                    | Where the engine source came from: the upstream base, the research commit, the tree digest, what the export removed and rewrote, and known issues.   |
| `/opt/tensorless/VERSION`                                          | The version label the image was built with. For `0.1.0-sm60` it is the candidate’s label, because the release is the candidate’s digest.             |
| `/opt/tensorless/SHA256SUMS`                                       | The sha256 of every other file under `/opt/tensorless`.                                                                                              |

The image holds no model weights and no build tree.

| Setting           | Value                                                                                                                                                                             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Entrypoint        | `/opt/tensorless/bin/llama-server`                                                                                                                                                |
| Exposed port      | `8080`                                                                                                                                                                            |
| `PATH`            | `/opt/tensorless/bin` first                                                                                                                                                       |
| `LD_LIBRARY_PATH` | `/opt/tensorless/bin`                                                                                                                                                             |
| Labels            | `org.opencontainers.image.title` “tensorless engine (sm_60)”, `.description`, `.source` <https://github.com/iammrduncan/tensorless>, `.licenses` “Apache-2.0 AND MIT”, `.version` |

Arguments after the image name go to `llama-server`. The launcher passes the model path, `--host 0.0.0.0 --port 8080`, the context, the alias, the thread counts and the recipe’s arguments; see [Command line](/docs/reference/cli/#serve).

## Build

[Section titled “Build”](#build)

The image is built from [`containers/engine.sm60.Containerfile`](https://github.com/iammrduncan/tensorless/blob/main/containers/engine.sm60.Containerfile), with the repository root as the build context. It has two stages.

| Stage   | Base image                                         | What happens                                                                                                                                                                                      |
| ------- | -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Build   | `docker.io/nvidia/cuda:12.6.3-devel-ubuntu24.04`   | Installs cmake, ninja and git; configures and builds `llama-server` and `test-pn-format`; runs `test-pn-format`, so a PN decoding error fails the build; copies the server and its libraries out. |
| Runtime | `docker.io/nvidia/cuda:12.6.3-runtime-ubuntu24.04` | Installs `libgomp1`; copies the server, libraries and license files into `/opt/tensorless`; writes `VERSION` and `SHA256SUMS`.                                                                    |

| Build argument | Default       | Meaning                                                                                |
| -------------- | ------------- | -------------------------------------------------------------------------------------- |
| `CUDA_VERSION` | `12.6.3`      | The CUDA base image version.                                                           |
| `UBUNTU`       | `ubuntu24.04` | The base image’s Ubuntu.                                                               |
| `VERSION`      | `dev`         | Written to `/opt/tensorless/VERSION` and the `org.opencontainers.image.version` label. |

### CMake configuration

[Section titled “CMake configuration”](#cmake-configuration)

| Option                                                                        | Value                                                                         | Why                                                  |
| ----------------------------------------------------------------------------- | ----------------------------------------------------------------------------- | ---------------------------------------------------- |
| `CMAKE_BUILD_TYPE`                                                            | `Release`                                                                     |                                                      |
| `GGML_CUDA`                                                                   | `ON`                                                                          | The CUDA backend.                                    |
| `CMAKE_CUDA_ARCHITECTURES`                                                    | `60-real`                                                                     | `sm_60` code only, as measured.                      |
| `GGML_CUDA_FA`                                                                | `ON`                                                                          | Flash attention.                                     |
| `GGML_CUDA_FA_ALL_QUANTS`                                                     | `OFF`                                                                         | Only the KV-cache type pairs below.                  |
| `GGML_CUDA_FA_QUANTS`                                                         | `q4_0-q4_0;q8_0-q8_0;f16-f16;bf16-bf16;f16-q8_0;q8_0-f16;bf16-q8_0;q8_0-bf16` | The measured KV-quant set.                           |
| `GGML_CUDA_GRAPHS`                                                            | `OFF`                                                                         | No CUDA graphs, as measured.                         |
| `GGML_CUDA_NO_VMM`                                                            | `ON`                                                                          | No virtual memory management, as measured.           |
| `GGML_CUDA_COMPRESSION_MODE`                                                  | `size`                                                                        |                                                      |
| `GGML_NATIVE`                                                                 | `OFF`                                                                         | Portable CPU code, independent of the builder’s CPU. |
| `GGML_AVX`, `GGML_AVX2`, `GGML_FMA`, `GGML_F16C`                              | `ON`                                                                          | The x86-64 baseline the CPU side is built for.       |
| `GGML_CCACHE`                                                                 | `OFF`                                                                         |                                                      |
| `LLAMA_BUILD_TESTS`, `LLAMA_BUILD_SERVER`, `LLAMA_BUILD_TOOLS`                | `ON`                                                                          |                                                      |
| `LLAMA_BUILD_EXAMPLES`, `LLAMA_OPENSSL`, `LLAMA_BUILD_UI`, `LLAMA_BUILD_MTMD` | `OFF`                                                                         | No examples, no TLS, no web UI, no multimodal tools. |
| `CMAKE_BUILD_RPATH_USE_ORIGIN`                                                | `ON`                                                                          | The server finds its libraries beside it.            |
| `CMAKE_PLATFORM_NO_VERSIONED_SONAME`                                          | `ON`                                                                          |                                                      |

The research build of the same source used `GGML_NATIVE=ON`; the release build does not. The CUDA side keeps the configuration the hill climbs measured.

To build one yourself, see [Build the engine image yourself](/docs/how-to/advanced/build-engine-image/).

# Engine switches

> The GGML_CUDA_* environment variables the P100 recipes set, what each one selects in the engine, and its default.

The engine’s P100 kernels are opt-in. Each is turned on by a `GGML_CUDA_*` environment variable, which a recipe sets in its `[run.env]` table, exactly as the release gate measured it. The launcher passes them to the container with `-e`, in alphabetical order. Leaving one out, or changing its value, runs a configuration that was not measured.

Both `engine-v0.1.0` recipes set the same 21 switches with the same values. The engine reads each one once, when it is first needed. The source files named below are under [`engine/ggml/src/ggml-cuda/`](https://github.com/iammrduncan/tensorless/tree/main/engine/ggml/src/ggml-cuda).

The PN weight types need no switch: their kernels run whenever a tensor is `pn3t` or `pn4t`.

## Attention

[Section titled “Attention”](#attention)

| Variable                    | Recipe value | Unset | What it selects                                                                                                                                                                                                                           |
| --------------------------- | ------------ | ----- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GGML_CUDA_FATTN_SM60`      | `1`          | off   | A split-KV flash-decoding attention kernel for `sm_60`, for a q4_0 K and V cache with head size 256, six query heads per KV head, and 1 to 4 query tokens. It replaces the stock vector and tile kernels in that case (`fattn-sm60.cu`).  |
| `GGML_CUDA_FATTN_SM60_CTAS` | `4`          | `8`   | How many thread blocks per streaming multiprocessor the split-KV kernel aims for when it splits the KV cache. `4` aims for one full wave of blocks. Used only when `GGML_CUDA_FATTN_SM60` applies.                                        |
| `GGML_CUDA_FATTN_TILE_OCC1` | `1`          | off   | For the tile attention kernel at head size 256 with more than a few query columns (prefill): launch bounds of one block per SM instead of two, which frees registers (`fattn-tile.cuh`). The engine checks only that the variable is set. |

## Matrix-vector products (decode)

[Section titled “Matrix-vector products (decode)”](#matrix-vector-products-decode)

These replace stock `mul_mat_vec_q` paths on compute capability 6.0. All but `GGML_CUDA_SM86_IQ3_SMEM_GRID` live in `mmvh-sm60.cu`, and they take effect only when `GGML_CUDA_MMV_SM60` is on.

| Variable                          | Recipe value | Unset | What it selects                                                                                                                                                                                                                                                         |
| --------------------------------- | ------------ | ----- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GGML_CUDA_MMV_SM60`              | `1`          | off   | The `sm_60` GEMV family. Its base kernel multiplies IQ3_XXS and IQ3_S weights by fp16 activations with codebook tables in shared memory, for 1 to 4 columns, in place of the stock path’s q8_1 activations and emulated `dp4a`.                                         |
| `GGML_CUDA_MMV_SM60_XXS_STOCK`    | `1`          | off   | Leaves IQ3_XXS products on the stock path, which is faster for them. Products of 2 or more columns get the IQ3_XXS repack first, which the stock path then reads.                                                                                                       |
| `GGML_CUDA_MMV_SM60_IQ3S_STOCK`   | `1`          | off   | Leaves single-column (one-token decode) IQ3_S products on the stock path, which is faster there. Wider IQ3_S products stay on the `sm_60` kernel.                                                                                                                       |
| `GGML_CUDA_MMV_SM60_XXS_KB_WIDE`  | `1`          | off   | With `GGML_CUDA_MMV_SM60_XXS_STOCK`, an exception: IQ3_XXS weights that were not repacked go to the `sm_60` kernel at 2 or 3 columns (`mmvh-sm60.cuh`).                                                                                                                 |
| `GGML_CUDA_MMV_SM60_INPLACE_XXS`  | `1`          | off   | Writes the IQ3_XXS repack (wide, aligned records the GEMV reads) in place, over the tensor’s own enlarged storage, instead of in a separate buffer of about 1 GB. Enlarges those tensors’ allocations (`ggml-cuda.cu`).                                                 |
| `GGML_CUDA_MMV_SM60_INPLACE_IQ3S` | `1`          | off   | The same for IQ3_S, instead of a separate buffer of about 2.6 GB.                                                                                                                                                                                                       |
| `GGML_CUDA_MMV_SM60_Q4K_HEAD`     | `1`          | off   | An `sm_60` kernel for Q4_K weights with at least 65,536 rows, at 1 to 4 columns: the output head. Weights decode as fp16 subnormals against power-of-two prescaled activations.                                                                                         |
| `GGML_CUDA_MMV_SM60_Q3K`          | `1`          | off   | The same kind of kernel for Q3_K weights at 1 to 4 columns. The weights are repacked once into 112-byte records (the 110-byte blocks plus 2 pad bytes) in a side buffer.                                                                                                |
| `GGML_CUDA_MMV_SM60_Q6K`          | `1`          | off   | The same kind of kernel for Q6_K weights at 1 or 2 columns, repacked once into 224-byte records.                                                                                                                                                                        |
| `GGML_CUDA_MMV_SM60_Q4K_DHEAD`    | `1`          | off   | For the draft head of MTP speculative decoding: gathers the fixed subset of output-head rows it scores into one contiguous buffer, and runs the head kernel on it. The current recipes use no speculative decoding.                                                     |
| `GGML_CUDA_SM86_IQ3_SMEM_GRID`    | `1`          | off   | Copies the IQ3_XXS and IQ3_S codebook grid into shared memory once per block in the stock `mul_mat_vec_q` path, on compute capability 8.6 and 6.0. Same values and order of operations, so the results are bit-identical (`mmvq.cu`). The variable must start with `1`. |

## Fusion and normalization

[Section titled “Fusion and normalization”](#fusion-and-normalization)

| Variable                 | Recipe value | Unset | What it selects                                                                                                                                                   |
| ------------------------ | ------------ | ----- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GGML_CUDA_FUSE_PRE_ADD` | `1`          | off   | Fuses a residual ADD, the RMS_NORM after it and the MUL after that into one launch, with a check that the fused write cannot overlap a read (`ggml-cuda.cu`).     |
| `GGML_CUDA_NORM_EMIT_Q8` | `1`          | off   | Lets that fused norm also write its output in q8_1, ready for the matrix-vector products that read it, when the shape and every reader allow it (`ggml-cuda.cu`). |

## Prefill

[Section titled “Prefill”](#prefill)

| Variable                          | Recipe value | Unset | What it selects                                                                                                                                                                                                                                   |
| --------------------------------- | ------------ | ----- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GGML_CUDA_PREFILL_DEQ_F4`        | `1`          | off   | Dequantizes weights to fp32 for prefill with one thread per group of 4 weights and one `float4` store: IQ4_XS weights, and, with `GGML_CUDA_PREFILL_REP_DIRECT`, the repacked IQ3_XXS and IQ3_S records (`convert.cu`).                           |
| `GGML_CUDA_PREFILL_REP_DIRECT`    | `1`          | off   | Dequantizes repacked IQ3_XXS and IQ3_S records directly, without rebuilding the original block in shared memory first. Same integers and order of operations, so bit-identical (`convert.cu`).                                                    |
| `GGML_CUDA_PREFILL_CHUNK_GEMM`    | `1`          | off   | For an fp32 matrix product with more than 1,024 columns, issues it as chunks of 1,024 columns plus a remainder (`ggml-cuda.cu`). The engine checks only that the variable is set.                                                                 |
| `GGML_CUDA_PREFILL_CHUNK_STREAMS` | `3`          | `0`   | With chunked GEMM, spreads one weight’s chunks over up to this many worker streams, for shapes with at least 5,120 rows. Each chunk is the same call, so the result is the same as in sequence. `0` turns it off.                                 |
| `GGML_CUDA_SM86_GDN_COLS`         | `8`          | `4`   | Columns of state per block in the gated delta net kernel, the recurrent layers of the model. One of `1`, `2`, `4` or `8`; another value means `4`. `8` applies only to batches of 64 tokens or more; shorter ones use `4` (`gated_delta_net.cu`). |

## Other switches

[Section titled “Other switches”](#other-switches)

The engine reads more `GGML_CUDA_*` variables than these: alternatives that lost, debug output and statistics. They are not part of any recipe, and the release gate did not measure them. The list of a recipe’s switches is in its file, and `tensorless info MODEL` prints them.

# Files and environment variables

> Where tensorless keeps models, verification markers and containers, and every environment variable the launcher and the install script read.

## Files and directories

[Section titled “Files and directories”](#files-and-directories)

| What                       | Where                                                                                                                                                                           |
| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Model cache                | `--cache DIR`, else `$HF_HUB_CACHE`, else `$HF_HOME/hub`, else `~/.cache/huggingface/hub`. The path is made absolute, because the container mounts it. It must not contain `:`. |
| Model file                 | `<cache>/models--<org>--<repo>/blobs/<sha256>`. For example `~/.cache/huggingface/hub/models--iamMrDuncan--Qwen3.8-27B-PN3-GGUF/blobs/15ce2d44…`.                               |
| Snapshot link              | `<cache>/models--<org>--<repo>/snapshots/<revision>/<file>`, a relative symlink to the blob, so other Hugging Face tools find the file.                                         |
| Partial download           | `<blob>.incomplete`. The next `pull` or `run` resumes it.                                                                                                                       |
| Verification markers       | `$XDG_STATE_HOME/tensorless/verified/<sha256>.json`, else `~/.local/state/tensorless/verified/<sha256>.json`.                                                                   |
| Engine images              | The container runtime’s own image storage.                                                                                                                                      |
| Containers                 | `tensorless-<repo id>`: lowercased, with every character other than `a`–`z`, `0`–`9`, `_`, `.` and `-` replaced by `-`. Started with `--rm`, so a stopped container is removed. |
| Model inside the container | `/models/<file name>`, mounted read-only.                                                                                                                                       |
| Recipes                    | Built into the binary. `--recipes DIR` reads `DIR/<org>/<repo>/<card>.toml` instead.                                                                                            |
| The launcher               | Wherever you put it. The install script puts it in `$HOME/.local/bin/tensorless` unless `TENSORLESS_INSTALL_DIR` says otherwise.                                                |

The cache directory names keep the recipe’s spelling of the repo id, whatever case you typed.

### Verification markers

[Section titled “Verification markers”](#verification-markers)

A marker records that a model file was hashed and matched its sha256, so that later commands need not hash 12 GB again. It holds the file’s sha256, size, modification time (in nanoseconds) and absolute path:

\~/.local/state/tensorless/verified/\<sha256>.json (shape)

```text
{"sha256":"…","size":12954114496,"mtime_ns":…,"path":"/home/user/.cache/huggingface/hub/models--iamMrDuncan--Qwen3.8-27B-PN3-GGUF/blobs/15ce2d44…"}
```

A marker counts only while the file’s size, modification time and path are the same. Otherwise the file is hashed again. Deleting the markers is safe: the next command hashes the file once and writes a new one. When the state directory cannot be written, the launcher says so and verifies again on the next run.

## Environment variables

[Section titled “Environment variables”](#environment-variables)

### Launcher

[Section titled “Launcher”](#launcher)

| Variable         | Default                  | Meaning                                                                                                                                         |
| ---------------- | ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| `HF_HUB_CACHE`   | not set                  | The model cache directory. `--cache` takes precedence.                                                                                          |
| `HF_HOME`        | not set                  | When `HF_HUB_CACHE` is not set, the cache is `$HF_HOME/hub`.                                                                                    |
| `HF_ENDPOINT`    | `https://huggingface.co` | The Hugging Face Hub address model files are downloaded from: `<endpoint>/<org>/<repo>/resolve/<revision>/<file>`.                              |
| `HF_TOKEN`       | not set                  | Sent as `Authorization: Bearer` with downloads, for private or gated repositories. It is never printed; `--dry-run` shows only that it is sent. |
| `XDG_STATE_HOME` | `~/.local/state`         | The base of the state directory, `$XDG_STATE_HOME/tensorless`, which holds the verification markers.                                            |

The launcher sets `OMP_NUM_THREADS` and the recipe’s `GGML_CUDA_*` [engine switches](/docs/reference/engine-switches/) inside the container. It does not pass its own environment into the container.

### Install script

[Section titled “Install script”](#install-script)

The install script, `https://tensorless.dev/install.sh`, reads:

| Variable                 | Default            | Meaning                                                             |
| ------------------------ | ------------------ | ------------------------------------------------------------------- |
| `TENSORLESS_VERSION`     | `0.1.0`            | The launcher release to install. Letters, digits, `.` and `-` only. |
| `TENSORLESS_INSTALL_DIR` | `$HOME/.local/bin` | Where to put the `tensorless` binary. Created if missing.           |

It downloads `tensorless-<version>-<platform>.tar.gz` and its `.sha256` from the GitHub release, checks the sha256 with `sha256sum` or `shasum -a 256`, and installs one file, the binary. Platforms: Linux x86_64 (`linux-x86_64`), Linux aarch64 (`linux-aarch64`) and macOS on Apple silicon (`macos-arm64`). It needs `curl` or `wget`, and `sha256sum` or `shasum`.

## Network

[Section titled “Network”](#network)

The launcher sends no telemetry. It uses the network for one thing: downloading model files from the Hub (`HF_ENDPOINT`). The container runtime pulls the engine image from `ghcr.io`. `tensorless bench` talks only to the server at `--host` and `--port`.

# Glossary

> The words tensorless uses, from alias to untuned, each in a sentence or two.

| Term                   | Meaning                                                                                                                                                                                                                                                                 |
| ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Alias**              | The served model name, which `/v1/models` lists and clients send as `model`. A recipe’s `run.alias`; the recipes use the repo id.                                                                                                                                       |
| **Card**               | A GPU model a recipe is written for, with an id such as `p100`. A recipe’s `[card]` table says which GPUs match: compute capability, memory and driver. See [Recipe file](/docs/reference/recipes/#card).                                                               |
| **Champion**           | A change that won a hill-climb campaign: it was faster than the campaign’s best so far, under that campaign’s guards. A champion only has to beat the previous one, so it may not contain every earlier winning change.                                                 |
| **Compute capability** | NVIDIA’s version number for a GPU architecture. The Tesla P100 is 6.0 (`sm_60`). `nvidia-smi` reports it as `compute_cap`.                                                                                                                                              |
| **Context**            | The number of tokens the server keeps for one conversation: prompt and output together. The recipes allocate 65,536 tokens. `--context` changes it and makes the run untuned.                                                                                           |
| **Decode**             | Generating output tokens one at a time, after the prompt has been read. Decode speed is in tokens per second (tok/s). On a P100 it is limited mostly by how fast the weights can be read from memory.                                                                   |
| **Digest**             | The sha256 of a container image’s manifest, such as `sha256:35409ff9…`. A digest names exactly one image; a tag can be moved. Recipes pin images by digest.                                                                                                             |
| **Draft**              | A recipe with `status = "draft"`: it has not passed the release gate. `pull`, `run` and `serve` refuse it unless `--allow-draft`.                                                                                                                                       |
| **Engine**             | The inference engine: a fork of llama.cpp tuned for the P100, in [`engine/`](https://github.com/iammrduncan/tensorless/tree/main/engine). It runs as `llama-server` inside the engine image.                                                                            |
| **Engine release**     | A tagged engine version, `engine-vX.Y.Z`, such as `engine-v0.1.0`: the exact source in `engine/` at that tag, and its build configuration. Versioned separately from the launcher.                                                                                      |
| **Engine switch**      | A `GGML_CUDA_*` environment variable that turns on one of the engine’s P100 kernels or changes its setting. A recipe sets them in `[run.env]`, exactly as measured. See [Engine switches](/docs/reference/engine-switches/).                                            |
| **Hill climb**         | An automated campaign in which AI agents propose changes to the engine, and a judge they cannot change measures each one and keeps only the ones that are faster and pass the guards. The engine-v0.1.0 decode climb made 131 judged attempts and kept 28 improvements. |
| **Image**              | One build of an engine release for one GPU architecture, published as `ghcr.io/iammrduncan/tensorless-engine:<X.Y.Z>-<arch>`. See [Engine image](/docs/reference/engine-image/).                                                                                        |
| **Judge**              | The measuring program of a hill climb. It builds the candidate, runs it on the card, times it from outside the engine and checks its numerics. Agents cannot see or change it.                                                                                          |
| **Launch target**      | A model you can launch: one released recipe. See [Launch targets](/docs/reference/launch-targets/).                                                                                                                                                                     |
| **Launcher**           | The `tensorless` command. It holds no inference code; it reads recipes, fetches and checks the image and the model, and runs the container.                                                                                                                             |
| **MTP**                | Multi-token prediction: a form of speculative decoding in which a small draft head proposes tokens that the model then checks. Not in the v0.1.0 recipes.                                                                                                               |
| **P0k+19**             | The tensor mix of Qwen3.8-27B-PN3: which tensors use `pn3t`, which use `pn4t`, and which keep standard types. See [PN weight format](/docs/reference/pn-format/#files-that-use-pn).                                                                                     |
| **Pascal**             | NVIDIA’s GPU architecture of 2016, which includes the Tesla P100. CUDA 13 dropped it, and the R580 driver branch is its last.                                                                                                                                           |
| **PN**                 | The P100-native block weight format of this engine, PNX-FORMAT-v0. Every block is 34 bytes and decodes exactly in fp32. See [PN weight format](/docs/reference/pn-format/).                                                                                             |
| **pn3t**               | The 3-bit PN type, ggml type 124: 80 weights in 34 bytes, 3.40 bits per weight.                                                                                                                                                                                         |
| **pn4t**               | The 4-bit PN type, ggml type 125: 64 weights in 34 bytes, 4.25 bits per weight.                                                                                                                                                                                         |
| **Prefill**            | Reading the prompt before the first output token. Prefill speed is in prompt tokens per second.                                                                                                                                                                         |
| **Q3Floor**            | Short for Qwen3.8-27B-UD-IQ3_XXS-Q3Floor: Unsloth’s UD-IQ3_XXS quantization of Qwen3.8-27B with its 91 sub-3-bit tensors raised to IQ3_XXS, a 3-bit floor, and its token embedding replaced by the Q3_K one from Unsloth’s UD-IQ4_XS.                                   |
| **Receipt**            | What the release gate measured for a recipe, with the scope of each number and the image digest and model hash it was measured on. See [Receipts](/docs/reference/receipts/).                                                                                           |
| **Recipe**             | One model on one card, `recipes/<org>/<repo>/<card>.toml`: the image digest, the model’s repository, revision, file and sha256, the engine switches and server flags, the hardware requirements and the measured numbers. See [Recipe file](/docs/reference/recipes/).  |
| **Release candidate**  | An engine image built for measuring, tagged `X.Y.Z-rcN-<arch>`. When it passes the release gate, its digest gets the release tag without a rebuild.                                                                                                                     |
| **Release gate**       | The checks a candidate image must pass for every recipe on every card before it is released: build checks, numerics against the previous release, speed, long-context decode and prefill, and quality checks.                                                           |
| **Released**           | A recipe with `status = "released"`, whose image was measured on its card. `pull`, `run` and `serve` accept it with no extra flag.                                                                                                                                      |
| **Repo id**            | A Hugging Face repository name, `org/repo`, such as `iamMrDuncan/Qwen3.8-27B-PN3-GGUF`. tensorless names models by repo id, and matches it without case.                                                                                                                |
| **Short context**      | The decode measurement at a short prompt: 256 greedy tokens after a prompt of about 600 tokens in the research receipts. Its receipt key is `short`.                                                                                                                    |
| **Short name**         | A recipe’s optional `model.name`, such as `pn3` or `q3floor`, which works anywhere a repo id does.                                                                                                                                                                      |
| **sm_60**              | The CUDA target for compute capability 6.0, the Tesla P100. Written `sm60` in image tags and recipes.                                                                                                                                                                   |
| **TTFT**               | Time to first token: from sending a request to receiving the first generated token. It includes prefill.                                                                                                                                                                |
| **UD**                 | Unsloth Dynamic, Unsloth’s name for its mixed-precision quantizations, such as UD-IQ3_XXS.                                                                                                                                                                              |
| **Unpublished**        | A recipe whose revision, sha256 or image digest is all zeros, or whose `size_bytes` is 0. It cannot be pulled or run, even with `--allow-draft`.                                                                                                                        |
| **Untuned**            | A run that differs from what the recipe measured: another `--context`, extra arguments after `--`, or another `@REVISION`. The launcher labels it, and the receipt’s numbers do not apply.                                                                              |

# Launch targets

> Every model tensorless can launch in this release: its files, hashes, image, settings and measured speed.

A launch target is one recipe: one model on one card, released after its image was measured on that card. tensorless v0.1.0 has two, both on the NVIDIA Tesla P100 16 GB and both on `engine-v0.1.0`.

| Launch command                                                   | Short name | Decode, published image | Decode, research receipt |
| ---------------------------------------------------------------- | ---------- | ----------------------: | -----------------------: |
| `tensorless run iamMrDuncan/Qwen3.8-27B-PN3-GGUF`                | `pn3`      |         **30.03 tok/s** |              30.06 tok/s |
| `tensorless run iamMrDuncan/Qwen3.8-27B-UD-IQ3_XXS-Q3Floor-GGUF` | `q3floor`  |         **26.18 tok/s** |              26.26 tok/s |

Both decode numbers are greedy, single-stream, at short context. “Published image” is `tensorless bench` on the released image: the median of 3 runs of 256 tokens after a 429-token prompt, on one Tesla P100-PCIE-16GB. “Research receipt” is the decode hill climb’s judge on a source build of the same engine commit: 256 tokens after a cold prompt of about 600 tokens. The full measurements are in [Receipts](/docs/reference/receipts/).

`tensorless models` lists the recipes your launcher knows, and `tensorless info MODEL` shows any of them in full.

## Qwen3.8-27B-PN3

[Section titled “Qwen3.8-27B-PN3”](#qwen38-27b-pn3)

Qwen3.8-27B in PN, the P100-native 3-bit weight format of this engine (mix P0k+19). Stock llama.cpp cannot load it. See [PN weight format](/docs/reference/pn-format/).

| Field          | Value                                                                                                                                                          |
| -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Repository     | [`iamMrDuncan/Qwen3.8-27B-PN3-GGUF`](https://huggingface.co/iamMrDuncan/Qwen3.8-27B-PN3-GGUF)                                                                  |
| Short name     | `pn3`                                                                                                                                                          |
| Revision       | `c701b6690ec360b16d40f85df1cb76062e30cd38`                                                                                                                     |
| File           | `Qwen3.8-27B-PN3-P0k19.gguf`                                                                                                                                   |
| Size           | 12,954,114,496 bytes (12.95 GB, 12.1 GiB)                                                                                                                      |
| sha256         | `15ce2d4469673f1f1753c55de3780cfe04311dd6f05cb41633d73df128e91476`                                                                                             |
| License        | Apache-2.0 (from Qwen/Qwen3.8-27B)                                                                                                                             |
| Card           | `p100`: compute capability 6.0, at least 16,000 MiB, driver 525.60.13 or newer                                                                                 |
| Engine release | `engine-v0.1.0`, `sm60`                                                                                                                                        |
| Image          | `ghcr.io/iammrduncan/tensorless-engine@sha256:35409ff97c2f518e468790c28bd19206ae777a945fcffe5775585e28fd9939a7` (tag `0.1.0-sm60`)                             |
| Context        | 65,536 tokens, q4_0 KV cache                                                                                                                                   |
| Recipe         | [`recipes/iamMrDuncan/Qwen3.8-27B-PN3-GGUF/p100.toml`](https://github.com/iammrduncan/tensorless/blob/main/recipes/iamMrDuncan/Qwen3.8-27B-PN3-GGUF/p100.toml) |

| Measure                     |                       Value | Scope                                                                     |
| --------------------------- | --------------------------: | ------------------------------------------------------------------------- |
| Decode, published image     |                 30.03 tok/s | `tensorless bench`, median of 3 runs, 256 tokens after a 429-token prompt |
| Decode, short context       |                 30.06 tok/s | research receipt; code prompt (prose: 30.18)                              |
| Prefill, about 4,100 tokens |                 121.4 tok/s | research receipt; code prompt (prose: 119.6)                              |
| Decode at 32K and 65K       | not measured on this engine |                                                                           |

## Qwen3.8-27B-UD-IQ3_XXS-Q3Floor

[Section titled “Qwen3.8-27B-UD-IQ3_XXS-Q3Floor”](#qwen38-27b-ud-iq3_xxs-q3floor)

Unsloth’s Qwen3.8-27B UD-IQ3_XXS with a 3-bit floor: its 91 sub-3-bit tensors re-quantized to IQ3_XXS, and `token_embd` taken from Unsloth’s UD-IQ4_XS. It uses standard llama.cpp types, and everything else in the file is Unsloth’s, byte for byte. It is the model the P100 kernels were first tuned on, and the comparison model of every Qwen3.8-27B-PN3 measurement. See [Models built on someone else’s file](/docs/explanation/models-built-on-others/).

| Field          | Value                                                                                                                                                                                        |
| -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Repository     | [`iamMrDuncan/Qwen3.8-27B-UD-IQ3_XXS-Q3Floor-GGUF`](https://huggingface.co/iamMrDuncan/Qwen3.8-27B-UD-IQ3_XXS-Q3Floor-GGUF)                                                                  |
| Built from     | [`unsloth/Qwen3.8-27B-GGUF`](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF) at `4ca720788d1e01f1bff70c033e0d0028fd02e502`, `Qwen3.8-27B-UD-IQ3_XXS.gguf`                                   |
| Short name     | `q3floor`                                                                                                                                                                                    |
| Revision       | `11f76ec7e1f7be8e76563ee2b9a912a8519e9801`                                                                                                                                                   |
| File           | `Qwen3.8-27B-UD-IQ3_XXS-Q3Floor.gguf`                                                                                                                                                        |
| Size           | 11,758,422,944 bytes (11.76 GB, 11.0 GiB)                                                                                                                                                    |
| sha256         | `4b4d6a39dca07c915472b3fdf70267385f6e6f3f38befd5223c9a1ac272cecd0`                                                                                                                           |
| License        | Apache-2.0 (from Qwen/Qwen3.8-27B and unsloth/Qwen3.8-27B-GGUF)                                                                                                                              |
| Card           | `p100`: compute capability 6.0, at least 16,000 MiB, driver 525.60.13 or newer                                                                                                               |
| Engine release | `engine-v0.1.0`, `sm60`                                                                                                                                                                      |
| Image          | `ghcr.io/iammrduncan/tensorless-engine@sha256:35409ff97c2f518e468790c28bd19206ae777a945fcffe5775585e28fd9939a7` (tag `0.1.0-sm60`)                                                           |
| Context        | 65,536 tokens, q4_0 KV cache                                                                                                                                                                 |
| Recipe         | [`recipes/iamMrDuncan/Qwen3.8-27B-UD-IQ3_XXS-Q3Floor-GGUF/p100.toml`](https://github.com/iammrduncan/tensorless/blob/main/recipes/iamMrDuncan/Qwen3.8-27B-UD-IQ3_XXS-Q3Floor-GGUF/p100.toml) |

| Measure                 |               Value | Scope                                                                                                                         |
| ----------------------- | ------------------: | ----------------------------------------------------------------------------------------------------------------------------- |
| Decode, published image |         26.18 tok/s | `tensorless bench`, median of 3 runs, 256 tokens after a 429-token prompt                                                     |
| Decode, short context   |         26.26 tok/s | research receipt on this engine commit; code prompt (prose: 26.27)                                                            |
| Decode at 32K / 65K     | 24.52 / 22.92 tok/s | research receipt on the climb’s starting commit; 1,024 tokens after a 31.7K / 64.5K-token prompt, code (prose: 24.54 / 22.97) |

## Settings both recipes share

[Section titled “Settings both recipes share”](#settings-both-recipes-share)

Both recipes run the same image with the same arguments and the same 21 [engine switches](/docs/reference/engine-switches/). The launcher adds the model path, host, port, context, alias and thread counts.

| `llama-server` argument       | Meaning                                                                                               |
| ----------------------------- | ----------------------------------------------------------------------------------------------------- |
| `-np 1`                       | One request slot.                                                                                     |
| `-ngl 99`                     | Every layer on the GPU.                                                                               |
| `-ot token_embd.weight=CUDA0` | The token embedding on the GPU too.                                                                   |
| `-ctk q4_0 -ctv q4_0`         | A q4_0 KV cache.                                                                                      |
| `-sm none`                    | One GPU, no split.                                                                                    |
| `-lm none`                    | Load mode `none`: no memory-mapping or locking of the model file.                                     |
| `-fit off`                    | Leaves unset arguments as they are, instead of adjusting them to fit in device memory.                |
| `--threads-http 4`            | Four HTTP threads.                                                                                    |
| `-b 512 -ub 512`              | Batch and micro-batch of 512 tokens.                                                                  |
| `-fa on`                      | Flash attention on.                                                                                   |
| `--jinja`                     | Applies the model’s chat template for `/v1/chat/completions`. Not part of the measured configuration. |

| Recipe setting | Value     |
| -------------- | --------- |
| `context`      | 65536     |
| `threads_max`  | 8         |
| `min_driver`   | 525.60.13 |

The model and its 65,536-token context take most of the card’s 16 GB: the published image used 13,997 MiB idle and 14,577 MiB generating with Qwen3.8-27B-PN3, and 13,157 / 13,737 MiB with Q3Floor.

# PN weight format

> PNX-FORMAT-v0: the pn3t and pn4t block layouts, how a value decodes, the reference quantizer, validation, and the files that use them.

PN is a family of block-quantized weight types designed for the NVIDIA Tesla P100 (`sm_60`). This page describes version 0 of the format, PNX-FORMAT-v0, and its two types, which Qwen3.8-27B-PN3 uses. Why the format looks like this is in [Why a P100-native weight format](/docs/explanation/pn-format/).

The implementation in the engine is normative:

* the layout: [`engine/ggml/src/ggml-common.h`](https://github.com/iammrduncan/tensorless/blob/main/engine/ggml/src/ggml-common.h);
* decoding, the reference quantizer and validation: [`engine/ggml/src/ggml-quants.c`](https://github.com/iammrduncan/tensorless/blob/main/engine/ggml/src/ggml-quants.c) (`dequantize_row_pn3t`, `dequantize_row_pn4t`, `quantize_row_pnt_ref`);
* the golden tests: [`engine/tests/test-pn-format.cpp`](https://github.com/iammrduncan/tensorless/blob/main/engine/tests/test-pn-format.cpp).

## Types

[Section titled “Types”](#types)

| ggml type id | Name   | Weights per block | Bytes per block | Bits per weight | Codes per u32 word          | Zero point `z` |
| -----------: | ------ | ----------------: | --------------: | --------------: | --------------------------- | -------------: |
|          124 | `pn3t` |                80 |              34 |            3.40 | 10 (5 pairs of 3-bit codes) |              4 |
|          125 | `pn4t` |                64 |              34 |            4.25 | 8 (4 pairs of 4-bit codes)  |              8 |

The type ids are specific to this engine’s ggml fork. Stock llama.cpp does not know them and cannot load a PN file.

## Block layout

[Section titled “Block layout”](#block-layout)

Every block is 34 bytes:

```text
byte 0..1    d       fp16, little-endian, SIGNED scale
byte 2..33   qs[32]  eight little-endian u32 words, w0..w7
```

Element `e` of a block (counting from 0) is in word `e / 10` for `pn3t` and `e / 8` for `pn4t`. Inside the word, its local index is `t = e mod 10` (`pn3t`) or `t = e mod 8` (`pn4t`):

| Type   | Even `t = 2j`     | Odd `t = 2j + 1`        | Unused bits                     |
| ------ | ----------------- | ----------------------- | ------------------------------- |
| `pn3t` | bits `3j .. 3j+2` | bits `16+3j .. 16+3j+2` | 15 and 31 (reserved, must be 0) |
| `pn4t` | bits `4j .. 4j+3` | bits `16+4j .. 16+4j+3` | none                            |

So the shift of code `t` is:

```text
shift = BITS * (t >> 1) + 16 * (t & 1)
```

Even codes fill the low 16-bit half of the word and odd codes the high half. No code crosses the halves.

Block `b` starts at byte `34 * b`. The scale is always 2-byte aligned, and a word is 4-byte aligned only in even blocks. Readers must not assume that a word inside a block is 4-byte aligned.

## Value

[Section titled “Value”](#value)

```text
w = d * (n - z)        n = the code (0..7 for pn3t, 0..15 for pn4t), z = 2^(bits-1)
```

| Type   | Codes decode to |
| ------ | --------------- |
| `pn3t` | `-4d .. +3d`    |
| `pn4t` | `-8d .. +7d`    |

* Zero is a code (`n = z`), so the types are mid-tread.
* Both steps are exact in fp32: `d` has 11 significant bits and `n - z` at most 4.
* The sign of a zero result is the sign of `d`: code `z` with a negative `d` decodes to `-0.0`.

## Reference quantizer

[Section titled “Reference quantizer”](#reference-quantizer)

For each group of 80 (`pn3t`) or 64 (`pn4t`) weights:

1. Compute `amax`, `max` and `min` in double precision. A group with `amax == 0` is the canonical zero group: `d = +0.0`, and every code is `z`.
2. Let `sigma = +1` if `|min| > max`, else `-1`. The largest-magnitude weight then maps to code 0, at the end with the larger magnitude.
3. For each `m` in `{0.50, 0.52, ..., 1.20}` (36 values), take `s = fp16(sigma * m * amax / z)`, rounded to fp16 before it is scored, and skip `s == 0`. The codes are `n = clamp(rint(w / s) + z, 0, 2z - 1)`, and the error is `sum (w - s (n - z))^2`, in double precision.
4. Keep the `s` with the smallest error. A tie goes to the larger `m`.
5. Write `d = s` and pack the codes as above, with the reserved bits 0.

## Validation

[Section titled “Validation”](#validation)

A row is invalid when:

* any block’s `d` is not finite (an fp16 NaN or infinity); or
* for `pn3t`, bit 15 or bit 31 of any word is set.

## Files that use PN

[Section titled “Files that use PN”](#files-that-use-pn)

A PN model is a GGUF file whose weight tensors use types 124 and 125 next to standard ggml types.

Qwen3.8-27B-PN3 is Qwen3.8-27B with mix **P0k+19**:

| Tensors                                                                | Type   |
| ---------------------------------------------------------------------- | ------ |
| Every matrix-vector tensor with K = 5120, except `attn_k` and `attn_v` | `pn3t` |
| `attn_k`, `attn_v`, and the tensors with K = 6144 and K = 17408        | `pn4t` |
| 19 more tensors, promoted                                              | `pn4t` |
| `token_embd`                                                           | Q3_K   |
| `output`                                                               | Q4_K   |
| `ssm_alpha`, `ssm_beta`                                                | Q8_0   |
| Norms                                                                  | F32    |

Qwen3.8-27B-PN3 also applies a Hadamard rotation to the activations that reach the PN weights. The engine performs it.

## Where PN runs

[Section titled “Where PN runs”](#where-pn-runs)

| Path                                                                | State in `engine-v0.1.0`                                                                                                                                                                                      |
| ------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| CUDA, `sm_60`                                                       | Correct. PN tensors use their own GEMV kernels (`ggml-cuda/pn-sm60.cu`) with no switch needed.                                                                                                                |
| CPU reference decode (`dequantize_row_pn3t`, `dequantize_row_pn4t`) | Correct.                                                                                                                                                                                                      |
| CPU dot product (`ggml_vec_dot_pn_t_q8_0`)                          | **Wrong**: it shifts by element index instead of pair index and decodes most PN weights wrong. Do not run PN models on the CPU backend or with partial offload. Every recipe runs the whole model on the GPU. |

The defect is recorded in [`engine/PROVENANCE.md`](https://github.com/iammrduncan/tensorless/blob/main/engine/PROVENANCE.md).

# Receipts

> What was measured for each engine-v0.1.0 recipe, by whom and on what: speed, quality checks and the published image's own run.

A receipt is what the release gate measured for a recipe: speed at stated contexts, quality checks, and the image digest and model hash they were taken on. Each number below comes with its scope. The source files are in [`docs/receipts/engine-v0.1.0/`](https://github.com/iammrduncan/tensorless/tree/main/docs/receipts/engine-v0.1.0).

| Recipe                                                      | Receipt file                                                                                                                                                               |
| ----------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `iamMrDuncan/Qwen3.8-27B-PN3-GGUF` on `p100`                | [Qwen3.8-27B-PN3-GGUF-p100.md](https://github.com/iammrduncan/tensorless/blob/main/docs/receipts/engine-v0.1.0/Qwen3.8-27B-PN3-GGUF-p100.md)                               |
| `iamMrDuncan/Qwen3.8-27B-UD-IQ3_XXS-Q3Floor-GGUF` on `p100` | [Qwen3.8-27B-UD-IQ3_XXS-Q3Floor-GGUF-p100.md](https://github.com/iammrduncan/tensorless/blob/main/docs/receipts/engine-v0.1.0/Qwen3.8-27B-UD-IQ3_XXS-Q3Floor-GGUF-p100.md) |

## Two kinds of measurement

[Section titled “Two kinds of measurement”](#two-kinds-of-measurement)

| Kind             | Who measured                  | On what                                                                                              | How                                                                                                                 |
| ---------------- | ----------------------------- | ---------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| Research receipt | The decode hill climb’s judge | A source build of the engine commit, with the same CUDA configuration as the image, on one P100      | Timed from outside the engine: each streamed token is stamped on the judge’s own clock. Greedy decoding.            |
| Published image  | `tensorless bench`            | The released image digest, with exactly the recipe’s switches and flags, on one Tesla P100-PCIE-16GB | The median of 3 runs of 256 greedy tokens after a 429-token prompt. See [Command line](/docs/reference/cli/#bench). |

The two use different prompts, so a few percent of difference between them is expected.

## Qwen3.8-27B-PN3

[Section titled “Qwen3.8-27B-PN3”](#qwen38-27b-pn3)

The engine source of v0.1.0 is the champion of an automated decode hill climb: research commit `cfcab96e`, exported unchanged except for six comment lines (see [`engine/PROVENANCE.md`](https://github.com/iammrduncan/tensorless/blob/main/engine/PROVENANCE.md)).

### Speed, research receipt

[Section titled “Speed, research receipt”](#speed-research-receipt)

| Measure                                 |     Qwen3.8-27B-PN3 | Q3Floor, same card, same cells | Scope                                                                                          |
| --------------------------------------- | ------------------: | -----------------------------: | ---------------------------------------------------------------------------------------------- |
| Decode, code prompt                     |     **30.06 tok/s** |                          26.30 | 256 greedy tokens after a cold prompt of about 600 tokens; the median of 3 confirmation blocks |
| Decode, prose prompt                    |     **30.18 tok/s** |                          26.31 | the same, prose prompt                                                                         |
| Decode, four days later on another card | 29.95 / 29.96 tok/s |                  26.26 / 26.27 | the same cells, the baseline of the next campaign                                              |
| Prefill, code / prose                   | 121.4 / 119.6 tok/s |                              — | a cold prompt of about 4,100 tokens, 8 micro-batches of 512                                    |
| Decode at the start of the climb        |   7.74 / 7.73 tok/s |                  26.30 / 26.31 | the same cells: 3.89× over the climb                                                           |

### Numerics held during the climb

[Section titled “Numerics held during the climb”](#numerics-held-during-the-climb)

Every step of the climb kept Qwen3.8-27B-PN3’s numerics where they started:

| Check                                                                  | Bar held                           |
| ---------------------------------------------------------------------- | ---------------------------------- |
| KL divergence to the starting point’s outputs, 128 forced prose tokens | within 2.593e-3                    |
| Same top token as the starting point                                   | at 95% or more of positions        |
| Decode-path perplexity                                                 | within 0.25%                       |
| Q3Floor’s greedy text                                                  | bit-identical                      |
| PN golden blocks, and one real tensor of each PN class                 | decoded bit-exactly on CPU and GPU |

### Quality, measured on the climb’s starting point

[Section titled “Quality, measured on the climb’s starting point”](#quality-measured-on-the-climbs-starting-point)

The climb held quality at its starting point. That starting point was gated against Q3Floor on one P100, each model with its own file:

| Check                                                          | Qwen3.8-27B-PN3 | Q3Floor | Bar                                              |
| -------------------------------------------------------------- | --------------: | ------: | ------------------------------------------------ |
| Perplexity, context 2,048 (6 chunks)                           |          2.5494 |  2.5682 | ≤ 1.005 × Q3Floor: passes (0.993×)               |
| Decode-path perplexity, code at 32K (512 forced tokens)        |         1.38147 | 1.37773 | ≤ 1.005 × Q3Floor: passes (1.003×)               |
| Decode-path perplexity, prose at 32K                           |         6.07793 | 5.65357 | ≤ 1.005 × Q3Floor: **misses (1.075×)**, accepted |
| Needle recall at 1K–31K (32K context) and 1K–63K (65K context) |          8 of 8 |   exact | exact                                            |
| Task screen (26 cases)                                         |              23 |      23 | at most 1 fewer than Q3Floor                     |

The prose miss belongs to the quantization, not the kernels: the format’s own fp32 reference path scores 6.07572, so the fast kernels account for 0.5% of the gap. It was accepted.

### The published image (2026-11-02)

[Section titled “The published image (2026-11-02)”](#the-published-image-2026-11-02)

`ghcr.io/iammrduncan/tensorless-engine@sha256:35409ff97c2f518e468790c28bd19206ae777a945fcffe5775585e28fd9939a7`, measured as `0.1.0-rc1`, on one Tesla P100-PCIE-16GB that reported compute capability 6.0 and 16,269 MiB, with the model file checked against sha256 `15ce2d44…`:

| Measure                                                                           |                                   Value |
| --------------------------------------------------------------------------------- | --------------------------------------: |
| `tensorless bench` decode, median of 3 runs (256 tokens after a 429-token prompt) | **30.03 tok/s** (30.03 / 29.83 / 30.13) |
| Against the research receipt (30.06)                                              |                                   −0.1% |
| `llama-server`’s own decode timer, 3 runs of 256 tokens after a 45-token prompt   |             29.97 / 29.94 / 29.94 tok/s |
| Time to first token, 429-token prompt                                             |                         4.46 s (median) |
| GPU memory, idle / generating                                                     |                     13,997 / 14,577 MiB |
| Load to listening, page cache warm                                                |                                   3.6 s |

`bench` ran on another computer, through Wi-Fi, ssh and a port-forward to the server. The network adds latency to the time to first token, but not to the decode rate, which is timed between streamed tokens.

### Not measured yet

[Section titled “Not measured yet”](#not-measured-yet)

* **32K and 65K context on this engine.** At those contexts only the climb’s starting point was measured: 7.55 / 7.39 tok/s for Qwen3.8-27B-PN3, against 24.52 / 22.92 for Q3Floor. Attention costs a larger share of a 30 tok/s token than of a 7.5 tok/s one, so expect this engine’s long-context decode to be below 30.
* **Speculative decoding** for Qwen3.8-27B-PN3.

## Qwen3.8-27B-UD-IQ3_XXS-Q3Floor

[Section titled “Qwen3.8-27B-UD-IQ3_XXS-Q3Floor”](#qwen38-27b-ud-iq3_xxs-q3floor)

### The published image (2026-11-02)

[Section titled “The published image (2026-11-02)”](#the-published-image-2026-11-02-1)

The same image digest, measured as `0.1.0-rc1` and released as `0.1.0-sm60`, on one Tesla P100-PCIE-16GB, with exactly the recipe’s switches and flags. The model file was checked against sha256 `4b4d6a39…`, the same bytes as the Hugging Face upload at revision `11f76ec7`.

| Measure                                                                           |                                   Value |
| --------------------------------------------------------------------------------- | --------------------------------------: |
| `tensorless bench` decode, median of 3 runs (256 tokens after a 429-token prompt) | **26.18 tok/s** (26.16 / 26.18 / 26.19) |
| Against the research receipt (26.26)                                              |                                   −0.3% |
| `llama-server`’s own decode timer, 3 runs of 256 tokens after a 45-token prompt   |             26.16 / 26.19 / 26.21 tok/s |
| Time to first token, 429-token prompt                                             |                         3.85 s (median) |
| GPU memory, idle / generating                                                     |                     13,157 / 13,737 MiB |
| Load to listening, page cache warm                                                |                                   3.5 s |

### Research receipt

[Section titled “Research receipt”](#research-receipt)

The hill climb’s judge decoded this model in every block, as the comparison model for Qwen3.8-27B-PN3.

| Measure               |        Code |       Prose | Scope                                                                                  |
| --------------------- | ----------: | ----------: | -------------------------------------------------------------------------------------- |
| Decode, short context | 26.26 tok/s | 26.27 tok/s | 256 greedy tokens after a cold prompt of about 600 tokens, on this engine commit       |
| Decode at 32K         | 24.52 tok/s | 24.54 tok/s | 1,024 greedy tokens after a prompt filling the context, on the climb’s starting commit |
| Decode at 65K         | 22.92 tok/s | 22.97 tok/s | the same                                                                               |

Its greedy text stayed bit-identical through the whole climb, and its speed was guarded within 3%.

### Not measured yet

[Section titled “Not measured yet”](#not-measured-yet-1)

* 32K and 65K context on this engine commit.
* Speculative decoding (MTP) on this engine. The earlier kernel campaign ran it at about 28 tok/s on its own engine version; that mode is not in this recipe.

# Recipe file

> The recipe file, schema 1: every table and key, the checks the launcher applies, and what makes a recipe a draft or unpublished.

A recipe is one model on one card. It names the exact engine image, the exact model file, the settings that were measured, and the numbers they gave. The launcher only runs what a recipe describes.

## Location

[Section titled “Location”](#location)

```text
recipes/<org>/<repo>/<card>.toml
```

* `<org>/<repo>` is the model’s Hugging Face repository, and must equal `model.hf_repo`.
* `<card>` is the card id, and must equal `card.id`.
* The built-in recipes are the repository’s [`recipes/`](https://github.com/iammrduncan/tensorless/tree/main/recipes) directory, embedded into the launcher when it is built. `--recipes DIR` reads `DIR/<org>/<repo>/<card>.toml` instead.
* Other files, and files and directories whose name starts with `.`, are ignored.
* One invalid recipe fails the whole load.

## Example

[Section titled “Example”](#example)

recipes/iamMrDuncan/Qwen3.8-27B-PN3-GGUF/p100.toml (part)

```toml
schema = 1
status = "released"


[model]
name       = "pn3"
title      = "Qwen3.8-27B-PN3 (P100-native 3-bit, mix P0k+19)"
hf_repo    = "iamMrDuncan/Qwen3.8-27B-PN3-GGUF"
revision   = "c701b6690ec360b16d40f85df1cb76062e30cd38"
file       = "Qwen3.8-27B-PN3-P0k19.gguf"
sha256     = "15ce2d4469673f1f1753c55de3780cfe04311dd6f05cb41633d73df128e91476"
size_bytes = 12954114496
license    = "Apache-2.0 (from Qwen/Qwen3.8-27B)"


[card]
id                 = "p100"
title              = "NVIDIA Tesla P100 16 GB"
compute_capability = "6.0"
min_memory_mib     = 16000
min_driver         = "525.60.13"


[engine]
release = "engine-v0.1.0"
arch    = "sm60"
image   = "ghcr.io/iammrduncan/tensorless-engine@sha256:35409ff97c2f518e468790c28bd19206ae777a945fcffe5775585e28fd9939a7"


[run]
alias       = "iamMrDuncan/Qwen3.8-27B-PN3-GGUF"
context     = 65536
threads_max = 8
args = ["-np", "1", "-ngl", "99", "-ot", "token_embd.weight=CUDA0", "-ctk", "q4_0", "-ctv", "q4_0",
        "-sm", "none", "-lm", "none", "-fit", "off", "--threads-http", "4", "-b", "512", "-ub", "512",
        "-fa", "on", "--jinja"]


[run.env]
GGML_CUDA_FATTN_SM60 = "1"
# ... the other engine switches


[receipt]
summary     = "Decode about 30 tok/s at short context, 1.14x the Q3 baseline model on the same card. Not yet measured at 32K or 65K."
decode_tps  = { short = 30.06 }
prefill_tps = { ctx_4k = 121.4 }
```

The full files are in [`recipes/`](https://github.com/iammrduncan/tensorless/tree/main/recipes). The schema’s own description is [recipes/README.md](https://github.com/iammrduncan/tensorless/blob/main/recipes/README.md).

## Top level

[Section titled “Top level”](#top-level)

| Key      | Type    | Required | Meaning                                                                                            |
| -------- | ------- | -------- | -------------------------------------------------------------------------------------------------- |
| `schema` | integer | yes      | The recipe schema. Must be `1`; any other value fails with a request to update tensorless.         |
| `status` | string  | yes      | `"draft"` or `"released"`. See [Draft, released and unpublished](#draft-released-and-unpublished). |

## \[model]

[Section titled “\[model\]”](#model)

| Key          | Type    | Required | Meaning                                                                                                                                                                       |
| ------------ | ------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name`       | string  | no       | A short alias, such as `pn3`, that works anywhere a repo id does. Lowercase letters, digits, `.`, `_` and `-`. It must not equal any repo id, nor another model’s short name. |
| `title`      | string  | no       | A one-line human name, shown by `info` and `models --json`.                                                                                                                   |
| `hf_repo`    | string  | yes      | The Hugging Face model repository, `org/name`. This is what users type, matched without case. Two repo ids that differ only in case are refused.                              |
| `revision`   | string  | yes      | The repository commit: 40 lowercase hex characters. Never a branch or tag name.                                                                                               |
| `file`       | string  | yes      | The weights file’s path inside the repository: relative, with forward slashes, clean (no `..`), without `:` or newlines.                                                      |
| `sha256`     | string  | yes      | The file’s sha256, 64 lowercase hex characters. The launcher checks the download against it.                                                                                  |
| `size_bytes` | integer | yes      | The file’s exact size, for the disk-space check and progress. Must not be negative. `0` marks the recipe unpublished.                                                         |
| `license`    | string  | yes      | The model’s license.                                                                                                                                                          |

## \[card]

[Section titled “\[card\]”](#card)

| Key                  | Type    | Required | Meaning                                                                                                             |
| -------------------- | ------- | -------- | ------------------------------------------------------------------------------------------------------------------- |
| `id`                 | string  | yes      | The card id, such as `p100`. Lowercase letters, digits, `.`, `_` and `-`. Must equal the file name without `.toml`. |
| `title`              | string  | no       | The card’s human name, such as `NVIDIA Tesla P100 16 GB`.                                                           |
| `compute_capability` | string  | yes      | Must equal the `compute_cap` that `nvidia-smi` reports, such as `"6.0"`.                                            |
| `min_memory_mib`     | integer | yes      | The GPU’s `memory.total`, in MiB, must be at least this. Positive.                                                  |
| `min_driver`         | string  | yes      | The oldest NVIDIA driver the recipe runs on, a dotted version such as `"525.60.13"`.                                |

A GPU matches a card when its compute capability equals `compute_capability` and its memory is at least `min_memory_mib`. A matching GPU with an older driver is refused, with the driver to install.

## \[engine]

[Section titled “\[engine\]”](#engine)

| Key       | Type   | Required | Meaning                                                                       |
| --------- | ------ | -------- | ----------------------------------------------------------------------------- |
| `release` | string | yes      | The engine release the recipe was measured on, such as `engine-v0.1.0`.       |
| `arch`    | string | yes      | The GPU architecture of the image, such as `sm60`.                            |
| `image`   | string | yes      | The engine image, always by digest: `name@sha256:<64 hex>`. A tag is refused. |

## \[run]

[Section titled “\[run\]”](#run)

| Key           | Type             | Required | Meaning                                                                                                                                    |
| ------------- | ---------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `alias`       | string           | yes      | The served model name, as `/v1/models` lists it. Letters, digits, `.`, `_`, `-` and `/`; the recipes use the repo id. Passed as `--alias`. |
| `context`     | integer          | yes      | The context length, passed as `-c`. Positive. `--context` overrides it and makes the run untuned.                                          |
| `threads_max` | integer          | yes      | The most CPU threads to use. `-t`, `-tb` and `OMP_NUM_THREADS` are the smaller of this and the host’s CPU count. Positive.                 |
| `args`        | array of strings | no       | Every other `llama-server` argument, in order.                                                                                             |
| `env`         | table of strings | no       | The engine switches, exactly as measured. Each key must be a valid variable name. See [Engine switches](/docs/reference/engine-switches/). |

The launcher sets these itself, so `args` must not contain them: `-m`, `--model`, `--host`, `--port`, `-c`, `--ctx-size`, `-a`, `--alias`, `-t`, `--threads`, `-tb`, `--threads-batch`. `env` must not set `OMP_NUM_THREADS`.

## \[receipt]

[Section titled “\[receipt\]”](#receipt)

What was measured, and where. Every key is optional. `info` shows them, and `bench` compares its result with `decode_tps.short`.

| Key           | Type             | Meaning                                                                               |
| ------------- | ---------------- | ------------------------------------------------------------------------------------- |
| `source`      | string           | Who measured the numbers, and on what build.                                          |
| `summary`     | string           | One or two sentences on the result.                                                   |
| `decode_tps`  | table of numbers | Decode speed in tokens per second. Keys: `short`, `ctx_32k`, `ctx_65k`. Not negative. |
| `prefill_tps` | table of numbers | Prefill speed in tokens per second, for example `ctx_4k`. Not negative.               |
| `notes`       | array of strings | The scope of each number, and anything else a reader needs.                           |

`info` shows measurement keys in the order `short`, `ctx_4k`, `ctx_32k`, `ctx_65k`, then any other keys alphabetically. The full measurements of a release are in its [receipts](/docs/reference/receipts/).

## Checks

[Section titled “Checks”](#checks)

The launcher refuses a recipe file that:

* has a key the schema does not have (`unknown keys`);
* has a value that breaks a rule in the tables above;
* sits at a path that does not match its `model.hf_repo` and `card.id`.

## Draft, released and unpublished

[Section titled “Draft, released and unpublished”](#draft-released-and-unpublished)

| State       | How it is set                                                         | `pull`, `run`, `serve`              | `info`                       | `--dry-run`                     | `models` shows                                  |
| ----------- | --------------------------------------------------------------------- | ----------------------------------- | ---------------------------- | ------------------------------- | ----------------------------------------------- |
| Released    | `status = "released"`                                                 | Allowed.                            | Shown.                       | Shown.                          | `released`                                      |
| Draft       | `status = "draft"`                                                    | Refused unless `--allow-draft`.     | Shown.                       | Refused unless `--allow-draft`. | `draft`                                         |
| Unpublished | An all-zero `revision`, `sha256` or image digest, or `size_bytes = 0` | Refused, even with `--allow-draft`. | Shown, with `NOT PUBLISHED`. | Shown.                          | `draft, unpublished` or `released, unpublished` |

A release moves a recipe to `released` only after the release gate has measured its image on its card. A pull request that changes `[run]` or `[engine]` needs a new receipt.

# Run your first model on a P100

> Install tensorless on a Linux host with a Tesla P100, check the machine, and ask Qwen3.8-27B-PN3 a question through its OpenAI-compatible API.

In this tutorial you install the tensorless launcher, check that your machine can run a model, start Qwen3.8-27B-PN3 and ask it a question. Qwen3.8-27B-PN3 is a 27-billion-parameter model, quantized to about 3 bits per weight for the Tesla P100.

At the end, the model answers on `http://127.0.0.1:8080`. It runs on the engine image and settings that the release measured at about 30 tokens per second on this card.

**How long it takes:** the downloads take most of the time. The model is 12.95 GB and the engine image is about 1.6 GB, so expect about 20 minutes at 100 Mbit/s or a few minutes at 1 Gbit/s. The rest takes about ten minutes.

## Before you start

[Section titled “Before you start”](#before-you-start)

You need:

* a Linux x86_64 machine with one NVIDIA Tesla P100 16 GB, which nothing else is using;
* an x86-64 CPU with AVX2, FMA and F16C;
* the NVIDIA driver, version 525.60.13 or newer, from the R580 branch or older;
* podman and the NVIDIA Container Toolkit, which provides the `nvidia-ctk` command;
* about 18 GB of free disk space: 12.95 GB for the model in your home directory, and a few GB for the engine image;
* `curl`.

If the driver or the toolkit is missing, follow [Prepare a Linux host for a P100](/docs/how-to/setup/prepare-host/) first, then come back here.

## 1. Install the launcher

[Section titled “1. Install the launcher”](#1-install-the-launcher)

The launcher, `tensorless`, is one program with no dependencies. This script downloads it from the GitHub release, checks its sha256 and puts it in `~/.local/bin`:

```sh
curl -fsSL https://tensorless.dev/install.sh | sh
```

You see:

```text
Downloading tensorless 0.1.0 for linux-x86_64
Checked sha256 cb0c796fd83a334eb67f163fffb30016f51513f8f6ca4a30e7144875487415eb
Installed /home/you/.local/bin/tensorless


Next: check this machine, then run a model.
  tensorless doctor
  tensorless run iamMrDuncan/Qwen3.8-27B-PN3-GGUF
```

If the script also says that `~/.local/bin` is not on your `PATH`, run the `export` line it prints. Then check the version:

```sh
tensorless version
```

```text
tensorless 0.1.0 (commit d104660be24427080948eafb0522265e1450d642), linux/amd64
```

## 2. Give podman the GPU

[Section titled “2. Give podman the GPU”](#2-give-podman-the-gpu)

The engine runs in a container. Podman learns how to hand a container the GPU from a CDI spec: a file that lists the driver’s devices and libraries. `nvidia-ctk` writes it:

```sh
sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
```

The file `/etc/cdi/nvidia.yaml` now exists. It names the driver’s files by version, so generate it again whenever you upgrade the driver.

## 3. Check the machine

[Section titled “3. Check the machine”](#3-check-the-machine)

```sh
tensorless doctor
```

`doctor` checks the GPU, the driver, the container runtime, the CDI spec and the free disk space. On a ready machine every line starts with `PASS`. The parts in angle brackets are your machine’s own values:

```text
PASS  nvidia-smi lists 1 GPU
PASS  GPU 0: Tesla P100-PCIE-16GB, compute 6.0, <memory> MiB, driver <driver>; fits iamMrDuncan/Qwen3.8-27B-PN3-GGUF on p100, iamMrDuncan/Qwen3.8-27B-UD-IQ3_XXS-Q3Floor-GGUF on p100
PASS  driver <driver> is at least 525.60.13 (needed by iamMrDuncan/Qwen3.8-27B-PN3-GGUF on p100)
PASS  container runtime: podman (/usr/bin/podman)
PASS  NVIDIA CDI spec for podman: /etc/cdi/nvidia.yaml
PASS  model cache /home/you/.cache/huggingface/hub: <free> GB free, the model needs 12.95 GB


all checks passed.
```

A `FAIL` line has a `fix:` line under it with the command that fixes it. Run that command, then run `tensorless doctor` again. [Fix a failed check or start](/docs/how-to/use/troubleshoot/) explains each check.

## 4. Start the model

[Section titled “4. Start the model”](#4-start-the-model)

```sh
tensorless run iamMrDuncan/Qwen3.8-27B-PN3-GGUF
```

The name is the model’s Hugging Face repository. The launcher finds its recipe for your card: the exact engine image, model file and settings that were measured on a P100. Then it does four things, and tells you as it goes:

1. It pulls the engine image by its digest:

   ```text
   pulling the engine image ghcr.io/iammrduncan/tensorless-engine@sha256:35409ff97c2f518e468790c28bd19206ae777a945fcffe5775585e28fd9939a7
   ```

   Podman’s own download progress follows.

2. It downloads the model into the Hugging Face cache, and checks the file’s sha256 as it goes:

   ```text
   model Qwen3.8-27B-PN3-P0k19.gguf from iamMrDuncan/Qwen3.8-27B-PN3-GGUF at c701b6690ec3
   ```

   A progress line follows. It starts with `downloading Qwen3.8-27B-PN3-P0k19.gguf` and shows the percentage, how much of the 12.95 GB has arrived, and the speed in MB/s. If the download stops, run the same command again: it resumes where it stopped.

3. It starts the engine in a container:

   ```text
   starting tensorless-iammrduncan-qwen3.8-27b-pn3-gguf (podman, GPU all)
   ```

   The engine’s own log lines follow while the model loads onto the GPU. The first start reads the whole 12.95 GB file from disk.

4. When the server answers, the launcher prints:

   ```text
   iamMrDuncan/Qwen3.8-27B-PN3-GGUF is ready at http://127.0.0.1:8080 (OpenAI-compatible API under /v1)
   try it:
     curl http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' \
       -d '{"model": "iamMrDuncan/Qwen3.8-27B-PN3-GGUF", "messages": [{"role": "user", "content": "Hello"}]}'
   Ctrl-C stops the server.
   ```

Leave this terminal open. The server runs for as long as the command does.

## 5. Ask it something

[Section titled “5. Ask it something”](#5-ask-it-something)

Open a second terminal, and send the model a chat request:

```sh
curl -s http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"iamMrDuncan/Qwen3.8-27B-PN3-GGUF","messages":[{"role":"user","content":"Say hello in one sentence."}],"max_tokens":64}'
```

The reply is one JSON object in the OpenAI chat format. The model’s answer is in `choices[0].message.content`. In the first terminal, the engine’s log shows the request.

The API under `/v1` is OpenAI-compatible, so chat apps and SDKs that speak to OpenAI can use this server too. The model name they send is the repository id.

## 6. Stop the server

[Section titled “6. Stop the server”](#6-stop-the-server)

Press Ctrl-C in the first terminal. The launcher prints:

```text
stopping tensorless-iammrduncan-qwen3.8-27b-pn3-gguf
```

The container is removed. The model stays in the Hugging Face cache and the image stays in podman’s storage, so the next `tensorless run iamMrDuncan/Qwen3.8-27B-PN3-GGUF` downloads nothing and starts in seconds.

## What you did

[Section titled “What you did”](#what-you-did)

You ran a 27-billion-parameter model on one P100, on the engine it was tuned with, from one command. The launcher pinned everything: it pulled the image by digest, downloaded the model at a fixed revision and checked its sha256.

Next:

* [Measure your card against the receipt](/docs/tutorials/measure-your-card/): see how fast your P100 runs this model.
* [Connect an OpenAI-compatible client](/docs/how-to/use/openai-clients/): use the model from a chat app or your own code.
* [Run a model in the background, and stop it](/docs/how-to/run/run-in-background/).
* [How tensorless works](/docs/explanation/how-tensorless-works/): recipes, images and receipts.

# Measure your card against the receipt

> Start Qwen3.8-27B-PN3 in the background, measure its decode speed with tensorless bench, and compare it with the speed the release measured.

In this tutorial you measure how fast your P100 generates text with Qwen3.8-27B-PN3, the same way the release was measured. Then you compare your number with the recipe’s receipt: the speed recorded in the recipe when it was released.

It takes about five minutes.

## Before you start

[Section titled “Before you start”](#before-you-start)

You need to have finished [Run your first model on a P100](/docs/tutorials/first-model/). The engine image and the model are then already on this machine. Nothing else should be using the GPU: another process on the card slows the model down.

## 1. Start the server in the background

[Section titled “1. Start the server in the background”](#1-start-the-server-in-the-background)

```sh
tensorless serve iamMrDuncan/Qwen3.8-27B-PN3-GGUF --detach
```

`serve` starts the model from the image and model file that are already on the machine, and `--detach` returns as soon as the server is ready. You see:

```text
starting tensorless-iammrduncan-qwen3.8-27b-pn3-gguf (podman, GPU all)
waiting for http://127.0.0.1:8080/health while the model loads (up to 20m)


iamMrDuncan/Qwen3.8-27B-PN3-GGUF is ready at http://127.0.0.1:8080 (OpenAI-compatible API under /v1)
try it:
  curl http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' \
    -d '{"model": "iamMrDuncan/Qwen3.8-27B-PN3-GGUF", "messages": [{"role": "user", "content": "Hello"}]}'
stop it with: tensorless stop iamMrDuncan/Qwen3.8-27B-PN3-GGUF
```

If loading takes longer than 30 seconds, a `still loading` line shows every 30 seconds until the server is ready.

## 2. Run the benchmark

[Section titled “2. Run the benchmark”](#2-run-the-benchmark)

```sh
tensorless bench iamMrDuncan/Qwen3.8-27B-PN3-GGUF
```

`bench` sends one short warm-up request. Then it sends three requests that each generate 256 tokens, and it times every token as it streams back. It takes about a minute. While it runs, it shows one line per run:

```text
benchmarking iamMrDuncan/Qwen3.8-27B-PN3-GGUF at http://127.0.0.1:8080: warm-up, then 3 runs of 256 tokens
run 1: <decode> tok/s decode, TTFT <ms> ms
run 2: <decode> tok/s decode, TTFT <ms> ms
run 3: <decode> tok/s decode, TTFT <ms> ms
```

Then it prints the result. Your own numbers replace the parts in angle brackets:

```text
iamMrDuncan/Qwen3.8-27B-PN3-GGUF on p100, 3 runs of 256 tokens (median):
  decode         <decode> tok/s
  TTFT           <ms> ms
  prompt tokens  429
  receipt        30.06 tok/s decode (short); this run is <difference>%
The receipt was measured with a different prompt, so expect a few percent of difference.
```

## 3. Read the result

[Section titled “3. Read the result”](#3-read-the-result)

* **decode** is how many tokens per second the model generates once it has started answering: the median of the three runs. This is the number you feel when you chat.
* **TTFT**, time to first token, is how long the request took before the first token arrived. Most of it is reading the 429-token prompt, so it grows with the length of the prompt.
* **receipt** compares your median with the recipe’s receipt, `decode_tps.short = 30.06`. The research hill climb’s judge measured that number when it tuned the engine.

When the release was made, the published image measured **30.03 tok/s** on a P100 with this same command: 0.1% below the receipt. If your result is within a few percent of 30.06, your card runs the model as it was measured.

If yours is much lower, look for these:

* **Another process on the GPU.** `nvidia-smi` lists the processes that use the card.
* **A hot or power-limited card.** `nvidia-smi` shows the card’s temperature and power draw. A card that runs too hot slows its clocks.
* **An untuned run.** If the server was started with another `--context`, extra server arguments or another model revision, the launcher printed `UNTUNED` when it started, and the receipt does not apply. [Tuned and untuned runs](/docs/explanation/tuned-and-untuned/) explains why.

## 4. Stop the server

[Section titled “4. Stop the server”](#4-stop-the-server)

```sh
tensorless stop iamMrDuncan/Qwen3.8-27B-PN3-GGUF
```

```text
stopped tensorless-iammrduncan-qwen3.8-27b-pn3-gguf
```

## What you did

[Section titled “What you did”](#what-you-did)

You measured your card the way the release gate measures a candidate engine, and you compared the result with the recipe’s receipt. Every number tensorless publishes is a measurement like this one, written down with its scope.

Next:

* [Share your results](/docs/how-to/use/share-results/): numbers from other P100s help everyone.
* [Benchmark a running server](/docs/how-to/use/benchmark/): more runs, longer runs and JSON output.
* [How a release is made, and why you can trust its numbers](/docs/explanation/releases-and-receipts/).
* [The receipts](/docs/reference/receipts/) of engine-v0.1.0, for both launch targets.