TensorRT Edge-LLM

The Thor served its models with llama.cpp only. NVIDIA’s engine for the Thor is TensorRT Edge-LLM. This chapter is the plan to test it, the design for two engines behind one agent, and the first results.

Status, 2026-10-06: Qwen3.6-35B-A3B in NVFP4 runs on Edge-LLM v0.11.0 as a second model in the Thor Tigress Cub’s picker, next to Nemotron 3 Nano on llama.cpp. The results are in “First results” below. The Nano on Edge-LLM hasn’t been built yet. Facts about Edge-LLM come from NVIDIA’s documentation and release notes; each source is named.

Why Edge-LLM

The Thor has TensorRT 10.16.2 and CUDA 13.2 installed (checked with pip and dpkg). Edge-LLM isn’t installed.

The reason to try it is speed. Generating a token reads every active weight once, so speed follows the bytes read per token (chapter “Model serving”, “Estimate its speed”). NVIDIA publishes the Nano in NVFP4, a 4-bit format the Thor’s Blackwell GPU runs natively. It is 19.3 GB against 33.6 GB for our Q8_0: about 0.61 bytes per parameter, so about 2 GB read per token for 3.2B active. The same formula estimates about 90 tok/s against the 53.5 measured today. That is an estimate from one calibration point, not a measurement.

The one published Thor result: in the MLPerf Inference v6.1 Edge Agentic benchmark, Edge-LLM ran Qwen3.6-27B in NVFP4 at 52.33 output tokens/s, with a median 247 ms to the first token, and finished 6.4 times faster than the llama.cpp reference run (24 min 36 s against 2 h 37 min, 1,007 turns, inputs up to about 23.5K tokens). That is a different model and workload from ours.

What Edge-LLM supports

From the supported-models page and the release notes:

System design

Design rules:

• Clients don’t change. The web chat, Claude Code and OpenCode keep sending a model name to :8080. Which engine serves it is the agent’s business.
• One model, one engine at a time. The Nano in Q8_0 on llama.cpp (57.8 GB with 4 × 1M context) and the Nano in NVFP4 on Edge-LLM side by side may not fit. The test runs Edge-LLM with the llama.cpp Nano unloaded, at a quiet hour.
• Engines are built, not downloaded. A TensorRT engine is compiled for the exact GPU that runs it, so every engine is built on the Thor.

The pipeline

The commands follow the Jetson AI Lab tutorial and the Edge-LLM docs. None were run here yet.

1. Tools. NVIDIA’s PyTorch container for export, and the C++ runtime built on the Thor:

docker pull nvcr.io/nvidia/pytorch:26.05-py3
git clone https://github.com/NVIDIA/TensorRT-Edge-LLM.git
cd TensorRT-Edge-LLM && git submodule update --init --recursive
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -DTRT_PACKAGE_DIR=/usr \
  -DCMAKE_TOOLCHAIN_FILE=cmake/aarch64_linux_toolchain.cmake \
  -DEMBEDDED_TARGET=jetson-thor -DCUDA_CTK_VERSION=13.0 -DENABLE_CUTE_DSL=ALL
make -j"$(nproc)"

The tutorial passes -DCUDA_CTK_VERSION=13.0; the Thor has CUDA 13.2. Whether that matters is the first thing to check.

2. Checkpoint and export. NVIDIA publishes the Nano already in NVFP4, so the quantize step is skipped. The export takes a checkpoint folder:

hf download nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 --local-dir ~/models/edge-llm/nemotron-3-nano/checkpoint
tensorrt-edgellm-export ~/models/edge-llm/nemotron-3-nano/checkpoint ~/models/edge-llm/nemotron-3-nano/onnx

3. Build the engine on the Thor:

./build/examples/llm/llm_build --onnxDir ~/models/edge-llm/nemotron-3-nano/onnx \
  --engineDir ~/models/edge-llm/nemotron-3-nano/engine

4. Serve. The experimental server, on a port of its own:

python -m experimental.server --model <engine or checkpoint> --port 8081

5. Measure. First with the C++ benchmark tool (./build/examples/llm/llm_bench --engineDir … --mode generation), then through the server with the same requests sent to llama.cpp.

What to measure

Each with the date, the Edge-LLM version, the precision, the context size and the number of slots, as for every number in this book.

Edge-LLM becomes the default engine for a model only if:

• it generates at least 1.5 times faster on one reply and on four,
• spark scores and compile rates are no worse,
• it fits with at least 8 GB free at the context the chat needs,
• and its server handles thinking and tool calls (below) for 24 hours without a restart.

First results

Measured on 2026-10-06 on the Thor, with Edge-LLM v0.11.0 installed from its aarch64 wheel into ~/.local/share/edge-llm/venv (no container: Docker needs sudo there, and the wheel carries the runtime built for jetson-thor-jp72-cu13-sm110).

Build. tensorrt-edgellm-serve built everything itself from nvidia/Qwen3.6-35B-A3B-NVFP4 (23.4 GB) on its first start:

The engines take 1.7 GB on disk; the weights stay in the checkpoint. The vision engine built without trouble, unlike issue #233.

Memory. Free memory went from 59 GB to 29 GB while the server started next to the Nano, with a 32K-token context and one reply at a time (--max-kv-cache-capacity 32768 --max-input-len 16384 --max-batch-size 1). It settled at about 35 GB free with both running.

Features the chat needs. With --reasoning-parser auto --tool-call-parser auto, through the agent:

The same 20 Rust tasks (in jetson-thor/model-serving/compare.py: median, config parsing, brackets, a ring buffer and so on), one turn each, thinking off, at most 2,048 tokens, first Rust block compiled with rustc --edition 2024 --test:

Against the bar in “What to measure”: faster, more answers compile, better on the five rules, no false claims; tests passing are equal. So it went into the chat as an option, not as the default. The speed estimate of about 90 tok/s was 15% high.

Not measured yet: four replies at once (the server was built for one), time to first token on long prompts, the longest context it holds, and 24 hours of use.

A clean-code model in the chat

The second use of Edge-LLM: a model picked for writing clean code, offered in the Thor Tigress Cub’s model picker next to the Nano.

Candidates. Models on Edge-LLM’s supported list with an NVFP4 checkpoint published by NVIDIA (sizes from the Hugging Face file listings):

Qwen3-Coder-Next, the coding model in chapter “Model serving”, isn’t on Edge-LLM’s list, so it stays a llama.cpp candidate.

How the winner is picked. Clean code is measured, not judged by eye:

The baseline is the Nano’s numbers on the same tasks, measured the same day. A coding model alone isn’t enough: in the earlier baseline (chapter “The checker: spark”), Qwen2.5-Coder-7B on TensorRT-LLM followed all five rules in 4% of its answers.

In the chat. The winner gets an engine of its own on the Edge-LLM server and appears in the picker as one more model. The agent sends requests for it to :8081 and everything else to :8079 (figure 16.1). Memory: the Nano takes 57.8 GB, so about 50 GB is left with 8 GB kept free. A 22 to 23 GB checkpoint plus its context fits, at a context to be measured.

Steps, each needing a GPU time window on the Thor:

1. Build both candidates’ engines (pipeline above).
2. Run the 20 Rust tasks through each, and through the Nano.
3. Score them with spark, cargo build and the tests.
4. Serve the winner on :8081, add the agent’s routing, and add it to the picker.

Open questions and risks

Sources

• TensorRT Edge-LLM supported models
• TensorRT Edge-LLM support matrix
• TensorRT Edge-LLM releases
• Jetson AI Lab: TensorRT Edge-LLM on Jetson
• NVIDIA blog: Edge-LLM completes the MLPerf Edge Agentic benchmark 6.4× faster on Jetson AGX Thor
• Issue #233: tensorrt-edgellm-serve on Thor
• NVIDIA forum: TensorRT-LLM engines on Jetson Thor