TensorRT-LLM
NVIDIA’s optimised LLM inference library for its GPUs.
An open-source library for high-throughput, low-latency LLM serving on NVIDIA GPUs, with in-flight batching, paged KV cache, quantisation (FP8, FP4) and speculative decoding.
- Vendor
- NVIDIA
- Category
- Inference & model hosting
- Pricing
- Open source
- Open source
- Yes
- License
- Apache 2.0
- Platforms
- Python, Linux, Self-host
- Launched
- 2023
- Website
- nvidia.github.io/TensorRT-LLM
Features
- In-flight batching
- FP8/FP4 quantisation
- Speculative decoding
Best for
- High volume / low cost
More inference & model hosting
- Ollama — Ollama. Run open models locally with one command.
- vLLM — vLLM project. High-throughput LLM serving engine.
- llama.cpp — ggml. LLM inference in C/C++ on any hardware.
- Amazon Bedrock — AWS. Foundation models on AWS.
- Vertex AI — Google Cloud. Google Cloud’s AI platform.
- LM Studio — Element Labs. Desktop app to discover and run local LLMs.
- Azure AI Foundry — Microsoft. Build and run AI apps on Azure.
- Groq — Groq. Ultra-fast inference on custom LPUs.