The core lesson
Do not treat aggregate VRAM as one flat memory pool. V100 systems are constrained by topology, backend support, attention kernels, KV-cache size, and the exact quantization path. A model that should fit on paper can still fail if the interconnect or backend is wrong.
Backend reality
The public findings identify GGUF through llama.cpp-style backends as the broadest compatibility lane. Ollama remains useful for quick management and GGUF tests. vLLM can matter for API serving and batching, but V100 support is version-sensitive and often requires careful flags or forks. Modern fast paths aimed at newer GPUs cannot be assumed to work on Volta.
What the benchmarks show
The April 2026 GGUF/Ollama report found that medium MoE models were the speed leaders on a 9 x V100-SXM2-32GB system, with several 30B-class lanes producing strong throughput at long context. The June vLLM probe showed that selected large AWQ lanes could load and generate when context and KV mode were chosen carefully, while the same families failed under less favorable KV settings or unsupported backend paths.
Successful V100 launch profiles
The public success profile now publishes measured successful rows without hostnames, network addresses, local paths, service names, or failure runbooks. The GGUF lane used Ollama-style local serving and model-specific GGUF quants; the API lane used AWQ4 artifacts with FP8 E5M2 KV cache storage on a V100-focused vLLM path.
| Lane | Model class | Context | Measured output |
|---|---|---|---|
| GGUF/Ollama | Qwen 30B-A3B MoE | 262K tested max | 94-97 decode tok/s |
| GGUF/Ollama | Qwen 122B-A10B MoE | 262K tested max | 36-37 decode tok/s |
| vLLM AWQ | 139B-172B MoE AWQ | 92K-131K | 13.10-14.70 completion tok/s |
Operational rule: public A100 or H100 numbers are only hypotheses for a V100 box. A V100 claim is real only after the exact model, backend, quantization, context length, and lane have been tested on that hardware.
Why this matters for law firms
Owned local hardware is attractive because it makes confidentiality an architecture choice, not a vendor promise. V100 research helps decide what work should run locally, what model families are practical, and where a firm should spend money if it wants private legal AI without chasing the newest data-center cards.