Local LLM inference diverges from reference logits by KLD values exceeding 0.1 on standard samplers

The Level1Techs post documents how mixed-generation GPUs, GGUF quantization, and non-default temperature or top-p settings alter logit outputs. Direct comparison of the same weights across Ollama, llama.cpp, and vLLM backends produces measurable KL divergence from the publishing lab's reported distributions. The analysis isolates sampler mismatch and calibration data as primary drivers rather than weight precision alone.

Reference model cards specify exact sampler parameters and chat templates; deviations move probability mass on low-probability tokens first, then propagate into repetitive or truncated outputs. KLD measurements on calibration text show that 4-bit GGUF files without matching calibration sets routinely exceed 0.08 divergence, while FP16 reference runs stay below 0.02. This gap explains why agentic tool-calling workloads degrade faster than short-prompt tests.

The common misunderstanding is that model weights determine capability independent of runtime. Evidence from the post and related quantization studies shows that inference stack variance accounts for more performance loss than bit-width reduction in many cases. Standardized evaluation environments and disclosed calibration sets are required before cross-user comparisons become reliable.

Next steps include publishing full runtime manifests with each HF model card and integrating KLD checks into benchmark suites for local deployments.