New inference techniques are bridging the gap between research and production reality. By translating optimization methods into open source tools, researchers are ensuring that the most capable models remain accessible and efficient for every developer.
AI models have become far more capable over the last few years, and much more expensive to run. The same systems that can write software, read scientific papers, and hold conversations in many languages also demand large amounts of memory, computation, and power.
For many organizations, getting access to a state-of-the-art model is no longer the hardest part. Running it efficiently enough to be useful is. Every token a model generates consumes compute, energy, and time, and those costs can quickly become prohibitive as models and workloads grow.
Making AI more accessible must mean more than publishing model weights. Open weights have limited value if serving them requires infrastructure that only a handful of organizations can afford. The software around the model, how it is compressed, served, accelerated, and optimized, matters just as much as the weights themselves.
Efficient inference, compression, and serving are not merely implementation details. Together, they decide how practical a model is to deploy.
We’ve seen this pattern before. Linux demonstrated how infrastructure built through open collaboration could become a shared foundation for an entire industry. Kubernetes later did something similar for application orchestration. AI infrastructure is now reaching a comparable stage, where the systems beneath the models are becoming as consequential as the models themselves.
As language models become core components of ordinary software, efficient inference, compression, and serving are not merely implementation details. Together, they decide how practical a model is to deploy.
That idea guides our work on the vLLM inference engine, the LLM Compressor library, the Speculators framework for speculative decoding, and the optimized models we publish through the Red Hat AI Hub on Hugging Face. These projects, detailed below, help more teams run capable models without a matching increase in their compute budgets.
Serving models efficiently with vLLM
A capable model provides little value if it responds slowly or struggles under concurrent demand.
Originally developed at the University of California, Berkeley, vLLM is now built by a broad open source community of companies, universities, and independent developers. It has become one of the most widely used inference engines for large language models, designed to keep GPUs well utilized and reduce memory overhead so the same hardware can serve more users and process more requests.
Making large models smaller
Language models are large because they store billions of parameters, but not every part of a model needs full precision to preserve accuracy.
In collaboration with the open source community, Red Hat engineers and researchers build and maintain LLM Compressor, the library where we turn state-of-the-art quantization research into production-ready algorithms, such as GPTQ, SparseGPT, AWQ, SmoothQuant, and AutoRound. It gives developers a practical way to reduce a model’s memory and compute footprint through quantization and sparsity, so a model can run on hardware that could not host it in its original form while keeping most or all of its accuracy.
Moving model weights from 16-bit to 8-bit can roughly halve their memory footprint, while 4-bit formats can approach a fourfold reduction. The exact end-to-end saving depends on the model architecture, the quantization format, and which components of the model are compressed. (For more on ISTA work in compression and quantization, see “Finding the next breakthrough: AI efficiency research from one of the world’s top optimization labs” in the Spring 2026 issue of RHRQ.)
In LLM Compressor we push the quantization Pareto frontier further with non-uniform quantization, which applies different numerical formats to different parts of the model. Sensitive layers can stay at higher precision while more tolerant layers are compressed more aggressively, for a better balance between memory use, performance, and accuracy. In practice, a team can adapt a model to its own hardware and latency constraints rather than reshape its entire deployment around the model. The target might be a high-end datacenter GPU, a smaller accelerator, or a system operating under a strict memory budget.
The recent push toward agentic workloads makes quantization especially important. As models plan, call tools, and work through long, multi-step tasks, they maintain an expanding history in a key-value cache (KV cache). For long-running sessions, that cache can consume a substantial share of available GPU memory. We compress the KV cache as well, often reducing its size by a factor of two, and sometimes more, with little or no measurable effect on model quality. For an agent that keeps a long context active across many steps, halving this cache can free memory for additional concurrent sessions on the same hardware.
The result is lower memory use, greater serving capacity, and better use of existing hardware. A smaller model also consumes less power and requires less cooling, which lowers electricity costs.
Generating faster with speculative decoding
People constantly predict while they speak, anticipating the next few words so a conversation can keep moving. Speculative decoding brings a similar idea to language-model inference. A smaller, faster model proposes several upcoming tokens, and the larger model verifies them in a single step. When the proposals are accepted, the larger model can produce multiple tokens while performing much less sequential work.
In collaboration with the open source community, Red Hat engineers and researchers build and maintain Speculators, an open framework for training and deploying these predictor models. As with LLM Compressor, it is where we turn state-of-the-art speculative-decoding research, including EAGLE-3, DFlash, and DSpark, into implementations teams can deploy. The larger model remains the final authority over every accepted token, so the output matches what the target model would have produced on its own. In favorable workloads, the speculators we train can accelerate decoding by as much as five times. The realized speedup depends on the target model, hardware, workload, generation settings, and how accurately the speculator predicts.
Compression and speculative decoding address different bottlenecks. Compression reduces the memory and computational footprint of the model, while speculative decoding changes how efficiently tokens are produced. Neither depends on the other, so they can be used together.
The result is a system in which a single GPU may be able to host a larger model, support more users, and produce output faster.
A deployment can begin with a model that occupies roughly one-quarter of its original weight footprint and then add a speculator to accelerate generation. The result is a system in which a single GPU may be able to host a larger model, support more users, and produce output faster.
From research to deployment
Even with strong infrastructure in place, developers still face practical hurdles. Most teams want to focus on building their application, not connecting a quantization and speculative-decoding pipeline from scratch. To reduce this setup work, Red Hat actively publishes compressed and optimized models through the Red Hat AI Hub on Hugging Face, often within just a few hours of the original model’s release. These releases apply techniques developed across our research and engineering work, so developers can deploy optimized checkpoints directly instead of spending weeks reproducing the same process. They span a broad spectrum of models, from small models with a few billion parameters, such as Qwen and Gemma, to mixture-of-experts models with hundreds of billions of parameters, such as DeepSeek-V4, Inkling, and Kimi.
The same base model can often be released in several forms: a highly compressed version for constrained hardware, a higher-precision version for workloads where every point of accuracy matters, or a speculator designed to accelerate a particular target model. Teams choose the trade-off that fits their own deployment rather than accept a single configuration chosen for everyone.
Our model cards include deployment guidance, evaluation results, and, wherever applicable, detailed reproduction recipes. Developers can inspect how a model was optimized, reproduce the process, and use the published recipe as a starting point for their own models.
Built in the open
None of this works as well behind closed doors. The communities around vLLM, LLM Compressor, and Speculators include contributors from companies, universities, and independent developers. A fix or optimization written by one contributor can benefit thousands of users, while improvements from the wider community strengthen the tools our own work depends on.
This is a familiar open source pattern: shared infrastructure improves faster when many groups can contribute to it.
What comes next
The next generation of models will be more capable and more deeply woven into everyday software. Some will also be larger and more computationally demanding.
Meeting that challenge will take more than better models. It will require open infrastructure that makes those models efficient, affordable, and practical to deploy. vLLM, LLM Compressor, Speculators, and our optimized model releases each address a different part of that problem, and all of them are open for anyone to use, inspect, and build on.