To assist in scaling and modelling concepts for AI factories, NVIDIA has created Dynamo, which works as open-source reasoning software.
An AI factory must efficiently manage and control the execution of GPU tasks which can yield maximized token revenue profit. This is a challenging yet critical endeavour.
With the advancement of AI reasoning, it is expected that any AI model must be able to generate tens of thousands of tokens with every prompt received. Inferring lowers cost while increasing performance, and that is where the opportunity resides to increase growth and revenue of service providers and pay for the cost from the opportunity.
A new generation of AI inference software
NVIDIA Dynamo orchestrates and speeds up inference communication across thousands of GPUs. It uses disaggregated serving, which rotates the processing and generating steps of the large language models on different GPUs. This allows for each step to be optimized for its specific needs and the maximum possibility of GPU resources to be met.
“NVIDIA’s CEO and founder Jensen Huang states, ‘Businesses globally are teaching AI frameworks to operate and reason differently. This makes them smarter with time.’ Huang added, ‘This is why NVIDIA Dynamo is making it possible to serve these models at scale so that reasoning AI can be more cost effective and efficient.’”
Currently, Dynamo has proved the capacity of increasing both productivity and profit two-fold without needing additional GPUs for AI factories using Llama models on the latest NVIDIA Hopper platform. It has also enabled DeepSeek-R1 GB200 NVL72 rack clusters to generate over thirty times more tokens with each GPU than before due to Do-it-yourself cost intelligent inference optimizations of AND NVIDIA’s Dynamo.
NVIDIA Dynamo aids in maximizing imaginary and optical muscle in a powerful server setup by incorporating AI and computer logic information to achieve the desired outcome of lowering average costs of doing business and higher uptime guarantees working with multiple features.
AltaDynamo allows for real-time adjustments in GPU allocation and distribution based on changing volumes and types of requests leading to reduced operational costing and maximized productivity. The software allows for superfast routing and pinpointing of particular GPUs within huge clusters that require minimal response actions being fed the queries.
To cut down on overall costs of inference, Dynamo is also able to get rid of data stored during inference to more affordable memory and storage devices with ease and speed when there is a need for it.
NVIDIA Dynamo has been announced as an entirely open-source project, enabling comprehensive integration with renowned frameworks such as PyTorch, SGLang, NVIDIA TensorRT-LLM, and vLLM. Such mechanisms will aid enterprises, startups, and even academic researchers in formulating and enhancing novel techniques for serving AI models on dispersed inference ecosystems.
Dynamo is expected to boost AI inference deployment at all levels, including the largest cloud customers and AI leaders like AWS, Cohere, CoreWeave, Dell, Fireworks, Google Cloud, Lambda, Meta, Microsoft Azure, Nebius, NetApp, OCI, Perplexity, Together AI, and VAST were mentioned.
NVIDIA Dynamo: Supercharging inference and agentic AI
NVIDIA Dynamo: Bringing agility to inference and agentic AI Supercharging One of the distinct features of NVIDIA Dynamo is how the software automatically virtualizes the knowledge that an inference system retains in memory, called the KV cache, for serving multiple requests in parallel on potentially thousands of GPUs.
The software uses intelligent routing to send new inference tasks to the GPUs with the best knowledge match, eliminating expensive recomputations and freeing other GPUs to start working on new requests. This eases the demands on the GPUs while improving efficiency and speed.
“Like any other service, to service hundreds of millions of requests every month, we turn to NVIDIA GPUs and inference software to provide the performance, dependability and scope Perplexity AI’s business, as well as its clients demand,” said Denis Yarats, CTO of Perplexity AI.
“Dynamo will assist in enabling more inference serving optimizations while catering to the computational needs from novel AI reasoning models with its improved distributed serving capabilities,” said an unnamed source.
Cohere, the AI platform, is already looking ahead to using NVIDIA’s Dynamo to improve agentic AI features in the Command model series.
“Deploying sophisticated AI models involves complex multi-GPU scheduling, controlling, and communication with the latency and context transfer across memory and storage devices,” Saurabh Baji, SVP engineering at Cohere explained.
“When it comes to enterprise users, NVIDIA Dynamo will undoubtedly aid me in ensuring they get the best user experience.”
Perplexity AI: support for disaggregated serving
The NVIDIA Dynamo disaggregation inference platform is well-known for efficient support of disaggregated serving. This innovative approach partitions the various computing stages of LLMs, for example, the logic of comprehending the user’s question and the generation of an adequate answer, and allocates them to different GPUs in the infrastructure.
Disaggregated serving is the perfect technique for improvement in reasoning models like the NVIDIA Llama Nemotron model family which utilizes advanced inference techniques that provide context and response generation wizardry. Disaggregated serving allows improved overall throughput because each phase can be fine-tuned and resourced independently, thus faster response times to users is achieved.
Together AI, a well-known actor in the AI Acceleration Cloud, is looking to merge its proprietary Together Inference Engine with NVIDIA Dynamo so that inference workloads can easily be scaled across many GPU nodes. In addition, this will enable Together AI to dynamically resolve traffic bottlenecks that certain model pipeline stages may cause.
“Scaling reasoning models cost effectively requires new Advanced inference techniques”, says Together AI’s CTO, Ce Zhang. Methods like disaggregated serving and context-aware routing are just a few examples.
The flexible and modular design of NVIDIA Dynamo allows us to connect its components to our engine in order to handle more requests without overleveraging resources – optimizing our investment in accelerated computing. We look forward to using the platform’s advanced capabilities to economically deliver reasoning models that are open-source to customers.”
Four Major Features of NVIDIA Dynamo
As a driver of economical deep learning services, NVIDIA commended Dynamo’s four features that lower the total cost of ownership per inference while increasing service quality.
1. GPU Planner: Intelligent resource allocation system that monitors user activity and dynamically scales the number of active GPUs to meet demand. It prevents both excess and insufficient GPU capacity provisioning.
2. Smart Router: An intelligent router minimizes GPU resource waste by recomputing overlapping inference requests. This intelligent, LLM-aware Smart Router efficiently redistributes numerous inference requests scattered around a multitude of GPUs to streamline resource use and reduce request overhead.
3. Low-Latency Communication Library: Supports GPU communication in an abstract manner. This inference-optimized library facilitates state-of-the-art communication speed and simplifies the data exchange process across heterogeneous devices.
4. Memory Manager: A smart system that handles the transferring of inference data to less expensive memories and storage devices. This system makes sure that there is no disturbance to the user experience.
Future editions of the AI Enterprise software will include support for NVIDIA Dynamo, which will be incorporated into NIM microservices later on.
