NVIDIA NeMo Delivers 10 Times Faster Processing Speed for ASR Models

NVIDIA NeMo has been at the forefront of developing automatic speech recognition (ASR) models that are setting new standards in the industry. Recent enhancements have significantly boosted the inference speed of these models by up to 10x, thanks to key optimizations implemented by NVIDIA.

Driving Speed Improvements

To achieve this remarkable speed enhancement, NVIDIA incorporated various upgrades, including autocasting tensors to bfloat16, a groundbreaking label-looping algorithm, and the integration of CUDA Graphs. These enhancements can be found in NeMo 2.0.0, providing a rapid and cost-effective solution compared to traditional CPUs.

Overcoming Performance Bottlenecks

Previously, NeMo ASR models faced challenges such as casting overheads, low compute intensity, and performance divergence. Through the implementation of full half-precision inference and batch processing optimization, these bottlenecks have been significantly reduced.

Tackling Casting Overheads

Issues like autocast behavior and parameter handling were causing casting overheads. By transitioning to full half-precision inference, NVIDIA has eliminated unnecessary casting while maintaining accuracy.

Optimizing Batch Processing

Shifting from sequential to fully batched processing for operations like CTC greedy decoding has increased throughput by 10%, resulting in an overall speedup of around 20%.

Addressing Low Compute Intensity

The introduction of CUDA Graphs conditional nodes has improved performance by eliminating kernel launch overhead, making models like RNN-T and TDT suitable for server-side GPU inference.

Improving Cost Efficiency

These enhancements have not only enhanced performance but also reduced costs. For example, using GPUs for RNN-T inference can lead to up to 4.5x cost savings compared to CPU-based options.

According to NVIDIA’s comparison, transcribing 1 million hours of speech using the NVIDIA Parakeet RNN-T 1.1B model on AWS instances showed significant cost advantages. CPU-based transcription cost $11,410, while GPU-based transcription cost only $2,499.

Future Developments

NVIDIA is constantly optimizing models like Canary 1B and Whisper to further decrease the cost of running attention-encoder-decoder and speech LLM-based ASR models. The integration of CUDA Graphs conditional nodes with compiler frameworks like TorchInductor is expected to provide additional GPU speedups and efficiency gains.

For more insights, refer to the official NVIDIA blog.

Image source: Shutterstock