Arm has released developer tools and prestandard Vulkan extensions to enable GPU programmers to insert neural networks into graphics processing, and the company plans to release compatible GPUs in 2026.
Resolution Improvements Come First
Resolution upscaling is Arm’s first neural network for graphics processing. Available to developers now, the new Neural Super Sampling (NSS) will replace Arm Accuracy Super Resolution (ASR) released earlier this year. ASR runs on all Arm Mali GPUs and depends on static rules, making it susceptible to artifacts, especially with new software for which it hasn’t been tuned. By contrast, the NSS approach yields better image quality and is a general upscaling mechanism. It entails a neural network trained on varied content that factors in numerous data features (e.g., color, motion vectors, metadata, and state information from previous frames) intrinsic to images.
Arm estimates that upscaling from 540p to 1080p requires 4 ms for the neural network and 1.4 ms for additional shader processing, which compares favorably to the time ASR needs. Overall, NSS should not reduce frame rates; in fact, rates could improve by performing graphics processing on reduced-resolution images.
In terms of computational throughput, Arm’s 4 ms time budget translates to a computational budget of about 27 GOPS. By contrast, GPUs that integrate engines for general, non-real-time AI processing have peak rates of 100 TOPS or more. Arm isn’t delivering prodigious raw AI throughput but instead enabling programmers to add neural networks inline with their graphics code.
Faster Frame Rates Come Next
Neural frame-rate upscaling (NFRU) and neural super-sampling and denoising (NSSD) are next on Arm’s agenda. The former helps games and other content maintain a consistent, high frame rate by inferring frames between conventionally drawn ones. The latter applies the above resolution upscaling and enhances the quality of ray-traced scenes by inferring pixel values between ray intersections.
Custom Models Are Possible
Developers will be able to deploy custom models on the forthcoming Arm GPUs with neural accelerators. In the PC market, upscaling, frame generation, and denoising have been the main AI models integrated into graphics pipelines. We expect many developers to use the Arm examples, possibly fine-tuning them for their applications.
However, the programmer experience is critical. In addition to disclosing the new technology, Arm is releasing a development kit. It includes a profiler, code samples, example models, and an Unreal Engine plugin. It also includes a PC-based Vulkan emulator supporting Arm’s machine-learning (ML) extensions.
In addition to Vulkan’s standard graphics, computing, and ray-tracing pipelines, the ML extensions define an additional pipeline for neural networks. A GPU workload can sequence multiple pipelines. Indeed, upscaling involves computing pipelines for pre- and postprocessing in addition to the neural-network pipeline. The ML extensions include the VK_Arm_tensors data structure to contain model data and the VK_Arm_data_graph for holding the model structure. Arm expects official Vulkan ML extensions to arrive next year and to be similar to its implementation.
Neural Nets Conserve GPU Power
For PCs and game consoles, resolution and frame-rate enhancements aim to improve the user experience. Smartphones and other mobile devices will also benefit from power savings. Compared with native 1080p rendering, Arm’s Enchanted Castle demo exhibits a 50% GPU workload reduction when rendering at 540p and upscaling to 1080p. The attendant power savings can extend battery-powered gameplay, or the GPU cycles freed by workload reduction can be applied toward additional graphics effects.
Bottom Line
Once far different from their desktop counterparts, mobile GPUs are now similar. They support Vulkan, run popular game engines, and get the same advanced features within a few years of Nvidia pioneering them. Motivating mobile-GPU makers like Arm is the rising value of smartphone games. About half of gaming revenue comes from mobile players. Improving graphics quality encourages more play, which funds additional investment. Meanwhile, the mobile GPU market is heating up as Arm faces AMD (which licenses to Samsung), Apple, Imagination, and Qualcomm. The biggest licensor, Arm is pushing mobile technology forward with its new neural-network capabilities.