Israeli startup Hailo promises to bring generative AI to edge-processing applications. Planned to sample by midyear, the 40 TOPS Hailo-10H NPU (AI accelerator) promises 50% more raw performance than the Hailo-8 and can run Llama2-7B at up to 10 tokens per second while requiring only 5 W. Whereas the company has previously focused on object detection and image enhancement, such as for smart surveillance cameras or ADAS, Hailo-10 enables the company to also target PC and automotive infotainment systems. The company will offer an M.2 card for adding AI acceleration to PCs and edge devices.
Notables
- 40 TOPS significance—The 40 TOPS threshold is Microsoft’s minimum for locally running Copilot. Significantly, it’s more than AMD Hawk Point and Intel Meteor Lake supply. Crucially for Hailo, the forthcoming Intel Arrow Lake laptop processors (Core Ultra 200) might not improve AI acceleration much beyond what Meteor offers. Thus, to bring better AI processing to the masses, PC OEMs will need an add-in card.
- Performance—Hailo claims its new accelerator can deliver up to 10 tokens per second running Llama2-7B. By contrast, Qualcomm claims its upcoming Snapdragon Elite X PC processor can deliver 30 tokens per second, despite rating its NPU at a similar 45 TOPS. Although raw TOPS is an imperfect metric, actual models like Llama2-7B leave room for gaming. Thus, we can’t conclude which NPU is actually faster.
- Applications—It’s likely that Hailo-10 will primarily be used to accelerate the same computer-vision tasks as Hailo-8 but handle more streams or higher-bandwidth video. Nonetheless, generative AI at the edge could emerge. Embedded systems can use large language models (LLMs) to respond to natural-language commands and to formulate human-sounding responses—for example in cars, where it’s better for drivers to keep their eyes on the road and hands on the steering wheel. Even security cameras can employ LLMs to infer captured behavior and report observations to an admistrator. Hailo-10 can run Stable Diffusion, which could be relevant for PC add-in cards.
Competition
- The null option—Hailo’s main alternatives are not performing AI at all or to do it on a central system (e.g., in the cloud). Smart cameras are an intriguing concept, but many surveillance systems merely archive footage that will never be examined. Increasing the intelligence of applications like this where AI conceptually makes sense but will only ever be required in a sliver of the market won’t spur further adoption.
- Ambarella announced in January that it would support gen AI on some of its SoCs. Like Hailo, Ambarella has heretofore addressed computer vision, such as security cameras, action cameras, and—increasingly—automotive applications. The greater integration of Ambarella’s gen-AI-capable chips is an advantage in systems that can employ their various functions. The Hailo-10, however, should be more cost effective and power efficient for customers seeking only to add AI processing.
- Kneron’s processors can execute transformer models, which underpin GPT and other gen-AI technologies, but address designs requiring one-tenth the performance that Hailo-10 promises.
Customers
Although it’s a startup, Hailo claims 300 customers. In the chip business, one big customer is worth more than 300 little ones, but 300 little ones are better than what many edge-AI startups can claim. Moreover, a broad customer base can guide product development, and we assume Hailo has already orders for the new NPU.
Customers already using Hailo-8 will find Hailo-10 boosts performance (albeit at higher power) and uses the same software tools. They can employ the greater performance to do more of the same or to add new capabilities, such as generative AI.
Bottom Line
Hailo is among the more successful AI startups. Selling NPUs requiring less than 5 W, it targets embedded systems and—with Hailo-10 and the rising AI-acceleration demand in PCs—potentially PCs. Hailo-10, thus, is both an upgrade for existing designs and an entry point for customers in new-to-Hailo segments.