A Raspberry Pi 5 can now run selected generative-AI workloads locally with an add-on board—but the headline number, 40 TOPS, does not tell you how fast a chatbot will answer or whether every model will fit.
Raspberry Pi’s AI HAT+ 2 pairs a Hailo-10H neural-network accelerator with 8GB of dedicated on-board memory and connects to Raspberry Pi 5 through PCIe. The manufacturer says it supports local generative-AI workloads, including selected language and vision-language models. That makes this a different kind of add-on from an accelerator aimed only at computer-vision pipelines.
What does 40 TOPS mean?
TOPS means trillions of operations per second. The 40 TOPS figure is specified for INT4 inference, a low-precision calculation format. It describes a peak accelerator capability under suitable conditions—not a promise of 40 trillion useful AI operations in every app, and not a direct measurement of tokens per second.
Real performance depends on the model, quantisation, supported operators, memory movement, software runtime, input size and the rest of the system. A model that is not supported by the software stack cannot use the headline throughput. Likewise, the 8GB on the HAT is its own accelerator memory; it does not turn a 2GB Raspberry Pi 5 into an 10GB system for ordinary Linux applications.
AI HAT+ versus AI HAT+ 2: check the workload
The earlier AI HAT+ comes in 13-TOPS and 26-TOPS variants and is a strong fit for supported vision inference. AI HAT+ 2 is the model to investigate when your goal specifically includes generative workloads. Do not choose on TOPS alone: check the current Raspberry Pi documentation for supported models, setup steps and software versions, then confirm that the camera or other input device in your project is supported.
That distinction matters for common maker projects. A camera that classifies a known set of objects is not the same workload as a local assistant that interprets a prompt and generates text. Both can involve neural networks, but they have different model, memory and software requirements.
Before you spend money, ask these four questions
- Is the exact model and runtime you want supported on the HAT+ 2 today?
- Does your project need a local generative model, or would a smaller vision accelerator be enough?
- Can the Pi 5, HAT, cooling and power supply fit your enclosure and thermal budget?
- Have you compared on-device inference with a cloud service for privacy, network dependence, latency and total cost?
Local inference can keep data on your device and continue when a network is unavailable, but it is not automatically more private or safer: applications still need secure updates, access controls and careful handling of prompts and stored data.
The useful takeaway
The AI HAT+ 2 is a specific accelerator for Raspberry Pi 5, not a magic upgrade that makes every AI model fast or compatible. The 40-TOPS figure is a useful hardware specification when compared on the same precision and workload; it is a poor substitute for checking supported models and application-level results.
Sources: Raspberry Pi’s AI HAT+ 2 announcement; AI HAT+ 2 product information; Raspberry Pi AI HAT documentation.
Featured photo: Photo of the Raspberry Pi AI HAT+ 2 packaging by Suyash Dwivedi. Photo via Wikimedia Commons, licensed under CC BY-SA 3.0. The image was cropped and resized for this article.
Leave a Reply