Neural Processing Units (NPU)

Neural Processing Units (NPU)

1. Artificial Neural Network (ANN)

• An Artificial Neural Network (ANN) is an assembly of interconnected nodes and weighted links, inspired by the architecture of the biological brain. wwwwbord lovel

• Nodes in an ANN are called neurons, drawing an analogy with biological neurons. These simple functional units are capable of learning classification problems after being trained with sufficient data.

• An ANN can consist of multiple layers, classified into three types: input, hidden and output layers.

■ Input layer : Receives the initial data (like an image or text).

■ Hidden layer(s) : Process the input data through multiple layers, extracting features and patterns.

■ Output layer : Produces the final result, such as a classification (e.g., identifying an object in an image) or a prediction.

• The simplest form of an ANN is the perceptron model, illustrated in Fig. 6.8.1. It receives input signals, each with a certain "strength" (weight). It adds up all the weighted input signals. If the total sum exceeds a certain "threshold," specified by a simple threshold function 0, the output 1; otherwise, the output is 0.

• The output function can be expressed as

• The perceptron model is a linear classifier with a simple and efficient learning algorithm. Beyond the threshold function, several other common activation functions, such as sigmoid, tanh and step functions, are also widely used.

2. Need for Specialized Hardware for ANN Processing

• Although ANNs are powerful, they require extremely large numbers of:

■ Arithmetic operations, especially Multiply ‒ Accumulate (MAC)

■ Matrix and vector computations

■ Parallel data processing

• Running these operations on general‒purpose processors such as CPUs is slow and energy‒ intensive. GPUs improve performance, but modern AI applications require even higher efficiency, especially on smartphones, IoT devices and embedded systems.

• This led to the development of Neural Processing Units (NPUs).

3. Processing Unit (PU)

• A Neural Processing Unit (NPU) is a special type of chip made to handle AI and Machine Learning tasks quickly and efficiently. It's like a brain for AI.

• While GPUs are good at doing many things at the same time (parallel processing), NPUs are even better for AI because they are designed only for AI tasks. They focus on the key math operations that help neural networks learn and make predictions, such as:

■ Matrix multiplications

■ Convolutions (used in image recognition)

■ Activation functions (which help models make decisions)

• By focusing only on these AI tasks, NPUS can work faster and use less energy than other chips like CPUs or GPUs.

4. Key Characteristics of NPU

• NPUs are optimized for the unique computational patterns of neural networks, focusing on:

1. Parallel processing : Like GPUs, NPUs are highly parallel, but their architecture is tailored to the data flow of neural networks. They can break down complex AI problems into many smaller, independent calculations and process them simultaneously, leading to significant speedups.

2. Low precision arithmetic : Many AI models, especially during inference (the stage where a trained model makes predictions), can operate effectively with lower numerical precision (e.g., 8‒bit integers or bfloat16 instead of 32‒bit floating‒point). NPUs are designed to leverage this, performing more operations per clock cycle with less power consumption.

3. Efficient memory access : Neural networks often involve moving large amounts of data (weights, biases, activations). NPUs are designed with optimized memory hierarchies and high‒bandwidth memory to minimize data transfer bottlenecks and ensure a continuous flow of data to their processing units.

4. Dedicated hardware blocks : Instead of general‒purpose cores, NPUS often feature specific hardware modules tailored for operations common in neural networks, such as:

♦ Matrix Multiplication Units (MMUs) : Similar to the systolic arrays in Google's TPUs, these are highly optimized for fast matrix computations.

♦ Activation function units : Hardware logic for common non‒linear activation functions (e.g., ReLU, sigmoid),

♦ Pooling and convolutional units : Accelerators for operations frequently found in Convolutional Neural Networks (CNNs) for image processing,

5. Energy efficiency : A primary driver for NPU development, especially for edge devices (smartphones, IoT devices, embedded systems), is power efficiency. By specializing in AI tasks and using lower precision, NPUs can perform complex AI computations with significantly less power than a general‒purpose CPU or even a high‒end GPU. This translates to longer battery life in mobile devices and lower energy consumption in always‒on systems.

6. "On‒Device" AI : NPUs enable AI processing to occur directly on the device, rather than relying solely on cloud servers. This offers several benefits:

♦ Reduced latency : Faster response times for AI applications, crucial for real‒time scenarios (e.g., autonomous driving, voice assistants).

♦ Enhanced privacy : User data stays on the device, reducing the need to send sensitive information to the cloud.

♦ Reduced bandwidth : Less data needs to be transmitted over networks, saving bandwidth and improving performance in areas with limited connectivity.

♦ Offline capability : AI features can function even without an internet connection.

5. NPU vs. CPU Vs GPU for AI

• The relationship between NPUs, CPUs, and GPUs is often complementary in modern computing systems:

6. Applications of NPUs

• NPUs are becoming ubiquitous across various domains, especially where on‒device AI and efficiency are critical:

■ Smartphones : Facial recognition (Face ID, Android unlock), real‒time language translation, advanced computational photography (e.g., image enhancement, background blur, object detection), voice assistants, gesture recognition.

■ Laptops and PCs (AI PCs) : Features like Windows Studio Effects (background blur, eye contact correction during video calls), AI‒powered noise cancellation, smart search, local execution of generative AI models (e.g., for text summarization or image generation within applications), enhanced security features.

■ Smart home devices : Voice assistants, smart cameras (object detection, facial recognition), intelligent thermostats.

■ IoT and edge devices : Industrial automation (predictive maintenance, quality control), smart city applications (traffic monitoring, security cameras), drones (navigation, object avoidance), wearables (health monitoring, activity tracking).

■ Autonomous vehicles : Real‒time processing of sensor data (cameras, lidar, radar) for object detection, lane keeping, pedestrian recognition and decision‒making.

■ Robotics : Vision processing, navigation, human‒robot interaction.

■ Medical devices : Portable diagnostic tools, real‒time image analysis.

7. Major NPU Manufacturers

1. Apple : Their "Neural Engine" integrated into A‒series (iPhone) and M‒series (Mac) chips.

2. Qualcomm : Hexagon NPU in their Snapdragon mobile platforms.

3. Intel : "AI Boost" NPU integrated into their Core Ultra (Meteor Lake, Lunar Lake) processors and their dedicated Habana Labs AI accelerators (Gaudi).

4. AMD : Ryzen AI engine in their Ryzen processors for laptops and desktops.

5. Arm : Designs NPU intellectual property (e.g., Ethos series) that other chip manufacturers license and integrate into their SoCs.

6. Samsung : Integrates NPUs into its Exynos mobile processors.

7. Huawei : Developed its own Ascend series of AI processors.

8. Dedicated AI Chip Startups : Companies like Hailo, Groq, and others are developing highly specialized NPU / AI accelerator solutions for various markets.

Review Questions

1. What is ANN?

2. What is the need for specialized hardware for ANN processing?

3. What is an NPU?

4. Explain the key characteristics of the NPU.

5. Give a comparison between NPU, CPU and GPU.

6. State the applications of NPU.

7. Name some major companies developing NPUs and explain how they integrate NPUs into their products.