AI accelerator

From Wikipedia, the free encyclopedia
Jump to: navigation, search

An AI accelerator is a class of microprocessor[1] or computer system[2] designed to accelerate artificial neural networks, machine vision and other machine learning algorithms for robotics, internet of things and other data-intensive or sensor-driven tasks.[3] They are often manycore designs and generally focus on low-precision arithmetic. A number of vendor-specific terms exist for devices in this space.

History of AI acceleration[edit]

Computer systems have frequently complemented the CPU with special purpose accelerators for specialized tasks, most notably video cards for graphics, but also sound cards for sound, etc. As Deep learning and AI workloads rose in prominence, specialize hardware was created or adapted from previous products to accelerate these tasks.

Early attempts[edit]

As early as 1993, DSPs have been used as neural network accelerators e.g. to accelerate OCR software,.[4] In the 1990s there were also attempts to create parallel high throughput systems for workstations aimed at various applications, including neural network simulations.[5][6][7] FPGA-based accelerators were also first explored in the 1990s for both inference[8] and training.[9] ANNA was a neural net CMOS accelerator developed by Yann LeCun.[10]

Heterogeneous computing[edit]

Heterogeneous computing began the incorporation of a number of specialized processors in a single system, or even a single chip, each optimized for a specific type of task. Architectures such as the Cell microprocessor[11] have features significantly overlapping with AI accelerators including: support for packed low precision arithmetic, dataflow architecture, and prioritising 'throughput' over latency. The Cell microprocessor would go on to be applied to a number of tasks[12][13][14] including AI.[15][16][17]

CPUs themselves also gained increasingly wide SIMD units (driven by video and gaming workloads) and support for packed low precision data types.[18]

Use of GPU[edit]

Graphics processing units or GPUs are specialized hardware for the manipulation of images. As the mathematical basis of neural networks and image manipulation are similar, embarrassingly parallel tasks involving matrices, GPUs became increasingly used for machine learning tasks.[19][20][21] As such, as of 2016 GPUs are popular for AI work, and they continue to evolve in a direction to facilitate deep learning, both for training[22] and inference in devices such as self-driving cars.[23] - and gaining additional connective capability for the kind of dataflow workloads AI benefits from (e.g. Nvidia NVLink).[24] As GPUs have been increasingly applied to AI acceleration, GPU manufacturers have incorporated neural network specific hardware to further accelerate these tasks.[25] Tensor cores are intended to speed up the training of neural networks.[25]

Use of FPGA[edit]

Deep learning frameworks are still evolving, making it hard to design custom hardware. Reconfigurable devices like field-programmable gate arrays (FPGA) make it easier to evolve hardware, frameworks and software alongside each other.[8][9][26]

Microsoft has used FPGA chips to accelerate inference.[27][28] The application of FPGAs to AI acceleration has also motivated Intel to purchase Altera with the aim of integrating FPGAs in server CPUs, which would be capable of accelerating AI as well as general purpose tasks.[citation needed]

Emergence of dedicated AI accelerator ASICs[edit]

Whilst GPUs and FPGAs perform far better than CPUs for these AI related tasks, a factor of 10 in efficiency[29][30] can still be gained with a more specific design, via an application-specific integrated circuit (ASIC).[citation needed] These include differences in memory use[citation needed] and the use of lower precision numbers.[31][32]


As of 2016, the field is still in flux and vendors are pushing their own marketing term for what amounts to an "AI accelerator", in the hope that their designs and APIs will dominate. There is no consensus on the boundary between these devices, nor the exact form they will take, however several examples clearly aim to fill this new space, with a fair amount of overlap in capabilities.

In the past when consumer graphics accelerators emerged, the industry eventually adopted Nvidia's self-assigned term, "the GPU",[33] as the collective noun for "graphics accelerators", which had taken many forms before settling on an overall pipeline implementing a model presented by Direct3D.


Stand alone products[edit]

GPU based products[edit]

  • Nvidia Tesla is Nvidia's line of GPU derived products marketed for GPGPU and AI tasks.
    • Nvidia Volta is a microarchitecture which augments the Graphics processing unit with additional 'tensor units' targeted specifically at accelerating calculations for neural networks[37]
    • Nvidia DGX-1 is a Nvidia workstation/server product which incorporates Nvidia brand GPUs for GPGPU tasks including machine learning.[38]
  • Radeon Instinct is AMD's line of GPU derived products for AI acceleration.[39]

AI accelerating co-processors[edit]

  • The processor in Qualcomms mobile platform Snapdragon 845 contains a Hexagon 685 DSP core for AI processing in camera, voice, XR and gaming applications
  • PowerVR 2NX NNA (Neural Net Accelerator) is an IP core from Imagination Technologies licensed for integration into chips.[40]
  • Neural Engine is an AI accelerator core within the Apple A11 Bionic SoC.[41]
  • Cadence Tensilica Vision C5 is a neural networks optimized DSP IP core[42]
  • The Neural Processing Unit is a neural network accelerator within the HiSilicon Kirin 970[43]
  • January 2018 CEVA, Inc. launched a family of four AI processors called NeuPro, each containing one programmable vector DSP and one hardwired implementation of 8-bit or 16-bit neural network layers supporting neural nets with performances ranging from 2 TOPS thru 12.5 TOPS. [44]

Research and unreleased products[edit]

Potential applications[edit]

See also[edit]


  1. ^ "Intel unveils Movidius Compute Stick USB AI Accelerator". 
  2. ^ "Inspurs unveils GX4 AI Accelerator". 
  3. ^ "google developing AI processors". google using its own AI accelerators.
  4. ^ "convolutional neural network demo from 1993 featuring DSP32 accelerator". 
  5. ^ "design of a connectionist network supercomputer". 
  6. ^ "The end of general purpose computers (not)". This presentation covers a past attempt at neural net accelerators, notes the similarity to the modern SLI GPGPU processor setup, and argues that general purpose vector accelerators are the way forward (in relation to RISC-V hwacha project. Argues that NN's are just dense and sparse matrices, one of several recurring algorithms)
  7. ^ "SYNAPSE-1: a high-speed general purpose parallel neurocomputer system". 
  8. ^ a b "Space Efficient Neural Net Implementation" (PDF). 
  9. ^ a b "A Generic Building Block for Hopfield Neural Networks with On-Chip Learning" (PDF). 
  10. ^ Application of the ANNA Neural Network Chip to High-Speed Character Recognition
  11. ^ "Synergistic Processing in Cell's Multicore Architecture". 
  12. ^ "Performance of Cell processor for biomolecular simulations" (PDF). 
  13. ^ "Video Processing and Retrieval on Cell architecture". 
  14. ^ "Ray Tracing on the Cell Processor". 
  15. ^ "Development of an artificial neural network on a heterogeneous multicore architecture to predict a successful weight loss in obese individuals" (PDF). 
  16. ^ "Parallelization of the Scale-Invariant Keypoint Detection Algorithm for Cell Broadband Engine Architecture". 
  17. ^ "Data Mining Algorithms on the Cell Broadband Engine". 
  18. ^ "Improving the performance of video with AVX". 
  19. ^ "microsoft research/pixel shaders/MNIST". 
  20. ^ "how the gpu came to be used for general computation". 
  21. ^ "imagenet classification with deep convolutional neural networks" (PDF). 
  22. ^ "nvidia driving the development of deep learning". 
  23. ^ "nvidia introduces supercomputer for self driving cars". 
  24. ^ "how nvlink will enable faster easier multi GPU computing". 
  25. ^ a b Harris, Mark (May 11, 2017). "CUDA 9 Features Revealed: Volta, Cooperative Groups and More". Retrieved August 12, 2017. 
  26. ^ "FPGA Based Deep Learning Accelerators Take on ASICs". The Next Platform. 2016-08-23. Retrieved 2016-09-07. 
  27. ^ "microsoft extends fpga reach from bing to deep learning". 
  28. ^ "Accelerating Deep Convolutional Neural Networks Using Specialized Hardware" (PDF). 
  29. ^ "Google boosts machine learning with its Tensor Processing Unit". 2016-05-19. Retrieved 2016-09-13. 
  30. ^ "Chip could bring deep learning to mobile devices". 2016-02-03. Retrieved 2016-09-13. 
  31. ^ "Deep Learning with Limited Numerical Precision" (PDF). 
  32. ^ Rastegari, Mohammad; Ordonez, Vicente; Redmon, Joseph; Farhadi, Ali (2016). "XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks". arXiv:1603.05279Freely accessible [cs.CV]. 
  33. ^ "NVIDIA launches the World's First Graphics Processing Unit, the GeForce 256,". 
  34. ^ Kampman, Jeff (17 October 2017). "Intel unveils purpose-built Neural Network Processor for deep learning". Tech Report. Retrieved 18 October 2017. 
  35. ^ "Intel Nervana Neural Network Processors (NNP) Redefine AI Silicon". Retrieved 20 October 2017. 
  36. ^ "The Evolution of EyeQ". 
  37. ^ "Nvidia goes beyond the GPU for AI with Volta". 
  38. ^ "nvidia dgx-1" (PDF). 
  39. ^ Smith, Ryan (12 December 2016). "AMD Announces Radeon Instinct: GPU Accelerators for Deep Learning, Coming in 2017". Anandtech. Retrieved 12 December 2016. 
  40. ^ "The highest performance neural network inference accelerator". 
  41. ^ "The iPhone X's new neural engine exemplifies Apple's approach to AI". The Verge. Retrieved 2017-09-23. 
  42. ^ "Cadence Unveils Industry's First Neural Network DSP IP for Automotive, Surveillance, Drone and Mobile Markets". 
  43. ^ "HUAWEI Reveals the Future of Mobile AI at IFA 2017". 
  44. ^ "A Family of AI Processors for Deep Learning at the Edge". 
  45. ^ Elon Musk confirms that Tesla is working on its own new AI chip led by Jim Keller Elon Musk confirms that Tesla is working on its own new AI chip led by Jim Keller Check |url= value (help).  Missing or empty |title= (help)
  46. ^ Chen, Yu-Hsin; Krishna, Tushar; Emer, Joel; Sze, Vivienne (2016). "Eyeriss: An Energy-Efficient Reconfigurable Accelerator for Deep Convolutional Neural Networks". IEEE International Solid-State Circuits Conference, ISSCC 2016, Digest of Technical Papers. pp. 262–263. 
  47. ^ "kalray MPPA" (PDF). 
  48. ^ "Graphcore Technology". 
  49. ^ "Wave Computing's DPU architecture". 
  50. ^ "A 2.9 TOPS/W Deep Convolutional Neural Network SoC in FD-SOI 28nm for Intelligent Embedded Systems" (PDF). 
  51. ^ "NM500, Neuromorphic chip with 576 neurons". 
  52. ^ "yann lecun on IBM truenorth". argues that spiking neurons have never produced leading quality results, and that 8-16 bit precision is optimal, pushes the competing 'neuflow' design
  53. ^ "IBM cracks open new era of neuromorphic computing". TrueNorth is incredibly efficient: The chip consumes just 72 milliwatts at max load, which equates to around 400 billion synaptic operations per second per watt — or about 176,000 times more efficient than a modern CPU running the same brain-like workload, or 769 times more efficient than other state-of-the-art neuromorphic approaches 
  54. ^ "Intel's New Self-Learning Chip Promises to Accelerate Artificial Intelligence". 
  55. ^ "BrainChip Accelerator". 
  56. ^ "India preps RISC-V Processors - Shakti targets servers, IoT, analytics". The Shakti project now includes plans for at least six microprocessor designs as well as associated fabrics and an accelerator chip 
  57. ^ "drive px". 
  58. ^ "design of a machine vision system for weed control" (PDF). 
  59. ^ "qualcomm research brings server class machine learning to every data devices". 
  60. ^ "movidius powers worlds most intelligent drone". 

External links[edit]