Jump to content

Michael Gschwind

Add links
From Wikipedia, the free encyclopedia
(Redirected from Draft:Michael Gschwind)
Michael Gschwind
Michael Gschwind
Born
Vienna, Austria
NationalityUSA
Alma materTechnische Universität Wien

Michael Karl Gschwind is an American computer scientist who currently is a director and principal engineer at Meta Platforms in Menlo Park, California. He is recognized for his seminal contributions to the design and exploitation of general-purpose programmable accelerators, as an early advocate of sustainability in computer design and as a prolific inventor.[1]

Accelerators[edit]

Gschwind led hardware and software architecture for the first general-purpose programmable accelerator Accelerators and is widely recognized for his contributionsHeterogeneous computing as architect of the Cell Broadband Engine processor used in the Sony PlayStation 3,[2][3] and RoadRunner, the first supercomputer to reach sustained Petaflop operation. As Chief Architect for IBM System Architecture, he led the integration of Nvidia GPUs and IBM CPUs to create the Summit and Sierra supercomputers.

Gschwind was an early advocate for accelerator virtualization[4][5] and as IBM System Chief Architect led I/O and accelerator virtualization [6]

Gschwind has had a critical influence on the development of accelerator programming models with the development of APIs and best practices for accelerator programming,[7][8][9][10][11] application studies for a diverse range of HPC[12] and non-HPC applications.[13] and as co-editor of books [14] and journals[15] on practice and experience of programming accelerator-based systems.

AI Acceleration[edit]

Gschwind was an early advocate of AI Hardware Acceleration with GPUs and programmable accelerators. As IBM's Chief Engineer for AI, he led the development of IBM's first AI products and initiated the PowerAI project which brought to market AI-optimized hardware (originally known as "Minsky systems"), and the first prebuilt hardware-optimized AI frameworks.[16]

At Facebook, Gschwind demonstrated accelerated Large Language Models (LLMs) for Facebook's First Generation ASIC accelerators and for GPUs, leading the first LLLM production deployments at scale for embedding serving for content analysis and platform safety, and for numerous user surfaces such as Facebook Assistant, and FB Marketplace starting in 2020.[17] Gschwind led the development of and is one of the architects of Multiray, an accelerator-based platform for serving foundation models and the first production system to serve Large Language Models at scale in the industry, serving over 800 billion queries per day in 2022.[18][19]

Gschwind led the company-wide adoption of ASIC[20] and Facebook's subsequent "strategic pivot" to GPU Inference, deploying GPU Inference at scale, a move highlighted by FB CEO Mark Zuckerburg in his earnings call. Among the first recommendation models deployed with GPU Inference was a Reels video recommendation model which delivered a 30% user surge within 2 weeks of deployment, as reported by FB CEO Mark Zuckerburg in his Q1 2022 earnings call.[21], and a subsequent $3B to $10B growth for REeels year-over-year[22]

Gschwind also led AI Accelerator Enablement for PyTorch with a particular focus on LLM acceleration, leading the development of Accelerated Transformers[23] (formerly "Better Transformer"[24]) and partnered with companies such as HuggingFace to drive industry-wide LLM Acceleration[25] to establish PyTorch 2.0 as the standard ecosystem for Large Language Models and Generative AI.[26][27][28][29]

Gschwind subsequently led expanding LLM acceleration to on-device AI models with ExecuTorch, the PyTorch ecosystem solution for on-device AI, making on-device generative AI feasible for the first time.[30] ExecuTorch LLM acceleration (across multiple surfaces including NPUs, MPS, and Qualcomm accelerators) delivered significant speedups making it practical to deploy Llama3 unmodified on servers and on-device (demonstrated on iOS, Android, and Raspberry Pi 5) at launch with developers reporting up to 5x-10x speedups over prior on-device AI solutions.[31][32]

Gschwind is a pioneer and advocate of Sustainable AI.[33]

Supercomputer Design[edit]

Gschwind was a chief architect for hardware design and software architecture for several supercomputers, including three top-ranked supercomputer systems Roadrunner (June 2008 – November 2009), Sequoia (June 2012 – November 2012), and Summit (June 2018 – June 2020).

Roadrunner was a supercomputer built by IBM for the Los Alamos National Laboratory in New Mexico, USA. The US$100-million Roadrunner was designed for a peak performance of 1.7 petaflops. It achieved 1.026 petaflops on May 25, 2008, to become the world's first TOP500 LINPACK sustained 1.0 petaflops system.[34][35] It was also the fourth-most energy-efficient supercomputer in the world on the Supermicro Green500 list, with an operational rate of 444.94 megaflops per watt of power used.

Sequoia was a petascale Blue Gene/Q supercomputer constructed by IBM for the National Nuclear Security Administration as part of the Advanced Simulation and Computing Program (ASC). It was delivered to the Lawrence Livermore National Laboratory (LLNL) in 2011 and was fully deployed in June 2012.[36] Sequoia was dismantled in 2020, its last position on the top500.org list was #22 in the November 2019 list.

Summit is a supercomputer developed by IBM for use at Oak Ridge Leadership Computing Facility (OLCF), a facility at the Oak Ridge National Laboratory. It held the number 1 position from November 2018 to June 2020.[37][38] Its current LINPACK benchmark is clocked at 148.6 petaFLOPS.[39]

Many-Core Processor Design[edit]

Gschwind was an early advocate of many-core processor design to overcome the power and performance limitations of single-processor designs. Gschwind co-authored an analysis of the limitations of frequency scaling which arguably led to an industry-wide transition to many-core designs.[40] Gschwind was a lead architect for several many-core designs, including the first commercial many-core processor Cell with 9 cores, BlueGene/Q with 18 cores, and several enterprise and mainframe processors (POWER7/POWER8/POWER9 with up to 24 cores; z10-z15 with up to 12 cores).

Compiler Technologies[edit]

Gschwind has made seminal contributions to compiler technology, with a particular emphasis on pioneering contributions to just-in-time compilation, dynamic optimization, binary translation and compilers in supercomputing.

Just-in-time-Compilation[edit]

Gschwid was an early proponent of just-in-time compilation and has been a driving force in the field. He has proposed critical improvements for the implementation of JIT compilation based systems, with a particular view to dynamic optimization, binary translation and virtual machine implementation. Gschwind's contributions includes implementation of precise exceptions with deferred state materialization,[41] high-performance computing optimization such as software pipelining at JIT translation time,[42][43] hardware/software co-design for binary emulation and dynamic optimization[44][45][46][47] Gschwind's seminal contributions to Virtual Machine design and implementation are reflected by being the most-cited author in the `Virtual Machines' textbook by Smith and Nair.[48]

Compilation for Accelerators and Accelerator-based Supercomputers[edit]

Gschwind is credited with seminal contributions for compiling general-purpose programmable accelerators and GPUs, supporting the launch of the nascent discipline as keynote speaker at the frst General-Purpose Programmable GPU workshop (GPGPU). His contributions include code partitioning, code optimization, code partitioning and APIs for accelerators[49] [50][51][52]

His innovations include compiler/hardware co-design for integrated register files to resolve phase ordering issues in auto-vectorization between unit assignment and vectorization decisions to simplify the cost model, an innovation adopted by general-purpose programmable accelerators, including the Cell SPU and GPUseneral-purpose CPU designs, starting with Gschwind's pioneering work for SIMD CPU accelerators.

More recently, his contributions to HPC compilation have included pioneering work in enabling high-performance execution of AI workloads[53][54][55]

System and Compiler APIs[edit]

Gschwind led the development of the ELFv2 Power execution environment, which has been broadly adopted for Power executin environments. Advantageously, the new environment updates the APIs and ABIs for object-oriented environments. Departing from traditional Power architecture big-endian data conventions, the ELFv2 ABI and APIs were first launched to support a new little-endian version of Linux on Power. This has since been adopted in particular for the new little-endian and to support GPU acceleration with Nvidia GPUs.[56][57][58]

SIMD Parallel Vector Architecture[edit]

Gschwind is a pioneer of SIMD parallel vector architecture to increase the number of operations which can be performed per cycle. To enable efficient compilation, Gschwind proposed the implementation of merged scalar and vector execution units, eliminating the cost of copies between scalar and vectorized code, and simplifying compiler architecture by resolving phase ordering problems in compilers.

The Cell's accelerator cores (Synergistic Processor Unit SPU) contain a single 128 element register file with 128 bit per register. Registers may hold either scalar or a vector of multiple values.[59] The simplified cost model leads to significantly improved vectorization success, improving overall program performance and efficiency.[60]

The vector-scalar approach was also adopted by the IBM Power VSX (Vector Scalar Extension) SIMD instructions,[61] BlueGene/Q vector instructions[62][63] and System/z mainframe vector instruction set,[64][65] the design of all three IBM vector-scalar architectures having been led by Gschwind as Chief Architect for IBM System Architecture.

Service, Education, Diversity, Inclusion and the Digital Inclusion[edit]

Gschwind is a strong believer in the power of education and its power to help overcome the effects of all types of discrimination and colonialism. He has served as faculty member at [Princeton] and [TU Wien] to advance education. To contribute to overcoming the effects of colonialism and bridge the digital divide, Gschwind has volunteered in Senegal to contribute to the expansion and improvement of Senegal's education and research network, snRER.

Background[edit]

Gschwind was born in Vienna and obtained his doctorate degree in Computer Engineering at the Technische Universität Wien in 1996. He joined the IBM Thomas J. Watson Research Center in Yorktown Heights, NY and also held positions IBM Systems product group and at its corporate headquarter in Armonk, NY. At Huawei, Gschwind served Vice President of Artificial Intelligence and Accelerated Systems at Huawei. Gschwind is currently a director at Meta Platforms where he has been responsible for AI Acceleration and AI infrastructure.[citation needed]

References[edit]

  1. ^ "Michael Karl Gschwind". www.ppubs.uspto.gov.
  2. ^ David Becker (December 3, 2004). "PlayStation 3 chip goes easy on developers". CNET. Retrieved January 13, 2019.
  3. ^ Scarpino, M. (2008). Programming the cell processor: for games, graphics, and computation. Pearson Education.
  4. ^ https://on-demand.gputechconf.com/gtc/2017/presentation/S7320-tim-kaldewey-optimizing-efficiency-of-deep-learning-workloads-through-gpu-virtualization.pdf, https://on-demand.gputechconf.com/gtc/2017/presentation/S7320-tim-kaldewey-optimizing-efficiency-of-deep-learning-workloads-through-gpu-virtualization.pdf
  5. ^ Optimizing the efficiency of deep learning through accelerator virtualization, https://ieeexplore.ieee.org/document/8030299
  6. ^ I/O Vrtualization and System Acceleration in Power9, https://old.hotchips.org/wp-content/uploads/hc_archives/hc27/HC27.24-Monday-Epub/HC27.24.30-HP-Cloud-Comm-Epub/HC27.24.340-IO-Virtualization-POWER8-Gschwind-IBM.pdf
  7. ^ Gschwind, M. The Cell Broadband Engine: Exploiting Multiple Levels of Parallelism in a Chip Multiprocessor. Int J Parallel Prog 35, 233–262 (2007). https://doi.org/10.1007/s10766-007-0035-4
  8. ^ Integrated execution: A programming model for accelerators, IBM JRD, https://www.researchgate.net/publication/224123640_Integrated_execution_A_programming_model_for_accelerators
  9. ^ Chip Multiprocessing and the Cell Broadband Engine, https://computingfrontiers.org/2006/cf06-gschwind.pdf
  10. ^ CBE Programming Handbook
  11. ^ CBE Programming Tutorial, https://public.dhe.ibm.com/software/dw/cell/CBE_Programming_Tutorial_v3.1.pdf
  12. ^ Application acceleration with the cell broadband engine, https://experts.illinois.edu/en/publications/application-acceleration-with-the-cell-broadband-engine
  13. ^ Cell GC: using the cell synergistic processor as a garbage collection coprocessor, ACM Virtual Execution Environments, https://dominoweb.draco.res.ibm.com/reports/rc24520.pdf
  14. ^ M. Gschwind, F. Gustavson, J. Prins (eds), High Performance Computing with the Cell Broadband Engine Scientific Programming 2009, https://www.semanticscholar.org/paper/High-Performance-Computing-with-the-Cell-Broadband-Gschwind-Gustavson/c6775765100eb3b9eb7b7bc003a8eba1ca90667f
  15. ^ M. Gschwind, M. Perrone (Eds), Topical Issue On Hybrid Systems IBM Journal of Research and Development 53(5):1-2 September 2009, DOI:10.1147/JRD.2009.5429079
  16. ^ PowerAI: A Co-Optimized Software Stack for AI on Power, GTC 2017, San Jose, https://www.researchgate.net/publication/316844295_PowerAI_A_Co-Optimized_Software_Stack_for_AI_on_Power
  17. ^ GTC Panel on Large Language Models: From Ingestion to Deployment for Large Language Models, https://www.nvidia.com/en-us/on-demand/session/gtcfall22-a41325/
  18. ^ "MultiRay: Optimizing efficiency for large-scale AI models". ai.meta.com. Retrieved 2023-10-28.
  19. ^ MultiRay: An Accelerated Embedding Service for Content Understanding, https://static.sched.com/hosted_files/pytorch2023/60/PyTorch_Conf_2023-Multiray.pdf
  20. ^ First-Generation Inference Accelerator Deployment at Facebook, https://arxiv.org/pdf/2107.04140.pdf
  21. ^ Mark Zuckerberg says AI boosts monetization by 30% on Instagram, 40% on Facebook, https://finance.yahoo.com/news/mark-zuckerberg-says-ai-boosts-monetization-by-30-on-instagram-40-on-facebook-181123177.html?fr=sycsrp_catchall
  22. ^ From $3B to $10B: Meta's AI-Driven Reels Skyrocketed Revenue Growth Beyond Expectations, https://www.benzinga.com/news/23/07/33414780/from-3b-to-10b-metas-ai-driven-reels-skyrocketed-revenue-growth-beyond-expectations
  23. ^ "PyTorch". www.pytorch.org. Retrieved 2023-10-28.
  24. ^ "A BetterTransformer for Fast Transformer Inference". pytorch.org. Retrieved 2023-10-28.
  25. ^ BetterTransformer, Out of the Box Performance for Hugging Face Transformers, https://medium.com/pytorch/bettertransformer-out-of-the-box-performance-for-huggingface-transformers-3fbe27d50ab2
  26. ^ PyTorch 2.0: Our next generation release that is faster, more Pythonic and Dynamic as ever, https://pytorch.org/blog/pytorch-2.0-release/
  27. ^ Accelerated Generative Diffusion Models with PyTorch 2, https://pytorch.org/blog/accelerated-generative-diffusion-models/
  28. ^ Accelerating Large Language Models with Accelerated Transformers, https://pytorch.org/blog/accelerating-large-language-models/
  29. ^ PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation, https://pytorch.org/assets/pytorch2-2.pdf
  30. ^ ExecuTorch Alpha: Taking LLMs and AI to the Edge with Our Community and Partners, https://pytorch.org/blog/executorch-alpha/
  31. ^ Layla v4.6.0 has been published!, https://www.layla-network.ai/post/layla-v4-6-0-has-been-published
  32. ^ Blazing fast LLama2-7B-Chat on 8GB RAM Android device via Executorch, https://www.reddit.com/r/LocalLLaMA/comments/1csw861/blazing_fast_llama27bchat_on_8gb_ram_android/
  33. ^ Sustainable AI: Environmental Implications, Challenges and Opportunities, https://arxiv.org/pdf/2111.00364.pdf
  34. ^ Gaudin, Sharon (2008-06-09). "IBM's Roadrunner smashes 4-minute mile of supercomputing". Computerworld. Archived from the original on 2008-12-24. Retrieved 2008-06-10.
  35. ^ Fildes, Jonathan (2008-06-09). "Supercomputer sets petaflop pace". BBC News. Retrieved 2008-06-09.
  36. ^ NNSA awards IBM contract to build next generation supercomputer, February 3, 2009
  37. ^ Lohr, Steve (8 June 2018). "Move Over, China: U.S. Is Again Home to World's Speediest Supercomputer". The New York Times. Retrieved 19 July 2018.
  38. ^ "Top 500 List - November 2022". TOP500. November 2022. Retrieved 13 April 2022.
  39. ^ "November 2022 | TOP500 Supercomputer Sites". TOP500. Retrieved 13 April 2022.
  40. ^ Optimizing pipelines for power and performance, MICRO 2002. https://www.researchgate.net/publication/4001353_Optimizing_pipelines_for_power_and_performance
  41. ^ Efficient Instruction Scheduling with Precise Exceptions, https://www.researchgate.net/publication/244186152_Efficient_instruction_scheduling_with_precise_exceptions
  42. ^ Optimizations and oracle parallelism with dynamic translation, https://www.researchgate.net/publication/3830428_Optimizations_and_oracle_parallelism_with_dynamic_translation
  43. ^ Dynamic and Transparent Binary Translation, https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=ee7ad16a1f0c1988e93209d4b56d7ff4e8b68566
  44. ^ Dynamic binary translation and optimization, https://www.researchgate.net/publication/3044344_Dynamic_binary_translation_and_optimization
  45. ^ Advances and future challenges in binary translation and optimization, https://ieeexplore.ieee.org/document/964447
  46. ^ Binary translation and architecture convergence issues for IBM System/390, https://www.researchgate.net/profile/Michael-Gschwind/publication/221235791_Binary_translation_and_architecture_convergence_issues_for_IBM_system390/links/0046352f27d9de5653000000/Binary-translation-and-architecture-convergence-issues-for-IBM-system-390.pdf
  47. ^ Advances and future challenges in binary translation and optimization, Proceedings of the IEEE, https://ieeexplore.ieee.org/document/964447
  48. ^ Smith, Nair, Virtual Machines: Versatile Platforms for Systems and Processes, https://www.amazon.com/Virtual-Machines-Versatile-Platforms-Architecture/dp/1558609105
  49. ^ Optimizing Compiler for the CELL Processor, Conference on Parallel Architectures and Compilation Techniques (PACT 2005), September 2005. https://dl.acm.org/doi/10.1109/PACT.2005.33
  50. ^ An Open Source Environment for Cell Broadband Engine System Software, https://www.researchgate.net/publication/2961855_An_Open_Source_Environment_for_Cell_Broadband_Engine_System_Software
  51. ^ Chip Multiprocessing and the Cell Broadband Engine, https://www.computingfrontiers.org/2006/cf06-gschwind.pdf
  52. ^ The Cell Broadband Engine: Exploiting Multiple Levels of Parallelism in a Chip Multiprocessor, https://link.springer.com/article/10.1007/s10766-007-0035-4
  53. ^ First-Generation Inference Accelerator Deployment at Facebook, https://research.facebook.com/publications/first-generation-inference-accelerator-deployment-at-facebook
  54. ^ PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation, https://pytorch.org/assets/pytorch2-2.pdf
  55. ^ ExecuTorch Alpha: Taking LLMs and AI to the Edge with Our Community and Partners, https://pytorch.org/blog/executorch-alpha/
  56. ^ OpenPOWER Reengineering a server ecosystem for large-scale data centers, https://old.hotchips.org/wp-content/uploads/hc_archives/hc26/HC26-12-day2-epub/HC26.12-7-Dense-Servers-epub/HC26.12.730-%20OpenPower-Gschwind-IBM.pdf
  57. ^ Power Architecture 64-Bit ELF V2 ABI Specification, https://ftp.rtems.org/pub/rtems/people/sebh/ABI64BitOpenPOWERv1.1_16July2015_pub.pdf
  58. ^ Reengineering a server ecosystem for enhanced portability and performance, https://www.researchgate.net/publication/322706081_Reengineering_a_server_ecosystem_for_enhanced_portability_and_performance
  59. ^ Synergistic Processing in Cell's Multicore Architecture, IEEE MICRO, https://ieeexplore.ieee.org/document/1624323
  60. ^ Optimizing Compiler for the CELL Processor, Conference on Parallel Architectures and Compilation Techniques (PACT 2005), September 2005. https://dl.acm.org/doi/10.1109/PACT.2005.33
  61. ^ Workload acceleration with the IBM POWER vector-scalar architecture, IBM JRD, https://ieeexplore.ieee.org/abstract/document/7442604
  62. ^ The IBM Blue Gene/Q Compute Chip, https://ieeexplore.ieee.org/document/6109225
  63. ^ Morgan, Timothy Prickett (22 November 2010). "IBM uncloaks 20 petaflops BlueGene/Q super". The Register.
  64. ^ The SIMD accelerator for business analytics on the IBM z13, https://dl.acm.org/doi/10.1147/JRD.2015.2426576
  65. ^ SIMD Processing on IBM z14, z13 and z13s, https://www.ibm.com/downloads/cas/WVPALM0N
Retrieved from "https://en.wikipedia.org/w/index.php?title=Michael_Gschwind&oldid=1229880080"