ACM, the Association for Computing Machinery, today named Torsten Hoefler, a professor at ETH Zurich, the recipient of the ACM Prize in Computing for fundamental contributions to high-performance computing and the ongoing AI revolution.
Hoefler, who was a guest on the @HPCpodcast on the site of this publication’s sister outlet, insideHPC, last November, developed many of the core capabilities of modern supercomputers and defined key aspects of the algorithms for distributing AI models on them, the ACM said.
The ACM Prize in Computing recognizes early-to-mid-career computer scientists whose research contributions have fundamental impact and broad implications. The award carries a prize of $250,000, from an endowment provided by Infosys Ltd., a digital services and consulting company.
ACM cited the work of Hoefler and his colleagues to scale network design and programming in supercomputers, which has revolutionized the capabilities of these
large systems. For example, AI algorithms can now be processed on hundreds of thousands of nodes. Hoefler’s advances in interconnection networks, programming, and parallel algorithms broke new ground in facilitating the use of large-scale massively parallel clusters.
“His numerous innovations across the whole supercomputer stack—including key contributions such as MPI-3 nonblocking collective operations, foundational parallelism strategies for AI models, and high-performance networking systems—have pushed the boundaries of parallel systems design and translated into dramatic improvements in supercomputer performance and scalability,” the ACM said. “Many of those innovations are incorporated into the largest and most powerful machines today.”
The ACM cited these examples of Hoefler’s HPC-AI R&D:
Message Passing Interface 3: Hoefler played a major role in the evolution of the Message Passing Interface (MPI), an informal industry standard for exchanging messages between numerous individual nodes throughout an HPC network. A messaging standard allows synchronization of the activities of each individual computer, sharing data between nodes, and direction and control of the entire parallel network.
The MPI-3 standard was adopted in 2012 and made possible advances in HPC for simulations and AI applications over the past several years. Hoefler chaired both the “Process Topologies” and “Collective Operations” working groups for MPI-3. His nonblocking collective operations, such as Allreduce, Allgather, Bcast, and their respective blocking versions are included in various collective communication libraries—even beyond MPI-3. These operations power the core of distributed deep learning today.
3D Parallelism: Hoefler was among the first to develop and discover the now well-known notion of “3D parallelism,” which drives infrastructure design for the AI industry. Subsequently, he and his collaborators continued to develop techniques for efficient pipelining, sparse communication, model sparsity and quantization. This work has
enabled a cumulative 10-1000x acceleration of AI workloads in modern computers.
Routing Protocols and Network Topologies: The low-level network routing protocols and network topologies that Hoefler and his colleagues developed for such networks as Myrinet and InfiniBand power thousands of AI and HPC supercomputers. These contributions are central to high-performance AI systems that train large-language models, such as ChatGPT.
“The capacity of high-performance computers has become mind-boggling,” said ACM President Yannis Ioannidis. “In just the last two years, we have ushered in the era of
exascale computers, which can perform a billion billion calculations per second. But high-performance computers could do little without the underlying algorithms and standards that allow them to process massive influxes of data.
“While Hoefler introduced many of these innovations while he was still a student, they remain the definitive way to program massively parallel systems today,” Ioannidis continued. “The resulting capabilities of these systems have allowed significant advances in AI, the natural sciences, and many other areas.”
Salil Parekh, CEO, Infosys, said, “We’re in the middle of an exciting era of AI, with great promise for the future. Torsten Hoefler played an important role in advancing high-performance computing, which in turn, fuelled the current AI revolution. The ACM Prize in Computing is intended to recognize to ‘early to mid-career professionals,’ whose work has depth and impact. With much of his important work being done in his 20s, Hoefler is an example for young people that age is not an obstacle to achievement in computing. Infosys is proud to be the sponsor of this award since it was founded in 2007.”
Hoefler is a Professor of Computer Science at ETH Zurich (the Swiss Federal Institute of Technology), where he serves as Director of the Scalable Parallel Computing
Laboratory. He is also the Chief Architect for AI and Machine Learning at the Swiss National Supercomputing Centre (CSCS). Hoefler received a Diplom Informatik (Master of Computer Science) from Chemnitz University of Technology and a PhD in Computer Science from Indiana University.
Hoefler’s honors include the Max Planck-Humboldt Medal, an award for outstanding mid- career scientists; the IEEE CS Sidney Fernbach Award, which recognizes outstanding contributions in the application of high-performance computers; and the ACM Gordon Bell Prize, which recognizes outstanding achievement in high-performance computing. He is a member of the European Academy of Sciences (Academia Europaea), a Fellow of IEEE, and a Fellow of ACM.
Hoefler will be formally presented with the ACM Prize in Computing at ACM’s annual awards banquet on Saturday, June 14 at The Palace Hotel in San Francisco.
