Fox received a Ph.D. in Theoretical Physics from Cambridge University, where he was Senior Wrangler. He is now a Professor in the Biocomplexity Institute and Computer Science Department at the University of Virginia. He previously held positions at Caltech, Syracuse University, Florida State University, and Indiana University. after being a postdoc at the Institute for Advanced Study at Princeton, Lawrence Berkeley Laboratory, and Peterhouse College Cambridge. He has supervised the Ph.D. of 79 students. He has an hindex of 90 with over 46,000 citations. He received the High-Performance Parallel and Distributed Computing (HPDC) Achievement Award and the ACM – IEEE CS Ken Kennedy Award for Foundational contributions to parallel computing in 2019. He is a Fellow of APS (Physics) and ACM (Computing) and works on the interdisciplinary interface between computing and applications. He is active in the Industry consortia MLCommons/MLPerf and the AI Alliance.
Presentation Title:
Benchmarking for HPC-AI in a rapidly changing Context
Presentation Abstract:
We discuss the use of benchmarking for Science based on work with MLCommons and the AI Alliance. Benchmarks consist of models, datasets and metrics where the latter can address both the computing machine and Science performance. They can help in testing and designing new hardware and software and in system procurements. However they can also act as exemplars for use in education and in capturing an algorithmic motif that can be applied across multiple scientific domains. Benchmarks are often viewed as relatively static to allow the effect of system changes to be tracked. However that doesn’t work very well in today’s AI arena where algorithms and hardware are both changing rapidly and exhibit great diversity across users and science domains. We examine two activities of the MLCommons Science/HPC working group. One is a catalog of science benchmarks organized by domain and computational motif. Another is for Time Series where over 100 new deep learning models have been proposed over the last four years. We suggest that the lack of careful benchmarks in this field seems to have limited progress. We also note that modern LLM’s are quite powerful for keeping track of progress in today’s chaotic changing diverse environment.