TPUv4 and More
Estimated Reading Time: 2 minutes
TL;DR
We discuss Google’s new TPUv4 paper and share some interesting articles from around the web.
Google releases TPUv4 Paper
Google has released an Arxiv paper detailing their TPUv4 architecture. The architecture uses optical circuit switches which allows for flexible interconnect topologies like toruses. Toruses have been heavily used in high performance computing, but are only recently finding their uses in machine learning infrastructure
The optical switches themselves are implemented by MEMS mirrors, shown below. Google has recently invested heavily in optical MEMS as part of its “Apollo project”, deploying optical switches in its data centers
TPUv4 also features a new “SparseCore” optimized for handling embeddings and improving the performance of deep learning recommender models on TPUs.
One of the interesting tricks that Google applied was using neural architecture search to automatically optimize architectures for placement onto TPUv4. This computational infrastructure allows for easy algorithmic adaptation of models to TPUs from GPUs.
Although Google has recently stumbled in the product race against ChatGPT, Google has an extremely deep technical foundation that is likely still years ahead of even its strongest rivals. Its investment in optical switches alone is completely groundbreaking. I wouldn’t count Google out of the AI race anytime soon.
Interesting Links from Around the Web
https://spectrum.ieee.org/chip-design-controversy: Controversy over Google’s reinforcement learning method for chip design continues.
https://serokell.io/blog/haskell-in-production-meta: A write-up about Haskell in production at meta.
https://www.nature.com/articles/d41586-023-00989-z: Some carcinogens may trigger tumor growth by causing inflammations and not by directly harming DNA.
https://www.quantamagazine.org/alien-calculus-could-save-particle-physics-from-infinities-20230406/: Resurgence theory is finding use in quantum field theory
Feedback and Comments
Please feel free to email me directly (bharath@deepforestsci.com) with your feedback and comments!
About
Deep Into the Forest is a newsletter by Deep Forest Sciences, Inc. We’re a deep tech R&D company building Chiron, an AI-powered scientific discovery engine. Deep Forest Sciences leads the development of the open source DeepChem ecosystem. Partner with us to apply our foundational AI technologies to hard real-world problems in drug discovery. Get in touch with us at partnerships@deepforestsci.com!
Credits
Author: Bharath Ramsundar, Ph.D.
Editor: Sandya Subramanian, Ph.D.