AI Has Not Yet Hit Scaling Limits
AI Has Not Yet Hit Scaling Limits
Estimated Reading Time: 6 minutes
TL;DR
This week we start by discussing the first potentially significant deviation from the predictions of the standard model of physics. Afterwards we cover two major advances in AI scaling this week, with OpenAI’s DALL-E 2 and Google’s Pathways Language Model. These models demonstrate that large AI models have yet to hit the limits of increased scaling. We also note the dangerous loss of US naval advantage with respect to the PLA Navy.
A Significant Deviation in the Mass of the W Boson?
The W-boson is the mediator of the weak nuclear force. The Fermilab Tevatron collider measured about 4 millions W-bosons and has found that the measured mass deviates from the predictions of the Standard Model of Physics. The result is highly significant at 7 sigma (that is, the measurement is 7 standard deviations away from what would be predicted by the standard model). A surprising postscript to the story is that the collider detector at Fermilab was actually shut down a decade ago! (source.) These new results came from a re-analysis of older collision data.
The standard model is a complicated construction that explains much of known physics (with the exception of gravity, dark matter, neutrino masses and a few other holdouts). Despite considerable experimental effort, until this result, particle physicists had not been able to demonstrate a measurable deviation from the predictions of the standard model within its domain of applicability. The Fermilab results tentatively look quite convincing, but past would-be deviations have been undone by human error.
OpenAI’s DALL-E 2 Draws Astounding Art
OpenAI’s newly released DALL-E 2 system is capable of generating artistic images from human provided text. Some of the produced images are really astounding. I’ve included a few below I liked, but there are many more online. DALL-E 2’s productions blur the line between human creativity and machine intelligence. Some of the pieces would likely pass muster as human produced artistic pieces even.
Google Trains 540 Billion Parameter Pathways Language Model
Google has trained a 540 billion parameter language model across 6144 TPUv3 chips (source). The resulting model achieves strong improvements over prior models across a broad range of natural language processing benchmarks. Impressively, the model is able to perform some compelling reasoning (see the joke explanation below!)
The Dangerous Loss of US Naval Advantage
Discussion
DALL-E 2 and PaLM continue the long march, from DeepBlue to AlphaGo to GPT-3, of large complex AI systems solving problems that were once thought to be only solvable by people. There is no doubt that DALL-E 2 or PaLM themselves are not “intelligent,” but each incremental advance draws into question what task, if any, can remain solely the province of human intelligence. Of course, systems like DALL-E 2 and PaLM remain far outside the reach of everyday programmers and researchers (even those at top tier research universities). There is no doubt these new models will lead to surprising new software products, but are we getting any closer to true intelligence? The jury will probably be out for decades yet.
Feedback and Comments
Please feel free to email me directly (bharath@deepforestsci.com) with your feedback and comments!
About
Deep Into the Forest is a newsletter by Deep Forest Sciences, Inc. We’re a deep tech R&D company building an AI-powered scientific discovery engine. Deep Forest Sciences leads the development of the open source DeepChem ecosystem. Partner with us to apply our foundational AI technologies to hard real-world problems. Get in touch with us at partnerships@deepforestsci.com!
Credits
Author: Bharath Ramsundar, Ph.D.
Editor: Sandya Subramanian, Ph.D.