Former Google AI Researcher Believes ChatGPT Could Have Achieved Success Sooner

Jakob Uszkoreit, a pivotal figure in the development of the Transformer architecture, shared his insights about his contributions to artificial intelligence (AI), the early work on large language models (LLMs) at Google, and his current venture in biological computing, Inceptive, during an interview at TED AI 2024 and with Ars Technica. Uszkoreit was one of the key researchers who helped create the Transformer model, which revolutionized how AI systems process and generate language, audio, and images.

In the groundbreaking paper “Attention is All You Need,” Uszkoreit made a significant contribution by proposing the use of the attention mechanism, specifically self-attention, as a replacement for the recurrent neural networks (RNNs) that dominated sequence transduction models at the time. The idea behind this innovation was to create a more efficient and effective model for tasks such as machine translation, which required processing sequences of data, such as sentences, and generating corresponding outputs. By eliminating the need for recurrence, which was computationally expensive and slow, and introducing a mechanism that could handle long-range dependencies within data more effectively, the Transformer model proved to be a game-changer. Today, variations of this architecture power some of the most advanced AI models in the world, including GPT-4, which is the basis of tools like ChatGPT, and other applications such as audio synthesis, image generation, and video synthesis.

Uszkoreit reflected on the development of the Transformer model and his team’s high hopes for the technology, noting that while they anticipated the potential of attention models, they did not foresee the rapid and widespread impact that the technology would have, particularly in products like ChatGPT. Although the paper was a significant step forward, Uszkoreit and his colleagues at Google had been working on attention models for years prior to its publication. The results they observed at the time were impressive, but the team was cautious about pushing the technology out into the world, partly due to a culture of conservatism around product development at Google. They were unsure whether these advancements would translate into commercially viable products. Despite these uncertainties, the team was excited about the potential and envisioned how the technology could push the boundaries of what AI could do.

When ChatGPT, built on the advancements of earlier versions of GPT, gained massive public attention, Uszkoreit was taken aback by the speed at which the technology was adopted. He wasn’t surprised by the capabilities of the models themselves, but rather by how quickly people were able to creatively use the technology and the diverse range of applications that emerged. He expressed that the success of ChatGPT was a breakthrough, not in the technology itself, but in how people realized the immense utility of the technology at such a high level of capability. He noted that this was a reminder of how important it is to experiment and take risks when developing new technologies, as some ideas may not seem promising at first, but can lead to breakthroughs when tested in the real world.

Uszkoreit also acknowledged that Google, at the time, did not have the appetite for risk-taking that might have allowed them to move more quickly with innovations like ChatGPT. However, he pointed to the development of Google Translate as an example of how innovation can evolve. Initially, Google Translate was seen as a joke due to its poor quality, but through continuous experimentation and iteration, it eventually became a widely used and indispensable tool. This process of trial and error, he argued, was central to developing truly valuable products, even when the initial results were far from perfect.

After leaving Google in 2021, Uszkoreit co-founded Inceptive, a company that applies deep learning techniques to biochemistry. The company’s goal is to develop “biological software,” a concept where AI-driven compilers are used to translate desired biological behaviors into RNA sequences. These sequences, when introduced into biological systems, can prompt cells and organisms to perform specific functions. Uszkoreit compared this idea to computer software, where a compiler translates human-readable code into machine instructions. In the case of biological software, however, the AI compiler would translate a behavior specification into a molecular sequence that could then be used in biological systems to carry out specific functions. This could lead to entirely new approaches to medicine, including the design of drugs and therapies that are precisely tailored to specific biological needs.

He further explained that a simple example of biological software at work is the mRNA COVID vaccines, which instruct cells to produce a protein to trigger an immune response. However, Uszkoreit’s vision goes beyond simple vaccines. He imagines molecules that could exhibit far more complex behaviors, similar to how RNA viruses operate within organisms. These viruses are essentially biological software that directs complex behaviors, such as spreading within an organism or even across the globe. If researchers can design molecules that exhibit even a small fraction of such complexity but for beneficial purposes, it could revolutionize medicine, offering more precise and effective treatments.

Uszkoreit was also mindful of the safety concerns surrounding the development of biological software. He pointed out that medicine has long been governed by a system of safeguards, including rigorous clinical trials, to prevent unintended consequences. This framework will continue to be essential as scientists work to develop and test biological software. The first experiments will focus on small-scale systems, such as individual cells, and will adhere to the same safety protocols that have been established for traditional medical practices. By following these established guidelines, Uszkoreit believes that researchers can safely explore the potential of AI-powered biological computing without risking harm.

Through his work on the Transformer model and his current focus on biological software, Uszkoreit is at the forefront of two groundbreaking fields—artificial intelligence and biochemistry. His work demonstrates the expanding role of AI in diverse industries, from tech to medicine, and highlights the transformative potential of AI when combined with other fields of science. As Inceptive continues to develop innovative solutions for designing medicines, Uszkoreit remains focused on creating technologies that not only push the boundaries of science but also have the potential to improve lives in meaningful and tangible ways. His journey reflects the ongoing evolution of AI, from its role in language models to its potential to revolutionize healthcare and other sectors.