Language models become more complicated as they become more difficult to design and run. Some businesses have been barred from operating.
Glean provides search features for applications such as Gmail, Slack, and Salesforce. New AI algorithms for parsing language, according to Qi, will help Glean's clients find the right file or chat much faster.
However, training a cutting-edge AI algorithm costs millions of dollars. As a result, Glean employs smaller, less capable AI models capable of extracting less meaning from text. “It's difficult for smaller businesses with fewer finances to achieve the same degree of results” as Google or Amazon, according to Qi. He claims that the most powerful AI models are "out of the question."
In the last decade, AI has created fascinating breakthroughs—programs that can beat humans at sophisticated games, guide cars through city streets under specific conditions, reply to spoken commands, and write intelligible text from a short prompt. Writing, in particular, is reliant on recent improvements in computers' ability to analyze and modify text.
Those advancements are essentially the result of providing more text as examples for the algorithms to learn from, as well as more chips with which to digest it. And that is costly.
Consider OpenAI's GPT-3 language model, which is a massive, mathematically constructed neural network that was fed reams of web content. GPT-3 can identify statistical patterns that predict which words should be followed by others with remarkable consistency. GPT-3 outperforms earlier AI models on tasks including answering questions, summarizing text, and correcting grammatical errors right out of the box. It is 1,000 times more capable than its predecessor, GPT-2, in one measure. However, some estimates put the cost of training GPT-3 at around $5 million.
"If GPT-3 was widely available and inexpensive, it would completely transform our search engine," Qi says. "That would be extremely effective." The rising expense of sophisticated AI training is also a concern for established businesses trying to expand their AI capabilities.
Dan McCreary is the leader of a team at Optum, a health IT business, that utilizes language models to analyze conversation transcripts in order to identify higher-risk patients and provide referral recommendations. Even training a language model a thousandth the scale of GPT-3, he claims, can swiftly deplete the team's budget.
Models must be taught for specific jobs and can cost more than $50,000 to rent machines and programs from cloud computing businesses. According to McCreary, cloud computing providers have little incentive to reduce costs. "We can't trust cloud providers to seek to reduce the price of generating AI models for us," he says. He's considering investing in customized chips designed to accelerate AI training.
Many academic labs and entrepreneurs were able to download and exploit the latest ideas and approaches, which explains why AI has evolved so quickly recently. Academic labs, for example, developed algorithms that led to advancements in image processing utilizing off-the-shelf gear and publicly available data sets.
Progress in AI, on the other hand, has become increasingly obvious as the underlying computer power has grown exponentially. Of course, big businesses have always had an advantage in terms of budget, scale, and reach. In businesses like drug research, enormous quantities of computer power are a given.
Some are now seeking to ramp things up even more. Microsoft announced last week that it had collaborated with Nvidia to create a language model that was more than twice the size of GPT-3. Chinese researchers claim to have created a language model four times larger than that.
"The cost of training AI is unquestionably increasing," says David Kanter, executive director of MLCommons, a group that monitors the performance of AI processors. Many parts of the computer industry, he argues, are embracing the idea that larger models might unlock important new capabilities. It could explain why Tesla is developing its own semiconductors solely for the purpose of training AI models for self-driving cars.
Some fear that the increased expense of accessing the most cutting-edge technology would limit innovation by reserving it for the largest corporations and those that lease their equipment. "I believe it inhibits creativity," says Chris Manning, a Stanford professor who specializes in artificial intelligence and language. "When there are only a few sites where people may play with the innards of these models of that magnitude, the amount of creative discovery that happens needs to be significantly reduced."
Manning claims that his group had enough computing capacity ten years ago to investigate any project. "A single PhD student working hard may provide cutting-edge work," he says. "That window appears to have closed now." Simultaneously, increased costs are driving people to seek out more efficient methods of training AI systems. Hundreds of firms are developing specialized computer chips for AI training and execution.
Glean's Qi and Optum's McCreary are both speaking with Mosaic ML, an MIT spinoff that is developing software methods to improve the efficiency of machine-learning training.
The startup is based on a technique invented by MIT professor Michael Carbin and one of his students, which entails "pruning" a neural network to remove inefficiencies and generate a much smaller network capable of comparable performance. Early results, according to Frankle, imply that the amount of computer power required to train something like GPT-3 might be cut in half, lowering development costs.
There are other approaches for enhancing the performance of neural network training, according to Carbin. Mosaic ML intends to open-source much of its technology, but it will also provide consultancy services to businesses looking to reduce the cost of AI adoption.
According to Carbin, one potential option is a tool that measures the trade-offs between alternative procedures in terms of accuracy, time, and cost. He claims that "nobody really knows how to bring all of these strategies together."
Mosaic ML's technology, according to MLCommons' Kanter, may help well-heeled organizations advance their models, but it might also help democratize AI for companies without deep AI experience. "If you can reduce the cost and provide access to expertise to those organizations, that will encourage adoption," he argues.
0 Comments