Building apps with GPT-3? Here’s what devs need to know about cost and performance – TNW

Last week, OpenAI removed the waitlist for the application programming interface to GPT-3, its flagship language model. Now, any developer who meets the conditions for using the OpenAI API can apply and start integrating GPT-3 into their applications.

Since the beta release of GPT-3, developers have built hundreds of applications on top of the language model. But building successful GPT-3 products presents unique challenges. You must find a way to leverage the power of OpenAIs advanced deep learning models to provide the best value to your users while keeping your operations scalable and cost-efficient.

Fortunately, OpenAI provides a variety of options that can help you make the best use of your money when using GPT-3. Heres what the people who have been developing applications with GPT-3 have to say about best practices.

OpenAI offers four versions of GPT-3: Ada, Babbage, Curie, and Davinci. Ada is the fastest, least expensive, and lowest-performing model. Davinci is the slowest, most expensive, and highest performing. Babbage and Curie are in-between the two extremes.

OpenAIs website doesnt provide architectural details on each of the models, but the original GPT-3 paper includes a list of different versions of the language model. The main difference between the models is the number of parameters and layers, going from 12 layers and 125 million parameters to 96 layers and 175 billion parameters. Adding layers and parameters improves the models learning capacity but also increases the processing time and costs.

OpenAI calculates the pricing of its models based on tokens. According to OpenAI, one token generally corresponds to ~4 characters of text for common English text. This translates to roughly of a word (so 100 tokens ~= 75 words).

Heres an example from OpenAIs Tokenizer tool:

In general, if you use good English (avoid jargon, use simple words with few syllables, etc.), youll get better token-to-word ratios. In the example below, aside from GPT-3, every other word counts as one token.

One of the benefits of GPT-3 is its few-shot learning capabilities. If youre not satisfied with the models response to a prompt, you can guide it by giving it a longer prompt that includes correct examples. These examples will work like real-time training and improve GPT-3s results without the need to readjust its parameters.

It is worth noting that OpenAI charges you for the total tokens in your input prompt plus the output tokens GPT-3 returns. Therefore, long prompts with few-shot learning examples will increase the cost of using GPT-3.

With a 75x cost difference between the cheapest and most expensive GPT-3 models, it is important to know which option best suits your application.

Matt Shumer, the co-founder and CEO of OthersideAI, has used GPT-3 to develop AI-powered writing tools. HyperWrite, OthersideAIs main product, uses GPT-3 for text generation, autocomplete, and rephrasing.

When choosing between different GPT-3 models, Shumer starts by considering the complexity of the intended use case, he told TechTalks.

If its something simple, like binary classification, I might start with Ada or Babbage. If its something very complex, like conditional generation where high-quality output and reliability is necessary, I start with Davinci, he said.

When unsure of complexity, Shumer starts by trying the biggest model, Davinci. Then, he works his way down toward the smaller models.

When I get it working with Davinci, I try to modify the prompt to use Curie. This typically means adding more examples, refining the structure, or both. If it works on Curie, I move to Babbage, then Ada, he said.

For some applications, he uses a multi-step system that includes a mix of different models.

For example, if its a generative task that requires some classification as a precursor step, I might use Babbage for the classification, then Curie or Davinci for the generative step, he said. After using it for a while, you get a feel for what might be useful for different use cases.

Paul Bellow, author and developer of LitRPG Adventures, used Davinci for his GPT-3-powered RPG content generator.

I wanted to generate the highest quality output possiblefor later fine-tuning, Bellow told TechTalks. Davinci is the slowest and most expensive, but the tradeoff is higher quality output which was important to me at this stage of development. Ive spent a premium, but I now have over 10,000 generations that I can use for future fine-tuning. Datasets have value. (More on fine-tuning later.)

Bellow says that the best way to find out if another model is going to work for a task is to run some tests on Playground, a tool you can use to directly try prompts on different GPT-3 models (note that OpenAI bills you for using Playground).

A lot of the time, a well-thought-out prompt can get good content out of the Curie model. It all just depends on the use-case, Bellow said.

When choosing a model for your application, youll have to weigh the balance between the cost and value. Choosing a high-performing model might provide better quality output, but the improved results might not justify the price difference.

You have to build a business model around your product that supports the engines youre using, Shumer said. If you want high-quality outputs for your users, itll be worth it to use Davinciyou can pass off the costs to your users. If youre looking to build a large-scale free product, and your users are okay with mediocre results, you can use a smaller engine. It all depends on your product goals.

OthersideAI has developed a solution that uses a mix of different GPT-3 models to enable different use cases, Shumer said. Paid users enjoy the power of large GPT-3 models, while free-tier users get access to the smaller models.

For LitRPG Adventures, quality is prime, which is why Bellow initially stuck to the Davinci model. He used the base Davinci model with one- or two-shot prompts, which increased the costs but made sure GPT-3 provided quality output.

OpenAI API Davinci model is a bit expensive at this time, but I see the cost going down eventually, he said. What provides flexibility right now is the ability to fine-tune the Curie and lower models, or Davinci with permission. This will bring my costs per generation down quite a bit while hopefully maintaining high quality.

He has been able to develop a business model that maintains a profit margin while using Davinci.

While not a huge money-maker, the LitRPG Adventures project is paying for itself and just about ready to scale up, he said.

OpenAIs scientists initially introduced GPT-3 as a task-agnostic language model. According to their initial tests, GPT-3 rivaled state-of-the-art models on specific tasks without the need for further training. But they also mentioned fine-tuning as a promising direction of future work.

In the months that followed the beta release of GPT-3, OpenAI and Microsoft fine-tuned the model for a number of different tasks, including database query and source-code generation.

Like other deep learning architectures, fine-tuning has several benefits for GPT-3. OpenAI API allows customers to create fine-tuned versions of its GPT-3 for a premium. You can create your own training dataset, upload it to OpenAIs servers, and use it to create a finetuned model of GPT-3. OpenAI will host your model and make it available to you through its API.

Fine-tuning will enable you to tackle problems that are impossible to solve with the basic models.

The vanilla models are highly capable and are usable for many tasks. However, some tasks (i.e., multi-step generation) are too complex for a vanilla model, even Davinci, to complete with high accuracy, Shumer said. In cases like this, you have two options: 1) create a prompt chain that feeds outputs from one prompt into another prompt, or 2) fine-tune a model. I typically first try to create a prompt chain, and if that doesnt work, I then move to fine-tuning.

If done properly, fine-tuning can also reduce the costs of using GPT-3. If youll be using GPT-3 for a specific application, a fine-tuned small model can produce results that are as good as those provided by a large vanilla model. Fine-tuned models also reduce the size of prompts, which further slashes your token usage.

One other case where I tend to fine-tune is when I can get something working with a vanilla model, but the prompt ends up being so long that it is costly to serve to users. In cases like these, I fine-tune, as it actually can reduce the overall serving costs, Shumer said.

But fine-tuning isnt without challenges. Without a quality training dataset, finetuning can have adverse effects.

Clean your dataset as much as you can. Garbage in, garbage out is one of my big mantras now when it comes to prompt engineering, Bellow said.

If you manage to gather a sizeable dataset of quality examples, however, fine-tuning can do wonders. After starting LitRPG with the Davinci model, Bellow gathered and cleaned a dataset of around 4,000 samples in a 7-megabyte JSON file. While he is still experimenting, the initial results show that he can move from Davinci to Curie without a noticeable change in quality, which reduces the costs of GPT-3 queries by 90 percent.

Another consideration is the time it takes to fine-tune GPT-3, which grows with the size of the model and the training dataset.

It can take as little as five minutes to fine-tune a smaller model on a few hundred examples, Shumer said. Ive also seen cases where it takes upwards of five hours to train a larger model on thousands of examples.

Theres also an inverse correlation between the size of the model and the amount of data you need to fine-tune GPT-3, according to Shumers experiments. Larger models require less data for fine-tuning.

For many tasks, you can think of increasing base model size as a way to reduce how much data youll need to fine-tune a quality model, Shumer said. A Curie fine-tuned on 100 examples may have similar results to a Babbage fine-tuned on 2,000 examples. The larger models can do remarkable things with very little data.

OpenAI received a lot of criticism for deciding not to release GPT-3 as an open-source model. Subsequently, other developers released GPT-3 alternatives and made them available to the public. One very popular project is GPT-J by EleutherAI. Like other open-source projects, GPT-J requires technical effort on the part of application developers to set up and run. It also doesnt benefit from the ease of use and scalability that comes with hosting and fine-tuning your models on Microsofts Azure cloud.

But open-source models are nonetheless useful and are worth considering if you have the in-house talent to set them up and they meet your applications requirements.

GPT-J isnt the same as full-scale GPT-3but it is useful if you know how to work with it. Its exponentially harder to get a complex prompt working on GPT-J, as compared with Davinci, but it is possible for most use-cases, Shumer said. You wont get the same super high-quality output, but you can likely get to something passable with some time and effort. Plus, these models can be cheaper to run, which is a big plus, considering the cost of Davinci. We have successfully used models like these at Otherside.

In my experience, they operate at about the level of the Curie model from OpenAI, Bellow said. Ive also been looking into Cohere AI, but theyre not giving details on the size of their model, so I imagine its around the same as GPT-J, et al. I do think (hope) that there will be even more options soon from other players. Competition between suppliers is good for consumers like me.

This article was originally published by Ben Dickson onTechTalks, a publication that examines trends in technology, how they affect the way we live and do business, and the problems they solve. But we also discuss the evil side of technology, the darker implications of new tech, and what we need to look out for. You can read the original articlehere.

See the original post here:
Building apps with GPT-3? Here's what devs need to know about cost and performance - TNW