Other industries
AI drift or is ChatGPT getting dumber?
ChatGPT getting less accurate is something users have been noticing for the past few weeks. While it’s possible that the chatbot is getting “dumber” over time, it could also be simply experiencing AI drift. Let’s look at what we know so far.
No, we haven't made GPT-4 dumber. Quite the opposite: we make each new version smarter than the previous one.
— Peter Welinder (@npew) July 13, 2023
Current hypothesis: When you use it more heavily, you start noticing issues you didn't see before.
Could ChatGPT be getting dumber?
OpenAI’s ChatGPT is a large language model (LLM), and as such, it’s trained on a massive dataset of text and code. It works by using a technique called deep learning, a type of machine learning that uses artificial neural networks to learn from data. This means that it’s constantly learning and evolving.
In a recent study, researchers from Stanford and Berkeley found that the response accuracy of both GPT-3.5 and GPT-4 appeared to become worse over time, highlighting the need to keep evaluating LLMs’ behavior.
The study has evaluated responses on a number of different tasks, including math problems, opinion surveys, and visual reasoning. It has then compared GPT-3.5 and GPT-4 responses from March and June.
“For example, GPT-4 (March 2023) was reasonable at identifying prime vs. composite numbers (84% accuracy) but GPT-4 (June 2023) was poor on these same questions (51% accuracy),” the study says.
Credit: How Is ChatGPT’s Behavior Changing over Time? by Lingjiao Chen, Matei Zaharia, James Zou
“We find that the performance and behavior of both GPT-3.5 and GPT-4 varied significantly across these two releases and that their performance on some tasks have gotten substantially worse over time, while they have improved on other problems,” according to the study.
These findings could potentially back the rumors of the LLM getting “dumber”. However, the study also suggests that GPT-3.5 and GPT-4 could be seeing improvements in other areas of performance.
Read also: How Google’s generative AI is shaping the future of content creation
Could ChatGPT be experiencing AI drift?
Artificial intelligence drift happens when sophisticated AI programs and entities, such as chatbots or LLMs, begin to deviate from their original parameters and instructions. This can lead to responses and activities not intended by their human creators.
AI drift is the change in data distribution over time, which can cause a machine-learning model to become less accurate. It can have a significant impact on the accuracy of machine learning models. For example, if a model is not updated to reflect the changes in data, it can start to make inaccurate predictions.
There are two main types of AI drift:
Gradual drift occurs slowly over time as data distribution gradually shifts. This can be caused by changes in customer behavior, the economy, or the law.
Sudden drift occurs suddenly when there is a major change in data distribution. This can be caused by factors such as a natural disaster, a terrorist attack, or a product recall.
Read also: How to enable and use Bard on Google Workspace
FAQs
Q1: Have users noticed a change in ChatGPT’s performance?
Yes, some users have been noticing for the past few weeks that ChatGPT seems to be getting less accurate. This has led to questions about whether the model is getting “dumber” over time.
Q2: Is there any evidence that ChatGPT’s performance has gotten worse?
A study by researchers from Stanford and Berkeley provided some evidence of this. They compared the performance of GPT-3.5 and GPT-4 between March and June and found that on certain tasks, performance had “gotten substantially worse.” For example, GPT-4’s accuracy in identifying prime numbers dropped from 84% to 51% in that period.
Q3: Does the study conclude that the models are only getting worse?
No, the study’s findings were mixed. It concluded that the performance and behavior of both models varied significantly. While they got worse on some tasks, they improved on others, highlighting a complex shift in their capabilities rather than a simple decline.
Q4: What is “AI drift”?
AI drift, also known as model drift, occurs when an AI model’s performance degrades over time because the real-world data it encounters in production changes and no longer matches the data it was trained on. This can cause the model to make less accurate predictions.
Q5: What are the two main types of AI drift described in the article?
The two main types are:
- Gradual drift: This occurs slowly over time as data distribution shifts gradually, for example, due to changes in customer behavior or economic trends.
- Sudden drift: This happens abruptly when there is a major and sudden change in data distribution, such as from a natural disaster or a large product recall.