Have you ever had someone space out right in the middle of a conversation? Happens all the time right! Well, in the world of Artificial Intelligence (AI), the same thing can happen if the AI doesn’t have enough "memory" to remember everything you're saying. This idea of how much an AI can keep in its "mind" at once is called context size, and it's super important for making sure the AI doesn’t lose track of the conversation and gives you appropriate responses.
Let’s break this down so it makes sense.
What Is Context Size?
When we talk to an AI, like ChatGPT, it can only "remember" a certain amount of what we say. You simply can't read a whole book to Siri or Alexa and expect it to understand the whole story. They have limited "memory" and is measured in something called tokens, which are basically little chunks of text. A token could be a word or even part of a word. The AI can only handle a certain number of tokens at one time, kind of like how you can only carry so many books before they start falling out of your arms.
Once the AI’s memory is full, it starts to forget the first things you said to make room for the new information. If this happens too often, the AI might not make sense anymore because it forgets what you were talking about earlier.
For ChatGPT depending on what version you use can handle 4096-128,000 tokens which is about 3000 to 96,000 words or 4 to 150 pages.
Why Does Short-Term Memory and Coherence Matter?
Coherence is a fancy word for how well a conversation flows and makes sense. Imagine telling your friend a story, but halfway through, they forget the beginning. If you say, "And then the cat jumped off the table," and they respond, "What cat?" it’s because they lost track of your story. The same thing happens with AI when its memory (or context size) gets overloaded.
Here’s how a small memory can mess things up:
Repeating Stuff: The AI might repeat itself because it forgets that it already said something.
Giving Weird Answers: It could give answers that don’t fit with what you were talking about, because it lost the original idea.
Messing Up Stories: If you’re telling the AI a story it might lose track of what happened before, making the whole conversation confusing. Sometimes it can also contradict itself by giving a completely different answer than before.
Forgetting Important Details: It might forget key facts that were important to the conversation, like your name and relationship to the subject matter.
Feeling Less Personal: If you’re having a chat with a personalized AI that’s supposed to remember things about you, like your favorite color or hobby, it might suddenly forget because it can’t keep all that information in its short-term memory.
How Do We Make AI Better at Remembering?
The good news is there are ways to help AI keep conversations more on track and avoid losing its place:
Performance Impact: A simple solution is to buy millions of dollars in GPU hardware to increase it's memory. But average people can't afford that.
Focus on What Matters Most: We can teach the AI to remember the most important parts of a conversation and let go of the things that aren’t as important. We do the same thing subconsciencly because we don't remember word for word every conversation.
Summarize Information: Pre processing the information is sometimes necessary, summarizing what you’ve talked about, can shorten a long drawn out story into a few sentences.
Organizing Post It Notes: Now that things are summarized, we store these post it memos into a relational vector database for relatively cheap as a long term memory. Then the AI can be recalled if needed and fit into the context of the conversation.
Why Does This Matter for Personalized AI?
AI specifically designed for you—the AI needs to be able to remember things about you over time. This makes the interaction feel more natural and less robotic. When the context size is too small, the AI might forget things like your favorite music or what you talked about last time, which can make the conversation less fun or meaningful.
Conclusion
AI’s "memory" is really important for making sure it can hold good conversations and without forgetting what you said. By understanding the limits of how much it can remember at once (its context size), we can make sure AI stays on topic, and doesn’t lose track of important details. This helps make the experience smoother, more natural, and a lot more fun to interact with!
Pat Bhakta
Founder
