No. A large language model (LLM) does not remember previous conversations on its own. Its learned knowledge is fixed after training, and each reply is built only from the text sent in that request. What feels like memory is the app re-sending your earlier messages, or saved notes, along with your new message.
Do LLMs remember anything at all?
An LLM keeps what it learned in training and nothing from your conversations. A large language model (LLM) is a model trained on huge amounts of text to predict the next piece of text; ChatGPT, Claude and Gemini are all built on LLMs. Training adjusts its weights, the billions of numbers that encode patterns in language. When training ends, the weights are frozen. Chatting with the model does not change them.
So “memory” means two different things, and most confusion comes from mixing them up:
| Kind of memory | What it is | Shared across users? | Changes when you chat? |
|---|---|---|---|
| Learned knowledge | Patterns stored in the weights during training | Yes, identical for everyone | No |
| Working memory | The text in front of the model for this one request | No, built per request | Yes, rebuilt every time |
The second kind is called the context window, and it is where all “remembering” actually happens.
How does a chatbot appear to remember what you said earlier?
The application re-sends the whole conversation with every new message. The model itself is stateless, which means it keeps no record between requests. Anthropic's API documentation says this directly: the Messages API is stateless, so you always send the full conversational history.
When you send your fifth message, the app builds one package containing:
- A hidden system prompt: instructions from the app's developer, such as tone and rules
- Your first four messages and the model's four replies
- Your fifth message
The model reads all of it as one long document and writes the next reply. Once the reply is returned, nothing is retained.
Here is the whole trick in a few lines. The “memory” is a Python list that the app owns, not something inside the model.
history = [] # lives in the app, not the model
def chat(user_text):
history.append({"role": "user", "content": user_text})
reply = call_model(messages=history) # the model sees ALL of history
history.append({"role": "assistant", "content": reply})
return replyDelete history and the next call knows nothing about the earlier ones. (call_model stands in for whichever provider's API you use.)
Why does the AI forget things in the same chat?
Because the conversation outgrew the context window, or the detail got lost in a long input. The context window is the maximum number of tokens a model can read in one request. A token is the unit of text a model reads: a whole word, part of a word or a punctuation mark. Everything has to fit: system prompt, history, attached files and the reply being written.
When a chat gets too long, the app has to cut something. The common strategies are dropping the oldest messages or replacing them with a summary. Either way, detail is lost, and the model is not told what is missing. This is why a rule you set an hour ago, such as “use rupees, not dollars”, can quietly stop being followed.
A bigger window does not fully fix it. Liu and colleagues (published in Transactions of the ACL, 2024) found that models often use information at the start or end of a long input more reliably than information buried in the middle. Fitting in the window is not the same as being used well.
For anything that must survive a long chat, restate it. Put standing instructions at the top of your prompt, and repeat the key constraint in the message where it matters.
How do the memory features in AI assistants work?
They save short notes about you in the app's own storage and paste the relevant ones into your prompt later. Many assistants now offer a memory setting. The model never holds the fact. Say the app saves “Arjun reports in INR lakhs”. Next week, when you ask for a summary, the app adds that line to the prompt behind the scenes, and only then does the model see it.
That has three practical consequences:
- Memory is text, so it can be wrong. If a saved note is stale or mistaken, it will be pasted into every future prompt.
- Memory uses tokens. Saved notes sit inside the same window as everything else.
- You can usually see and delete it. Whether a memory feature exists, and whether it is on by default, varies by product, so check the settings of the tool you use.
What are the ways to give an LLM memory?
There are five common approaches, and each solves a different problem.
| Approach | Where the information lives | Survives a new chat? | Best for | Main failure |
|---|---|---|---|---|
| Full history in the window | Inside the current prompt | No | Short, single-session tasks | Runs out of space; cost grows each turn |
| Summarised history | A running summary in the prompt | Only if the app saves it | Long sessions | The summary drops a detail that later matters |
| Retrieval (RAG) | An external database, searched per question | Yes | Large document sets, company knowledge | Fetches the wrong passage; poor chunking |
| Saved memory notes | App storage, inserted into prompts | Yes, if the feature is on | Preferences and recurring context | Stale or wrong notes; privacy exposure |
| Fine-tuning | Inside the model’s weights | Yes, for everyone | Style, format, narrow behaviours | Poor at storing facts; costly to update |
Retrieval-augmented generation (RAG) means searching an external store for relevant passages and placing them in the prompt before the model answers. Lewis and colleagues introduced the method in 2020. For when to retrieve and when to retrain, see RAG vs fine-tuning.
Researchers are also building agents that manage their own memory. Packer and colleagues (MemGPT, 2023) proposed treating the context window like a computer's fast memory, with the agent moving information in and out of slower storage as needed. Every one of these designs ends the same way: whatever is retrieved still has to be placed in the window before it can affect an answer.
Does talking to an LLM train it?
Not in real time. Your messages do not change the model's weights while you chat. Two other things are easy to confuse with it.
First, models can reproduce fragments of text they saw during training. That is learned knowledge, not memory of your conversation. Second, whether your chats are later used to improve future models depends on the provider's policy and your account settings. These vary by product and plan and change over time.
Read the data-use terms before you paste anything confidential. For company work, use the enterprise plan or API terms your organisation has approved.
When don't you need AI memory?
Often. In our experience training teams on generative AI, people add memory layers to systems that would be simpler, cheaper and more reliable without them.
| If your situation is… | Memory is… | Do this instead |
|---|---|---|
| Classifying tickets or extracting fields from invoices, one request at a time | ✗ Unnecessary | Send each item independently with a fixed prompt |
| An audit-sensitive task where the same input must give the same output | ✗ A risk | Send identical inputs every time; avoid hidden saved notes |
| A knowledge base of a few dozen pages | ✗ More machinery than needed | Put the pages straight into the prompt; skip the retrieval pipeline |
| Teaching the model facts that change monthly | ✗ The wrong tool (fine-tuning) | Use retrieval and update the source documents |
| A support assistant that must recall a customer’s open case across sessions | ✓ Genuinely useful | Store the case summary in your database and insert it per request |
Start stateless. Every piece of memory is data you must secure, expire and delete on request, and a way for yesterday's mistake to reach today's answer. Add the simplest memory that fixes a failure you have actually observed, and nothing more.
A worked example: a support bot over twelve turns
The numbers below are illustrative arithmetic with assumed sizes, not measured data. Assume a support chat where the system prompt is 500 tokens, each user message is about 60 tokens, and each model reply is about 250 tokens.
| Turn | Input tokens the model reads | New text added this turn |
|---|---|---|
| 1 | 560 | 310 |
| 3 | 1,180 | 310 |
| 6 | 2,110 | 310 |
| 9 | 3,040 | 310 |
| 12 | 3,970 | 310 |
| Across all 12 turns | ~27,180 | ~3,720 |
The conversation contains about 3,720 tokens of actual dialogue (12 × 310). But the model reads about 27,180 input tokens in total, because earlier turns are re-sent each time. That is roughly seven times the text anyone wrote. Providers offer techniques such as prompt caching to reduce the cost of re-reading unchanged text, but the principle holds: the model conditions on the whole history on every turn.
The practical fix is to summarise. Once a chat passes a few thousand tokens, ask the model for a short summary of decisions and constraints, then start a new chat with only that summary. You pay for a few hundred tokens instead of thousands, and you can check the summary for errors before the model builds on it.
How can you use AI memory more safely and effectively?
Own the memory yourself: keep it short, keep it visible, and keep it checkable.
- Find the memory and data settings in the assistant you use, and decide what you are comfortable having saved.
- Open your saved memories now and delete anything stale, wrong or confidential.
- Keep a project brief: goals, constraints and decisions so far. Paste it at the start of each new chat instead of letting one chat run for hours.
- Restate critical instructions in the message where they matter, not only at the start of the chat.
- If you build with an API, log one full request and read exactly what the model receives.
- Count the tokens in a typical conversation from your own use case. If it sits well inside the window, send full history and add nothing else.
- Test long inputs with the answer placed in the middle, since that is where models are weakest.
Here is a project brief you can adapt and paste at the start of a fresh chat:
DECISIONS SO FAR: All figures in INR lakhs. Compare against the same month last year.
CONSTRAINTS: No external data. Flag any number you are unsure about.
TASK: Summarise the attached March report in five bullet points.
Frequently asked questions
Does ChatGPT, Claude or Gemini remember me between chats?
By default, the underlying model does not. Some assistants offer an optional memory feature that saves notes about you and adds them to future prompts. Check the settings of your specific product to see whether it is available and whether it is switched on.
Why did the AI forget something from earlier in the same chat?
Usually the conversation grew past the context window and the app trimmed or summarised older messages. It can also happen when the detail sits in the middle of a long input, where models tend to use information less reliably.
Is a bigger context window the same as better memory?
No. A bigger window lets you fit more text, but it does not make the model use that text well, and it raises cost and latency because more is re-read each turn. It extends working memory within one chat; it does not create memory that lasts across separate chats.
What is the difference between AI memory and RAG?
Memory stores facts about you or your work and inserts them into prompts. RAG searches a document collection and inserts the passages that match the current question. Both place text into the context window; they differ in what is stored and how it is chosen.
Does talking to an LLM train it?
Not in real time. Your messages do not change the model’s weights while you chat. Whether conversations are later used to improve future models depends on the provider’s policy and your settings, so read the data terms.
Should I use fine-tuning to make a model remember my company’s facts?
Usually no. Fine-tuning is better for style, format and narrow behaviours. For facts that change, retrieval is easier to update and easier to check against the source.
Sources
- Anthropic. “Messages API examples” (the API is stateless; send the full conversation history). Anthropic documentation
- Liu, N. F. et al. “Lost in the Middle: How Language Models Use Long Contexts.” Transactions of the ACL, 2024. Read the paper
- Lewis, P. et al. “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.” 2020. Read the paper
- Packer, C. et al. “MemGPT: Towards LLMs as Operating Systems.” 2023. Read the paper
- Current context-window sizes: Anthropic, OpenAI models, and Google Gemini.