AI: Inside Context Windows - How AI Manages Memory

tl;dr

• Context windows are measured in tokens: Roughly equivalent to words, these units determine AI's working memory capacity
• Memory isn't like human memory: AI has no permanent recall between conversations, only what's in the active context window
• Different types of memory serve different purposes: From active conversation memory to persistent project context
• Size comes with trade-offs: Larger context windows mean higher costs, slower processing, and potential quality degradation
• Innovation is accelerating: New techniques are expanding context from hundreds of thousands to millions of tokens

Picture this: You're deep in conversation with an AI about your company's software architecture. Twenty minutes later, you reference something from the beginning of the chat, and it doesn't quite remember. The response is vague, missing key details you discussed. Welcome to the peculiar world of AI memory—where context is everything and nothing is permanent.

Context Windows: The AI's Working Memory

A context window is fundamentally the amount of information an AI model can "hold in mind" during a single interaction. Think of it as the AI's RAM—its working memory for active processing.

AI measures this capacity in tokens. While not exactly words, tokens are close enough for practical purposes:

• A token is roughly one word in English
• "Hello" = 1 token
• "Artificial Intelligence" = 2-3 tokens
• Common words are usually 1 token; longer or unusual words might be 2-3
• Punctuation and spaces also consume tokens

This means a model with a 100k token context window can process approximately 100,000 words—about 200 pages of text.

But here's the critical part to understand: context is additive. Every time you send a prompt, those tokens get added to the running total. Every time the AI responds, those tokens also get added. So if you start with a 100-token question, get a 500-token response, then ask a short 50-token follow-up, you've still already consumed 750 tokens—and that's before counting system instructions which are also added to the context. This accumulation continues throughout your conversation until you hit the context limit[1].

Memory Architecture: The Technical Details

The evolution of AI memory has been fascinating. When ChatGPT first launched in November 2022, it actually did maintain some form of cross-conversation memory, remembering user preferences and context across chats. However, this feature was removed and later reintroduced in different forms due to privacy and technical concerns[2].

Today, most AI systems operate with three distinct memory types:

1. Training Memory (Foundational)
This is all the knowledge from the model's training data—facts about the world, language patterns, technical information. It's what enables the AI to understand concepts and generate coherent responses. This knowledge is "frozen" at a specific point in time (the knowledge cutoff) and it is not updated through conversations. When you provide information preferences through your prompts, the model is not updated.

2. Instruction Memory (Persistent)
These are system and behavioral guidelines provided by the platform (ChatGPT, Claude, Gemini, etc.) hosting the AI. They are not built into the model itself, but injected by the platform at the start of each conversation. They remain consistent across all your chats, shaping how the model responds—its tone, capabilities, and limitations. Think of these as configuration settings that the platform applies to customize the base model's behavior. Unlike context memory, these persist across sessions, but they're not conversational memory—they don't help the AI remember you or your previous discussions.

3. Context Window Memory (Active)
This is everything currently in your conversation—all the messages, instructions, and information exchanged during this session. It's immediately accessible and influences all responses. However, there's a critical limitation: when the conversation reaches its token limit, many LLMs will automatically compact or truncate the oldest parts of the conversation to make room for new information. This compression can cause the model to "forget" specific details from earlier in the chat. And when you start a new conversation? Nothing carries over. Each new chat begins with a completely blank slate[1].

This three tier architecture has profound implications. When you spend an hour explaining your project's architecture to an AI, that understanding exists only for that session. Close the window, and it's gone forever.

Project Context: A "Fourth" Type of Memory

Many modern AI platforms have introduced a fourth type of memory: Project Context or Custom Instructions. This feature, available in ChatGPT, Claude, Perplexity, and others, allows users to save information that persists across all conversations[3].

Unlike the three fundamental memory types, project context is:

• User-controlled: You explicitly choose what information to save
• Cross-conversation: Applied to every new chat you start in the project
• Limited in scope: Usually restricted to a few thousand tokens
• Platform-specific: Doesn't transfer between different AI services

For example, you might save:

• Your company's style guidelines
• Project-specific terminology
• Your coding preferences and tech stack
• Personal preferences for how the AI should respond

This feature bridges the gap between completely ephemeral conversations and true persistent memory, but it's still limited—it can't remember specific conversations and does not dynamically update based on your interactions.

Technical Constraints: Why Context Size Matters

The constraints on context windows aren't arbitrary—they're rooted in computational reality. When a LLM processes larger contexts, it requires exponentially more computational resources.

The computational complexity of processing context scales quadratically with length. This means that doubling the context length quadruples the processing requirements. To put this in perspective, a 1 million-token context (which is 10x larger than a 100,000-token context) requires 100x more computational power. This exponential scaling creates practical limits on how large context windows can grow[4].

Memory requirements present another significant constraint. GPU memory limitations create hard caps on context size, and larger models need even more memory per token processed. Current hardware architecture simply can't store and process unlimited amounts of context simultaneously[5].

Finally, the cost implications are substantial. Longer contexts mean higher API costs since providers typically charge per token. The increased processing time also affects real-time applications, making them less responsive. Infrastructure costs scale with context requirements, making it expensive to offer extremely large context windows at scale.

Context Capacity: Current Model Capabilities

These numbers represent massive improvements from just two years ago when 4,000-8,000 tokens was standard:

• Open-source models: Typically 32,000-128,000 tokens
• GPT-4.1: 128,000 tokens
• Claude Opus 4: 200,000 tokens (approximately 400 pages)
• Claude Sonnet 4: 200,000 tokens
• Gemini 1.5 Pro: Up to 2 million tokens (research preview)

Business Optimization: Insider Tips and Best Practices

Understanding these technical details unlocks powerful optimization strategies:

Strategic Model Selection
Don't use a sledgehammer for every nail. Here's how insiders match models to tasks[6]:

• Quick queries: Use GPT-3.5 or Claude Instant (faster, cheaper, adequate for simple tasks)
• Research tasks: Perplexity excels with real-time web access and source citations
• Long documents: Gemini 1.5 Pro can handle entire books or codebases
• Creative writing: Claude Opus 4 maintains consistency over long narratives
• Coding: Claude for complex logic, GitHub Copilot for autocomplete

Context Management Tactics
Master these techniques to maximize AI performance while minimizing token usage and maintaining conversation coherence:

• Front-load critical information: Models pay most attention to the beginning and end of context
• Use the "context sandwich": Place key info at start, details in middle, summary at end
• Implement progressive disclosure: Start conversations with essential context, add details as needed
• Create context checkpoints: In long conversations, periodically summarize key decisions

Cost Optimization Strategies
Enterprise API costs can spiral quickly. Smart organizations use tiered approaches:

• Route 80% of simple queries to cheaper models (saves 60-70% on costs)
• Reserve premium models for complex analysis or customer-facing content
• Implement caching for frequently asked questions
• Use context compression techniques before hitting limits

What's Next?

Now that you understand how context works technically, our next article explores practical business applications—from customer service to document analysis—showing how to leverage these technical capabilities for real-world value.

Final Thoughts

The technical architecture of AI context windows reveals both the impressive power of LLMs and the fundamental constraints of current systems. As context windows expand, memory systems improve, and computing power increases, we will begin to see AI systems that can maintain coherent understanding across increasingly complex interactions. Research labs are exploring infinite context approaches, selective memory systems that mimic human recall, and cross-session continuity that maintains context without compromising privacy. These innovations have the potential to eliminate many current limitations.

But even with today's constraints, knowing how these systems work enables you to use them far more effectively. The gap between average and excellent AI results often comes down to understanding the underlying system, and managing context is a critical tool you can now leverage.

Continue exploring our AI Context series: Understanding Context | Dos and Don'ts | Technology Basics | Business Applications | Industry Trends

References

1. What is a context window? – IBM
2. Memory in ChatGPT - Remembering what you chat about – OpenAI Help Center
3. Custom instructions for ChatGPT – OpenAI Help Center
4. Efficient Memory Management for Large Language Model Serving with PagedAttention – arXiv
5. A Gentle Introduction to 8-bit Matrix Multiplication – Hugging Face Blog
6. What We Learned from a Year of Building with LLMs – O'Reilly