A problem with LLMs: LLM context space is limited.

A similar problem has been solved before in traditional operating systems.

The OS problem:

Physical memory (RAM) is limited.

A solution is virtual memory.

Virtual memory compensates for the lack of RAM.

Instead of just RAM, now consider RAM and the hard drive.

As long as RAM is not full, we can continue to add more data.

When RAM becomes full, we can:
1. find the least important data in RAM
2. move that data to the hard drive for storage and later access

Now there is room in the RAM for more data.

Let's review:

Physical memory (RAM) is limited.
LLM context is limited.

A solution is virtual memory.
A possible solution is virtual context.

Virtual memory compensates for the lack of RAM.
Virtual context compensates for the lack of context space.

Instead of just RAM, now consider RAM and the hard drive.
Instead of just LLM context, consider in-context (same as current LLM context) and out-of-context spaces.

As long as RAM is not full, we can continue to add more data.
As long as the in-context space has is not full, we can continue to add to the in-context space.

When RAM becomes full, we can:
1. find the least important data in RAM
2. move that data to the hard drive for storage and later access

When the in-context space becomes full, we can:
1. find the least important data the in-context space
2. move that data to out-of-context space for storage and later access

 Now there is room in the RAM for more data.
Now there is room in the in-context space for more data.

The LLM problem:

LLM context is limited.

A possible solution is virtual context.

Virtual context compensates for the lack of context space.

Instead of just LLM context, consider in-context (same as current LLM context) and out-of-context spaces.

As long as the in-context space has is not full, we can continue to add to the in-context space.

When the in-context space becomes full, we can:
1. find the least important data the in-context space
2. move that data to out-of-context space for storage and later access

 Now there is room in the in-context space for more data.



From:
Packer, C., Fang, V., Patil, S. G., Lin, K., Wooders, S., & Gonzalez, J. E. MemGPT: Towards LLMs as Operating Systems. 2023.