Context Language Models

Context Language Models (CLMs) are introduced as language models that manage their own context by treating it as a file for unrestricted updates. This approach allows CLMs to learn essential context and extends to multi-agent systems. CLMs outperform state-of-the-art context management strategies, achieving 11.4% higher accuracy with 21.5% fewer FLOPs on BrowseComp-Plus and 5% higher scores with 59% fewer FLOPs on 12-hour EdgeBench. They also enable in-context and parametric learning of context-management strategies, improving held-out accuracy by up to 35.9 points.