The October 31, 2025 paper introduces **Continuous Autoregressive Language Models (CALM)**, a new paradigm designed to overcome the efficiency bottleneck of traditional Large Language Models (LLMs) by shifting from discrete token-by-token prediction to **continuous next-vector prediction**. This approach compresses a chunk of multiple tokens into a single continuous vector using a **high-fidelity autoencoder**, thereby reducing the number of generative steps and significantly improving the performance-compute trade-off. To manage the challenges of operating in this continuous, likelihood-free domain, the framework includes a comprehensive toolkit: an **energy loss function** for training, a novel, sample-based evaluation metric called **BrierLM**, and **likelihood-free algorithms for temperature sampling**. Ultimately, the CALM framework establishes **semantic bandwidth** as a powerful new axis for scaling language models, enabling superior efficiency compared to discrete baselines.

Source:

October 31, 2025

CONTINUOUS AUTOREGRESSIVE LANGUAGE MODELS

https://arxiv.org/pdf/2510.27688