T-LLM: Teaching Large Language Models to Forecast Time Series via Temporal Distillation

This paper proposes T-LLM, a temporal distillation framework that enables general-purpose Large Language Models (LLMs) to perform time series forecasting. By transferring predictive behavior from a lightweight temporal teacher during training, T-LLM consistently outperforms existing LLM-based forecasting methods and offers an efficient deployment pipeline.

State-of-the-art Large Language Models (LLMs) have shown potential in time series forecasting due to their pattern matching abilities. However, they are prohibitively large (billions of parameters), slow to infer, and expensive to run for real-time applications. Existing smaller models often lack the generalization capabilities and context understanding of these LLMs.

The authors propose a Knowledge Distillation framework where a frozen Large Language Model acts as the Teacher. The input time series is patched and tokenized. The Teacher processes this to generate a forecast and internal attention maps. A smaller Student model (based on a lightweight Transformer architecture) is trained to minimize two losses: the standard forecast error against ground truth, and a distillation loss that forces the Student's internal representations and attention weights to match the Teacher's. This effectively transfers the Teacher's ability to recognize long-term dependencies and semantic patterns in the data to the Student.

T-LLM was evaluated on ETTh1, ETTh2, Weather, and Traffic datasets. The student model achieved a Mean Squared Error (MSE) comparable to the Teacher model (within 5% margin) while reducing the parameter count by nearly 95%. T-LLM outperformed supervised baselines like DLinear and PatchTST on zero-shot transfer tasks where the student was distilled on diverse data and tested on unseen domains.

The paper presents a compelling solution to the latency/cost bottleneck of using LLMs for time series. However, the reliance on a powerful teacher implies that the upper bound of performance is capped by the teacher's zero-shot capabilities, which are not perfect for numerical data. The 'Temporal Distillation' is a smart addition, but the paper lacks a robust analysis of how the model behaves when the teacher hallucinates or fails to grasp the trend, potentially cementing errors into the student model. Furthermore, comparing against standard Transformers is good, but comparison against specialized foundation models for time series (like Chronos) is necessary for a complete picture.