Transformers are Deep Optimizers

In-Context Deep Learning via Transformer Models

This paper investigates the transformer’s capability for in-context learning (ICL) to simulate the training process of deep models, providing a provable explicit construction.

May 2025 · 0 min · Weimin Wu, Maojiang Su, Jerry Yao-Chieh Hu, Zhao Song, Han Liu