Transformers

Transformers are Deep Optimizers: Provable In-Context Learning for DeepModel Training

This paper investigates the transformer’s capability for in-context learning (ICL) to simulate the training process of deep models, providing a provable explicit construction.

Computational limits of low-rank adaptation (lora) for transformer-based models

We study the computational limits of Low-Rank Adaptation (LoRA) update for finetuning transformer-based models using fine-grained complexity theory.