ICML 2025
Fast and Low-Cost Genomic Foundation Models via Outlier Removal
Remove outliers to make genomic foundation models easier to adapt and quantize.
Abstract
To address the challenge of scarce computational resources in genomic modeling, this paper introduces GERM, a genomic foundation model with strong compression performance and fast adaptability. GERM eliminates outliers during both pretraining and fine-tuning stages, enhancing both LoRA-based low-rank adaptation and post-training quantization robustness. Additionally, a continual learning variant named GERM-T further reduces retraining costs by applying outlier removal in small steps. Compared to DNABERT-2, GERM improves fine-tuning performance by 37.98%, quantization robustness by 64.34%, and reduces average kurtosis by 92.14%. These results demonstrate GERM’s practical effectiveness for genomic tasks in resource-constrained environments.
Key results
Outlier-free model design
Replace standard attention with an outlier-free mechanism to improve the behavior of genomic foundation models.
Efficient adaptation and quantization
Evaluate how outlier removal improves low-rank fine-tuning and makes low-precision deployment more robust.
Reuse existing checkpoints
Introduce GERM-T, a small-step continual-learning strategy that reduces the need to pretrain again from scratch.
Poster
BibTeX
Official BibTeX from PMLR.
@InProceedings{pmlr-v267-luo25g,
title = {Fast and Low-Cost Genomic Foundation Models via Outlier Removal},
author = {Luo, Haozheng and Qiu, Chenghao and Su, Maojiang and Zhou, Zhihan and Mehta, Zoe and Ye, Guo and Hu, Jerry Yao-Chieh and Liu, Han},
booktitle = {Proceedings of the 42nd International Conference on Machine Learning},
pages = {41254--41289},
year = {2025},
editor = {Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry},
volume = {267},
series = {Proceedings of Machine Learning Research},
month = {13--19 Jul},
publisher = {PMLR},
pdf = {https://raw.githubusercontent.com/mlresearch/v267/main/assets/luo25g/luo25g.pdf},
url = {https://proceedings.mlr.press/v267/luo25g.html},
abstract = {To address the challenge of scarce computational resources in genomic modeling, we introduce GERM, a genomic foundation model optimized for accessibility and adaptability. GERM improves upon models like DNABERT-2 by eliminating outliers that hinder low-rank adaptation and post-training quantization, enhancing both efficiency and robustness. We replace the vanilla attention layer with an outlier-free mechanism inspired by associative memory models. By removing outliers during both pre-training and fine-tuning, this approach accelerates adaptation, reduces computational costs, and enhances quantization robustness within acceptable loss margins. Additionally, we propose GERM-T, a strategy that employs small-step continual learning within the outlier-free framework, leveraging original checkpoints to avoid retraining from scratch. Empirically, GERM improves fine-tuning performance by 37.98% and quantization by 64.34% over the baseline model. It also reduces average kurtosis by 92.14% and maximum infinity norm by 82.77%. Compared to leading methods, GERM consistently delivers superior performance, offering a practical solution for genomic modeling in resource-constrained settings.}
}