Demo Data from VLDB 2026 / reviewers in the wild / expert
Megi Dervishi
dblp:295/8579
· DBLP profile ↗
1ranked-venue papers
1as first author
1since 2021 · last 2025
—ORCID · none
Domains — the database's venue-derived domains; a paper can count in several
Artificial intelligence and machine learning · 1 · 1 first-author · 1 since 2021
Expertise — from the expertise taxonomy: the topics of the expert's papers under the CCF categories. A weight counts papers with recency: 1 for a paper about the topic, 0.3 when the topic is its context, halved every five years.
| Artificial intelligence
1 paper |
Efficient and distributed learning · 33% Language models and text generation · 33% Deep learning architectures and training · 33% |
Topics — the 3 heaviest of 3, each with the papers that count most for it
| Topic | Weight | Papers | Last | Evidence papers |
|---|---|---|---|---|
Machine learning › Efficient and distributed learning › efficient training
compute-optimal training |
0.9 | 1 | 2025 | Training compute-optimal transformer encoder models · EMNLP 2025 |
Natural language and speech › Language models and text generation
masked language modeling |
0.9 | 1 | 2025 | Training compute-optimal transformer encoder models · EMNLP 2025 |
Machine learning › Deep learning architectures and training
transformer |
0.9 | 1 | 2025 | Training compute-optimal transformer encoder models · EMNLP 2025 |
Methods — techniques the papers use, named apart from their topics
scaling laws · 0.9hyperparameter tuning · 0.9
| Year | Publication | Venue | Position |
|---|---|---|---|
| 2025 | Training compute-optimal transformer encoder modelsabstractTransformer encoders are critical for a wide range of Natural Language Processing (NLP) tasks, yet their compute-efficiency remains poorly understood.We present the first comprehensive empirical investigation of compute-optimal pretraining for encoder transformers using the Masked Language Modeling (MLM) objective.Across hundreds of carefully controlled runs we vary model size, data size, batch size, learning rate, and masking ratio, with increasing compute budget.The compute-optimal data-to-model ratio of Transformer encoder models is 10 to 100 times larger than the ratio of auto-regressive models.Using these recipes, we train OptiBERT, a family of compute-optimal BERT-style models that matches or surpasses leading baselines-including ModernBERT and NeoBERT-on GLUE and MTEB while training with dramatically less FLOPS. Megi Dervishi, Alexandre Allauzen, Gabriel Synnaeve, Yann LeCun |
EMNLP | 1 |