Demo Data from EDBT 2026 / reviewers in the wild / expert
James Lee-Thorp
dblp:292/3901
· DBLP profile ↗
6ranked-venue papers
1as first author
6since 2021 · last 2024
0000-0001-6445-7155ORCID · corroborated
Domains — the database's venue-derived domains; a paper can count in several
Artificial intelligence and machine learning · 6 · 1 first-author · 6 since 2021
Expertise — from the expertise taxonomy: the topics of the expert's papers under the CCF categories. A weight counts papers with recency: 1 for a paper about the topic, 0.3 when the topic is its context, halved every five years.
| Artificial intelligence
4 papers |
Deep learning architectures and training · 56% Efficient and distributed learning · 37% Language models and text generation · 7% |
Topics — the 11 heaviest of 11, each with the papers that count most for it
| Topic | Weight | Papers | Last | Evidence papers |
|---|---|---|---|---|
Machine learning › Deep learning architectures and training
attention mechanism |
0.7 | 1 | 2023 | GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints · EMNLP 2023 |
Machine learning › Efficient and distributed learning › distributed training
distributed training systems |
0.7 | 1 | 2023 | Scaling Up Models and Data with t5x and seqio · J. Mach. Learn. Res. 2023 |
Machine learning › Deep learning architectures and training › transformer
efficient transformer |
0.7 | 1 | 2023 | CoLT5: Faster Long-Range Transformers with Conditional Computation · EMNLP 2023 |
Machine learning › Efficient and distributed learning › large-scale learning
large-scale model training |
0.7 | 1 | 2023 | Scaling Up Models and Data with t5x and seqio · J. Mach. Learn. Res. 2023 |
Machine learning › Deep learning architectures and training
mixture of experts |
0.7 | 1 | 2023 | Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints · ICLR 2023 |
Machine learning › Efficient and distributed learning
model compression |
0.7 | 1 | 2023 | Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints · ICLR 2023 |
Machine learning › Deep learning architectures and training › mixture of experts
sparse mixture-of-experts |
0.7 | 1 | 2023 | Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints · ICLR 2023 |
Machine learning › Deep learning architectures and training
transformer |
0.7 | 1 | 2023 | GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints · EMNLP 2023 |
Machine learning › Efficient and distributed learning
inference efficiency |
0.2 | 1 | 2023 | GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints · EMNLP 2023 |
Natural language and speech › Language models and text generation
large language model |
0.2 | 1 | 2023 | Scaling Up Models and Data with t5x and seqio · J. Mach. Learn. Res. 2023 |
Natural language and speech › Language models and text generation › language modeling
long-context language modeling |
0.2 | 1 | 2023 | CoLT5: Faster Long-Range Transformers with Conditional Computation · EMNLP 2023 |
Methods — techniques the papers use, named apart from their topics
uptraining · 0.7sparse attention · 0.7knowledge transfer · 0.7dense checkpoint initialization · 0.7conditional computation · 0.7checkpoint conversion · 0.7
| Year | Publication | Venue | Position |
|---|---|---|---|
| 2024 | Memory Augmented Language Models through Mixture of Word ExpertsabstractCicero Nogueira dos Santos, James Lee-Thorp, Isaac Noble, Chung-Ching Chang, David Uthus. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024. Cícero Nogueira dos Santos, James Lee-Thorp, Isaac Noble, Chung-Ching Chang, David C. Uthus |
NAACL-HLT | 2 |
| 2023 | CoLT5: Faster Long-Range Transformers with Conditional ComputationabstractJoshua Ainslie, Tao Lei, Michiel de Jong, Santiago Ontanon, Siddhartha Brahma, Yury Zemlyanskiy, David Uthus, Mandy Guo, James Lee-Thorp, Yi Tay, Yun-Hsuan Sung, Sumit Sanghai. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023. Joshua Ainslie, Tao Lei 0001, Michiel de Jong, Santiago Ontañón, Siddhartha Brahma, Yury Zemlyanskiy, David C. Uthus, Mandy Guo, James Lee-Thorp, Yi Tay, Yun-Hsuan Sung, Sumit Sanghai |
EMNLP | 9 |
| 2023 | GQA: Training Generalized Multi-Query Transformer Models from Multi-Head CheckpointsabstractMulti-query attention (MQA), which only uses a single key-value head, drastically speeds up decoder inference.However, MQA can lead to quality degradation, and moreover it may not be desirable to train a separate model just for faster inference.We (1) propose a recipe for uptraining existing multi-head language model checkpoints into models with MQA using 5% of original pre-training compute, and (2) introduce grouped-query attention (GQA), a generalization of multi-query attention which uses an intermediate (more than one, less than number of query heads) number of key-value heads.We show that uptrained GQA achieves quality close to multi-head attention with comparable speed to MQA. Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, Sumit Sanghai |
EMNLP | 2 |
| 2023 | Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints
Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani 0001, Neil Houlsby |
ICLR | 3 |
| 2023 | Scaling Up Models and Data with t5x and seqioabstractScaling up training datasets and model parameters have benefited neural network-based language models, but also present challenges like distributed compute, input data bottlenecks and reproducibility of results. We introduce two simple and scalable software libraries that simplify these issues: t5x enables training large language models at scale, while seqio enables reproducible input and evaluation pipelines. These open-source libraries have been used to train models with hundreds of billions of parameters on multi-terabyte datasets. Configurations and instructions for T5-like and GPT-like models are also provided. The libraries can be found at https://github.com/google-research/t5x and https://github.com/google/seqio. Adam Roberts, Hyung Won Chung, Anselm Levskaya, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowycz, Alex Salcianu, Marc van Zee, Jacob Austin, Sebastian Goodman, Livio B. Soares, Haitang Hu, Sasha Tsvyashchenko, Aakanksha Chowdhery, Jasmijn Bastings, Jannis Bulian, Xavier Garcia, Jianmo Ni, Kathleen Kenealy, Kehang Han, Michelle Casbon, Jonathan H. Clark, Stephan Lee, Dan Garrette, James Lee-Thorp, Colin Raffel, Noam Shazeer, Marvin Ritter, Maarten Bosma, Alexandre Tachard Passos, Jeremy Maitin-Shepard, Noah Fiedel, Mark Omernick, Brennan Saeta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlan, Andrea Gesmundo |
J. Mach. Learn. Res. | 32 |
| 2022 | FNet: Mixing Tokens with Fourier TransformsabstractJames Lee-Thorp, Joshua Ainslie, Ilya Eckstein, Santiago Ontanon. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022. James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, Santiago Ontañón |
NAACL-HLT | 1 |