See Also

Miscellaneous

• /doc/reinforcement-learning/exploration/active-learning/data-pruning/2024-lin-figure1-rho1lossacceleratestrainingofllmswhilereducingdatause.jpg⁠:

  /doc/reinforcement-learning/exploration/active-learning/data-pruning/2024-lin-figure1-rho1lossacceleratestrainingofllmswhilereducingdatause.jpg

• /doc/reinforcement-learning/exploration/active-learning/data-pruning/2024-lin-figure2-exampleoftokenlevelcleaningofnoisyinternettextdata.png⁠:

  /doc/reinforcement-learning/exploration/active-learning/data-pruning/2024-lin-figure2-exampleoftokenlevelcleaningofnoisyinternettextdata.png

• /doc/reinforcement-learning/exploration/active-learning/data-pruning/2024-lin-figure3-thefourkindsoftokensarealreadylearnedlearnableunlearnableandworsening.png⁠:

  /doc/reinforcement-learning/exploration/active-learning/data-pruning/2024-lin-figure3-thefourkindsoftokensarealreadylearnedlearnableunlearnableandworsening.png

• /doc/reinforcement-learning/exploration/active-learning/data-pruning/2024-lin-figure6-rho1trainingcurvesvsdownstreamlosssplitbytokentype.png⁠:

  /doc/reinforcement-learning/exploration/active-learning/data-pruning/2024-lin-figure6-rho1trainingcurvesvsdownstreamlosssplitbytokentype.png

• /doc/reinforcement-learning/exploration/active-learning/data-pruning/2024-lin-figure8-perplexityofrho1filteredtokensoverthecourseoftraining.png⁠:

  /doc/reinforcement-learning/exploration/active-learning/data-pruning/2024-lin-figure8-perplexityofrho1filteredtokensoverthecourseoftraining.png

• https://aclanthology.org/2023.findings-emnlp.18/⁠:

  View External Link:

  https://aclanthology.org/2023.findings-emnlp.18/

• https://github.com/Guang000/Awesome-Dataset-Distillation⁠:

  View HTML:

  /doc/www/github.com/5294f2e4b637c7d1b221963608b3ffa465a2ac32.html

Bibliography

1. https://arxiv.org/abs/2406.11794: “DataComp-LM: In Search of the next Generation of Training Sets for Language Models”, Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Abbas, Cheng-Yu Hsieh, Dhruba Ghosh, Josh Gardner, Maciej Kilian, Hanlin Zhang, Rulin Shao, Sarah Pratt, Sunny Sanyal, Gabriel Ilharco, Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jieyu Zhang, Khyathi Chandu, Thao Nguyen, Igor Vasiljevic, Sham Kakade, Shuran Song, Sujay Sanghavi, Fartash Faghri, Sewoong Oh, Luke Zettlemoyer, Kyle Lo, Alaaeldin El-Nouby, Hadi Pouransari, Alexander Toshev, Stephanie Wang, Dirk Groeneveld, Luca Soldaini, Pang Wei Koh, Jenia Jitsev, Thomas Kollar, Alexandros G. Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, Vaishaal Shankar

   link-bibliography

2. https://arxiv.org/abs/2405.20541: “Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models”, Zachary Ankner, Cody Blakeney, Kartik Sreenivasan, Max Marion, Matthew L. Leavitt, Mansheej Paul

   link-bibliography

3. https://arxiv.org/abs/2404.07965#microsoft: “Rho-1: Not All Tokens Are What You Need”, Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan, Weizhu Chen

   link-bibliography

4. https://arxiv.org/abs/2402.07625: “Autonomous Data Selection With Language Models for Mathematical Texts”, Yifan Zhang, Yifan Luo, Yang Yuan, Andrew Chi-Chih Yao

   link-bibliography

5. https://arxiv.org/abs/2401.16380#apple: “Rephrasing the Web (WARP): A Recipe for Compute and Data-Efficient Language Modeling”, Pratyush Maini, Skyler Seto, He Bai, David Grangier, Yizhe Zhang, Navdeep Jaitly

   link-bibliography

6. https://arxiv.org/abs/2312.05328#deepmind: “Bad Students Make Great Teachers: Active Learning Accelerates Large-Scale Visual Understanding”, Talfan Evans, Shreya Pathak, Hamza Merzic, Jonathan Schwarz, Ryutaro Tanno, Olivier J. Henaff

   link-bibliography

7. https://arxiv.org/abs/2309.17425#apple: “Data Filtering Networks”, Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, Vaishaal Shankar

   link-bibliography

8. https://arxiv.org/abs/2309.10818#cerebras: “SlimPajama-DC: Understanding Data Combinations for LLM Training”, Zhiqiang Shen, Tianhua Tao, Liqun Ma, Willie Neiswanger, Zhengzhong Liu, Hongyi Wang, Bowen Tan, Joel Hestness, Natalia Vassilieva, Daria Soboleva, Eric Xing

   link-bibliography

9. https://arxiv.org/abs/2302.03169: “Data Selection for Language Models via Importance Resampling”, Sang Michael Xie, Shibani Santurkar, Tengyu Ma, Percy Liang

   link-bibliography

10. https://arxiv.org/abs/2206.14486: “Beyond Neural Scaling Laws: Beating Power Law Scaling via Data Pruning”, Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, Ari S. Morcos

    link-bibliography