the open
    research
    Contribute

    2026 edition · Top 65

    Deep learning

    Neural network architectures, training methods, and the systems that make them run efficiently.

    Architectures & training: 30 · Efficient ML & systems: 35 · N = 65, scaled from 159 candidates.

    0 of 65 of this page's 65 read · saved in this browser only

    1. 1
      Architectures & training2014

      Adam: A Method for Stochastic Optimization

      Diederik P. Kingma · 172,427 citations · Papers With Code

      AwardEditor's note
    2. 2
      Architectures & training2015

      Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

      Sergey Ioffe · 48,020 citations · Papers With Code

      AwardEditor's note
    3. 3
      Architectures & training1997

      Long Short-Term Memory

      Sepp Hochreiter · 102,283 citations · OpenAlex

      Editor's note
    4. 4
      Efficient ML & systems2015

      Distilling the Knowledge in a Neural Network

      Geoffrey Hinton · 26,833 citations · Papers With Code

      Editor's note
    5. 5
      Architectures & training2015

      Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

      Kaiming He · 21,102 citations · Papers With Code

      Award
    6. 6
      Architectures & training1986

      Learning representations by back-propagating errors

      David E. Rumelhart · 31,877 citations · OpenAlex

      Editor's note
    7. 7
      Efficient ML & systems2021

      LoRA: Low-Rank Adaptation of Large Language Models

      Edward J. Hu · 24,332 citations · Papers With Code

      Editor's note
    8. 8
      Architectures & training1998

      Gradient-Based Learning Applied to Document Recognition

      Yann LeCun · 58,786 citations · OpenAlex

      Editor's note
    9. 9
      Efficient ML & systems2016

      TensorFlow: A system for large-scale machine learning

      Martín Abadi · 19,933 citations · Papers With Code

      Editor's note
    10. 10
      Architectures & training2012 · published 2014

      Dropout: A Simple Way to Prevent Neural Networks from Overfitting

      Nitish Srivastava · 45,097 citations · Semantic Scholar

      Editor's note
    11. 11
      Architectures & training2006

      A Fast Learning Algorithm for Deep Belief Nets

      Geoffrey E. Hinton · 16,335 citations · OpenAlex

      Editor's note
    12. 12
      Efficient ML & systems2019

      PyTorch: An Imperative Style, High-Performance Deep Learning Library

      Adam Paszke · 56,399 citations · Papers With Code

      Editor's note
    13. 13
      Efficient ML & systems2022

      FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

      Tri Dao · 5,579 citations · Papers With Code

      Editor's note
    14. 14
      Efficient ML & systems2023

      Efficient Memory Management for Large Language Model Serving with PagedAttention

      Woosuk Kwon · 9,202 citations · Papers With Code

      Editor's note
    15. 15
      Architectures & training2010

      Understanding the difficulty of training deep feedforward neural networks

      Xavier Glorot · 19,870 citations · Semantic Scholar

      Editor's note
    16. 16
      Architectures & training1943

      A logical calculus of the ideas immanent in nervous activity

      Warren S. McCulloch · 18,563 citations · OpenAlex

      Editor's note
    17. 17
      Architectures & training1958

      The perceptron: A probabilistic model for information storage and organization in the brain

      Frank Rosenblatt · 12,279 citations · OpenAlex

      Editor's note
    18. 18
      Architectures & training2017

      Decoupled Weight Decay Regularization

      Ilya Loshchilov · 41,440 citations · Semantic Scholar

      Editor's note
    19. 19
      Efficient ML & systems2015 · published 2016

      Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding

      Song Han · 10,709 citations · Semantic Scholar

    20. 20
      Efficient ML & systems2023

      QLoRA: Efficient Finetuning of Quantized LLMs

      Tim Dettmers · 6,016 citations · Papers With Code

    21. 21
      Efficient ML & systems2017

      Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

      Noam Shazeer · 5,951 citations · Papers With Code

      Editor's note
    22. 22
      Architectures & training2016

      Layer Normalization

      Jimmy Lei Ba · 13,326 citations · Papers With Code

    23. 23
      Efficient ML & systems2017

      In-Datacenter Performance Analysis of a Tensor Processing Unit

      Norman P. Jouppi · 5,560 citations · Semantic Scholar

      Editor's note
    24. 24
      Architectures & training1982

      Neural networks and physical systems with emergent collective computational abilities

      John J. Hopfield · 19,269 citations · OpenAlex

      Editor's noteClassic quota
    25. 25
      Efficient ML & systems2022

      GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

      Elias Frantar · 2,747 citations · Papers With Code

    26. 26
      Efficient ML & systems2019 · published 2020

      ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

      Samyam Rajbhandari · 2,157 citations · Papers With Code

      Editor's note
    27. 27
      Architectures & training1989

      Approximation by superpositions of a sigmoidal function

      George Cybenko · 13,883 citations · OpenAlex

      Editor's note
    28. 28
      Architectures & training2011

      Deep Sparse Rectifier Neural Networks

      Xavier Glorot · 9,040 citations · Semantic Scholar

    29. 29
      Efficient ML & systems2016

      XGBoost: A Scalable Tree Boosting System

      Tianqi Chen · 63,958 citations · Papers With Code

    30. 30
      Efficient ML & systems2019

      Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

      Mohammad Shoeybi · 3,290 citations · Papers With Code

    31. 31
      Efficient ML & systems2021

      Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

      William Fedus · 5,061 citations · Papers With Code

    32. 32
      Architectures & training1989

      Multilayer feedforward networks are universal approximators

      Kurt Hornik · 21,722 citations · OpenAlex

    33. 33
      Architectures & training1994

      Learning long-term dependencies with gradient descent is difficult

      Yoshua Bengio · 8,539 citations · OpenAlex

    34. 34
      Efficient ML & systems2017

      MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

      Andrew G. Howard · 26,359 citations · Papers With Code

    35. 35
      Architectures & training2016

      Identity Mappings in Deep Residual Networks

      He · 11,461 citations · Papers With Code

    36. 36
      Architectures & training2010

      Rectified Linear Units Improve Restricted Boltzmann Machines

      Vinod Nair · 19,244 citations · Semantic Scholar

    37. 37
      Efficient ML & systems2019

      Parameter-Efficient Transfer Learning for NLP

      Neil Houlsby · 7,563 citations · Papers With Code

    38. 38
      Efficient ML & systems2023

      FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning

      Tri Dao · 3,489 citations · Papers With Code

    39. 39
      Architectures & training1990

      Finding Structure in Time

      Jeffrey L. Elman · 11,131 citations · OpenAlex

    40. 40
      Efficient ML & systems2012

      Large Scale Distributed Deep Networks

      Jeffrey Dean · 4,325 citations · Semantic Scholar

    41. 41
      Efficient ML & systems2022 · published 2023

      Fast Inference from Transformers via Speculative Decoding

      Yaniv Leviathan · 2,196 citations · Papers With Code

    42. 42
      Architectures & training2010 · published 2011

      Adaptive Subgradient Methods for Online Learning and Stochastic Optimization

      John Duchi · 11,918 citations · Semantic Scholar

    43. 43
      Efficient ML & systems2019

      DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

      Victor Sanh · 10,754 citations · Papers With Code

    44. 44
      Efficient ML & systems2018 · published 2019

      The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks

      Jonathan Frankle · 4,519 citations · Papers With Code

    45. 45
      Architectures & training1951

      A Stochastic Approximation Method

      Herbert Robbins · 9,642 citations · OpenAlex

    46. 46
      Architectures & training2013

      On the importance of initialization and momentum in deep learning

      Ilya Sutskever · 5,434 citations · Semantic Scholar

    47. 47
      Efficient ML & systems2021

      Prefix-Tuning: Optimizing Continuous Prompts for Generation

      Xiang Lisa Li · 6,520 citations · Papers With Code

    48. 48
      Architectures & training2016

      SGDR: Stochastic Gradient Descent with Warm Restarts

      Ilya Loshchilov · 11,568 citations · Papers With Code

    49. 49
      Architectures & training2018

      Neural Tangent Kernel: Convergence and Generalization in Neural Networks

      Arthur Jacot · 4,360 citations · Papers With Code

    50. 50
      Architectures & training1980

      Neocognitron: A Self-organizing Neural Network Model for a Mechanism of Pattern Recognition Unaffected by Shift in Position

      Kunihiko Fukushima · 5,724 citations · OpenAlex

      Editor's note
    51. 51
      Efficient ML & systems2015

      Learning both Weights and Connections for Efficient Neural Networks

      Song Han · 8,121 citations · Papers With Code

    52. 52
      Efficient ML & systems2011

      Scikit-learn: Machine Learning in Python

      Fabian Pedregosa · 69,647 citations · OpenAlex

    53. 53
      Efficient ML & systems2014

      Caffe: Convolutional Architecture for Fast Feature Embedding

      Yangqing Jia · 14,907 citations · Papers With Code

    54. 54
      Efficient ML & systems2023 · published 2024

      AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

      Ji Lin · 1,957 citations · Papers With Code

    55. 55
      Efficient ML & systems2004

      MapReduce: Simplified Data Processing on Large Clusters

      Jeffrey Dean · citations unknown

      Editor's note
    56. 56
      Efficient ML & systems2018

      MobileNetV2: Inverted Residuals and Linear Bottlenecks

      28,747 citations · OpenAlex

    57. 57
      Efficient ML & systems2016

      Theano: A Python framework for fast computation of mathematical expressions

      The Theano Development Team · 2,384 citations · Papers With Code

    58. 58
      Efficient ML & systems2017 · published 2018

      Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference

      Benoit Jacob · 4,917 citations · Papers With Code

    59. 59
      Efficient ML & systems2017

      LightGBM: A Highly Efficient Gradient Boosting Decision Tree

      17,818 citations · Semantic Scholar

    60. 60
      Architectures & training1989

      Backpropagation Applied to Handwritten Zip Code Recognition

      Yann LeCun · 11,891 citations · OpenAlex

    61. 61
      Architectures & training2017 · published 2018

      mixup: Beyond Empirical Risk Minimization

      12,564 citations · Papers With Code

    62. 62
      Efficient ML & systems2020

      GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

      Dmitry Lepikhin · 2,467 citations · Papers With Code

    63. 63
      Architectures & training2018 · published 2019

      Reconciling modern machine-learning practice and the classical bias–variance trade-off

      Mikhail Belkin · 1,598 citations · OpenAlex

    64. 64
      Efficient ML & systems2022 · published 2023

      SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

      Guangxuan Xiao · 2,022 citations · Papers With Code

    65. 65
      Efficient ML & systems2014

      Scaling Distributed Machine Learning with the Parameter Server

      Mu Li · 2,024 citations · Semantic Scholar

    Ranked on another page

    Also relevant

    Top 500 papers ranked on another domain page that matter here too.

    1. Computer vision & multimodal2012

      ImageNet Classification with Deep Convolutional Neural Networks

      Alex Krizhevsky · 132,655 citations · Semantic Scholar

      AwardEditor's note
    2. Computer vision & multimodal2015 · published 2016

      Deep Residual Learning for Image Recognition

      Kaiming He · 242,121 citations · Papers With Code

      AwardEditor's note
    3. Language, speech & LLMs2014

      Sequence to Sequence Learning with Neural Networks

      Ilya Sutskever · 22,497 citations · Papers With Code

      AwardEditor's note
    4. Reinforcement learning2015

      Continuous control with deep reinforcement learning

      Timothy P. Lillicrap · 16,206 citations · Papers With Code

      Award
    5. Computer vision & multimodal2014 · published 2015

      Going Deeper with Convolutions

      Christian Szegedy · 48,226 citations · Papers With Code

      Award
    6. Language, speech & LLMs2020

      Scaling Laws for Neural Language Models

      Jared Kaplan · 9,325 citations · Papers With Code

      Editor's note
    7. Computer vision & multimodal2006

      Reducing the Dimensionality of Data with Neural Networks

      Geoffrey E. Hinton · 20,937 citations · OpenAlex

      Editor's note
    8. Computer vision & multimodal2014 · published 2015

      Very Deep Convolutional Networks for Large-Scale Image Recognition

      Karen Simonyan · 114,859 citations · Papers With Code

    9. Language, speech & LLMs2022

      Training Compute-Optimal Large Language Models

      Jordan Hoffmann · 3,889 citations · Papers With Code

      Editor's note
    10. Machine learning foundations2016 · published 2017

      Understanding deep learning requires rethinking generalization

      Chiyuan Zhang · 5,332 citations · Papers With Code

      Editor's note
    11. Language, speech & LLMs2022

      PaLM: Scaling Language Modeling with Pathways

      Aakanksha Chowdhery · 8,476 citations · Papers With Code

    12. Computer vision & multimodal2016 · published 2017

      Densely Connected Convolutional Networks

      Gao Huang · 48,554 citations · OpenAlex

    Data sources

    Catalogue data from OpenAlex (CC0), Crossref (CC0 metadata; no abstracts are shown), arXiv, and Papers With Code (CC BY-SA 4.0), and Semantic Scholar logoSemantic Scholar (used under its non-commercial API licence). Each count names its source. See the method.