the open
    research
    Contribute
    Back to the reading lists

    Deep learning· 2019

    ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

    Samyam Rajbhandari (first author) · 2019 · published 2020

    Read the original

    DOI 10.1109/sc41405.2020.00024arXiv 1910.02054OpenAlex W3129831491

    On the reading lists

    Why it matters

    Sharded optimizer states, gradients, and parameters, fundamentally changing large-model memory management.

    Condensed from the expert lists' notes.

    The record

    First released
    2019 (from arXiv 1910.02054)
    Published
    2020
    Citations
    2,157 citations, as counted by Papers With Code (CC BY-SA 4.0)
    Named by
    both expert lists
    Score
    0.541 · computed rank 198 · consensus 0.9, must-read 1, citations percentile 0.488 (2010s · deep-learning), awards 0, existing lists 0. Method

    Editor's notes

    • Readability check: fail: A specialised distributed-training systems paper. (AI draft (Claude), not yet reviewed, 2026-10-29)

    Data sources

    Catalogue data from OpenAlex (CC0), Crossref (CC0 metadata; no abstracts are shown), arXiv, and Papers With Code (CC BY-SA 4.0). Each count names its source. See the method.