the open
    research
    Contribute
    Back to the reading lists

    Deep learning· 2019

    Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

    Mohammad Shoeybi (first author) · 2019

    Read the original

    arXiv 1909.08053OpenAlex W2973727699

    On the reading lists

    Why it matters

    Popularized practical intra-layer tensor parallelism for training multi-billion-parameter Transformers.

    Condensed from the expert lists' notes.

    The record

    First released
    2019 (from arXiv 1909.08053)
    Citations
    3,290 citations, as counted by Papers With Code (CC BY-SA 4.0)
    Named by
    both expert lists
    Also relevant to
    Language, speech & LLMs
    Score
    0.495 · computed rank 236 · consensus 0.82, must-read 0.5, citations percentile 0.57 (2010s · deep-learning), awards 0, existing lists 0. Method

    Data sources

    Catalogue data from OpenAlex (CC0), Crossref (CC0 metadata; no abstracts are shown), arXiv, and Papers With Code (CC BY-SA 4.0). Each count names its source. See the method.