the open
    research
    Contribute
    Back to the reading lists

    Deep learning· 2021

    Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

    William Fedus (first author) · 2021

    Read the original

    arXiv 2101.03961OpenAlex W3119866685

    On the reading lists

    Why it matters

    Simplified expert routing and demonstrated efficient trillion-parameter sparse language-model training.

    Condensed from the expert lists' notes.

    The record

    First released
    2021 (from arXiv 2101.03961)
    Citations
    5,061 citations, as counted by Papers With Code (CC BY-SA 4.0)
    Named by
    both expert lists
    Also relevant to
    Language, speech & LLMs
    Score
    0.488 · computed rank 242 · consensus 0.68, must-read 0, citations percentile 0.812 (2020s · deep-learning), awards 0, existing lists 0. Method

    Data sources

    Catalogue data from OpenAlex (CC0), Crossref (CC0 metadata; no abstracts are shown), arXiv, and Papers With Code (CC BY-SA 4.0). Each count names its source. See the method.