the open
    research
    Contribute
    Back to the reading lists

    Deep learning· 2019

    DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

    Victor Sanh (first author) · 2019

    Read the original

    arXiv 1910.01108OpenAlex W2978017171

    On the reading lists

    Why it matters

    Practical pre-training distillation that produced widely used smaller BERT.

    Condensed from the expert lists' notes.

    The record

    First released
    2019 (from arXiv 1910.01108)
    Citations
    10,754 citations, as counted by Papers With Code (CC BY-SA 4.0)
    Named by
    the grok list only
    Score
    0.387 · computed rank 389 · consensus 0.3, must-read 0, citations percentile 0.849 (2010s · deep-learning), awards 0, existing lists 0. Method

    Data sources

    Catalogue data from OpenAlex (CC0), Crossref (CC0 metadata; no abstracts are shown), arXiv, and Papers With Code (CC BY-SA 4.0). Each count names its source. See the method.