the open
    research
    Contribute
    Back to the reading lists

    Language, speech & LLMs· 2018

    SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

    Taku Kudo (first author) · 2018

    Read the original

    DOI 10.18653/v1/d18-2012arXiv 1808.06226OpenAlex W2885185669

    On the reading lists

    Why it matters

    Enabled language-independent subword tokenization directly from raw text at scale.

    Condensed from the expert lists' notes.

    The record

    First released
    2018 (from arXiv 1808.06226)
    Citations
    4,363 citations, as counted by Papers With Code (CC BY-SA 4.0)
    Named by
    both expert lists
    Score
    0.420 · computed rank 336 · consensus 0.66, must-read 0, citations percentile 0.633 (2010s · language), awards 0, existing lists 0. Method

    Data sources

    Catalogue data from OpenAlex (CC0), Crossref (CC0 metadata; no abstracts are shown), arXiv, and Papers With Code (CC BY-SA 4.0). Each count names its source. See the method.