Wingetlywingetly
← All apps

Tag · microsoft.deepdev.tokenizerlib

Microsoft.deepdev.tokenizerlib apps on winget

1 microsoft.deepdev.tokenizerlib app indexed from microsoft/winget-pkgs. Each one ships with a ready-to-paste winget install command and a silent variant for unattended deployment.

  • T
    TokenizerMicrosoft Corporation
    Microsoft.Tokenizerv1.3.3

    Contains Typescript and C# implementation of byte pair encoding (BPE) tokenizer for OpenAI LLMs, it is based on open sourced rust implementation in the OpenAI tiktoken.