claudegoodies
Skill

huggingface-tokenizers

From NousResearch

Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.

Tokenizes text with Rust-based BPE, WordPiece, or Unigram algorithms and trains custom vocabularies.

Use it when

  • Tokenizing large text corpora fast on CPU
  • Training a custom tokenizer vocabulary from raw files
  • Need token-to-original-text alignment tracking
  • Building a production NLP pipeline needing batch padding/truncation

Skip it if

  • Just loading a pretrained tokenizer — transformers AutoTokenizer already wraps this
  • Need SentencePiece (T5/ALBERT) or tiktoken (GPT) specifically instead
  • Requires Python/Rust dependency install (tokenizers, transformers, datasets)

Facts

Repository
NousResearch/hermes-agent
Status
Actively maintained
Last commit
Source file
optional-skills/mlops/huggingface-tokenizers/SKILL.md

Source preview

The instructions Claude Code reads when this skill runs.

# HuggingFace Tokenizers - Fast Tokenization for NLP

Fast, production-ready tokenizers with Rust performance and Python ease-of-use.

## When to use HuggingFace Tokenizers

**Use HuggingFace Tokenizers when:**
- Need extremely fast tokenization (<20s per GB of text)
- Training custom tokenizers from scratch
- Want alignment tracking (token → original text position)
- Building production NLP pipelines
- Need to tokenize large corpora efficiently

**Performance**:
- **Speed**: <20 seconds to tokenize 1GB on CPU
- **Implementation**: Rust core with Python/Node.js bindings
- **Efficiency**: 10-100× faster than pure Python implementations

**Use alternatives instead**:
- **SentencePiece**: Language-independent, used by T5/ALBERT
- **tiktoken**: OpenAI's BPE tokenizer for GPT models
- **transformers AutoTokenizer**: Loading pretrained only (uses this library internally)

## Quick start

### Installation

```bash
# Install tokenizers
pip install tokenizers

# With transformers integration
pip install tokenizers transformers
```

### Load pretrained tokenizer

```python
from tokenizers import Tokenizer

# Load from HuggingFace Hub
tokenizer = Tokenizer.from_pretrained("bert-base-uncased")

# Encode text
output = tokenizer.encode("Hello, how are you?")
print(output.tokens)  # ['hello', ',', 'how', 'are', 'you', '?']
print(output.ids)     # [7592, 1010, 2129, 2024, 2017, 1029]

# Decode
View full source on GitHub →

Other skills