Tokenizer viewPro
Text split into tokens with every boundary drawn, ids on demand, and counts per tokenizer side by side.
On this page
ProOne support note through three tokenizers: the smallest vocabulary pays for every byte.
Installation
Usage
import { TokenizerView } from "@/components/ui/chatcn/pro/tokenizer-view"
<TokenizerView tokenizers={[corvid, lumen]} defaultValue={prompt} />A tokenizer is { id, label, detail?, encode(text) }, where encode returns { id, text, start, end } per token. Wrap a real tokenizer (a WASM build of your provider's, for example) to show billing accurate counts.
createFixtureTokenizer from chatcn-pro-eval-a is a deterministic stand-in for demos and tests: it pre-splits like real byte pair tokenizers, keeps common words whole, splits long words, and can spell ideographs byte by byte the way small vocabularies with byte fallback do.
Examples
States
ProToken ids in gray on a log line and an empty field.
Props
| Prop | Type | Notes |
|---|---|---|
tokenizers | Tokenizer[] | One or more; each runs on every change |
value, defaultValue | string | The text |
tokenizer, defaultTokenizer | string | Which one is drawn. Default: the first |
display, defaultDisplay | "text" | "ids" | Default text |
tone | "color" | "mono" | Chart hues or two grays |
editable | boolean | Show the text field. Default true |
maxTokens | number | Tokens drawn at most. Default 4000 |
Keyboard
| Key | Action |
|---|---|
Tab | Focus the token panel |
Arrow left, Arrow right | Previous or next token (mirrored in right to left) |
Home, End | First or last token |
Accessibility
- Tokenizers are a radio group; the token panel is a listbox with the active token announced through
aria-activedescendant. - Spaces show as faint dots and line breaks as a return glyph, so boundaries read without color.
Motion
Count bars resize with a 300 ms transform transition when the text changes; reduced motion jumps them. Hover and arrow keys move the highlight instantly.