Fine-Tuning Dataset Formatter
Turn question and answer pairs into a JSONL training file for OpenAI, Llama or Gemini.
Estimate how many tokens your text costs before you send it.
How much of a window this text fills, before any answer is generated.
Enter your model's rates per million tokens. Both are editable.
This is an estimate, not a tokenizer. A real count comes from the model's own vocabulary file, which differs between providers and model generations. Expect to be within about ten percent on ordinary English prose, and wider than that on code, non-Latin scripts and heavy punctuation. Confirm with your provider's token counting endpoint before you rely on a hard limit.
The Token Counter estimates how many tokens your text costs before you send it, alongside the word and character counts and what that volume would cost at rates you can edit. A token is roughly three quarters of an English word — common words are one token, rarer ones split into pieces.
Models charge by the token, not the word, and a token is roughly three quarters of an English word — common words are one token, rarer ones split into pieces, and punctuation, code symbols and emoji each take their own. Paste your text and this gives you an estimated token count, the word and character counts beside it, and what that volume would cost at rates you can edit.
Be clear about what this is: an estimate, not a tokenizer. A real count comes from the model's own byte-pair vocabulary, which is megabytes of data and differs between providers and model generations. This uses a heuristic calibrated on English prose and code, which lands within about ten percent for normal text. That is accurate enough to size a context window or sanity-check a bill, and not accurate enough to bet a hard limit on — if you need the exact number, count it through the provider's own token counting endpoint.
A heuristic calibrated on English prose and code, accurate enough to size a context window.
The three counts sit together, which makes the roughly-four-characters-per-token relationship easy to sanity-check.
Type the current per-million rates from your provider and see what the text would cost to send.
A real count comes from the model's own byte-pair vocabulary, so the tool tells you to verify through the provider's token counting endpoint before betting a hard limit on it.
Prompts and documents are measured in your browser, so unreleased content stays local.
Drop in a prompt, a document or a block of code. The estimate updates as you type.
You get estimated tokens, words, characters and the ratio between them, plus how much of a context window it fills.
Set the input and output rates for the model you use and see what the text would cost per call.
For a hard limit, confirm with your provider token counting endpoint. This is a fast estimate, not the tokenizer itself.
Within roughly ten percent for ordinary English prose. Code, heavy punctuation, non-Latin scripts and long unusual words all tokenize less predictably, so treat those figures as a wider range.
A real tokenizer needs the model vocabulary file, which is megabytes and different for each provider and model generation. Downloading that on every page load would break the promise that this site stays fast and fully client-side.
About 0.75 words for English prose, so 1,000 tokens is roughly 750 words. Code runs denser, closer to 0.5 words per token, because symbols and indentation each cost a token.
No. Output is typically several times the price of input, which is why long generated answers cost far more than long prompts. Both rates are editable here so you can price them separately.
What people who use the Token Counter think of it. Reviews come from visitors and are posted as written.
No ratings yet
Yours would be the first.
0 reviews
Newest first
No reviews yet
Be the first to rate the Token Counter. It takes half a minute and no account.
Write the first reviewTurn question and answer pairs into a JSONL training file for OpenAI, Llama or Gemini.
Fill a prompt template from spreadsheet rows and get one prompt per row.
Work out how much GPU memory a local LLM needs to run or fine-tune.
Generate robots.txt rules to block GPTBot, ClaudeBot, CCBot and other AI crawlers.
Scan untrusted text for prompt injection patterns and hidden characters before a model reads it.
Validate a JSONL file line by line, and check fine-tuning datasets before you upload.
Write valid tool-calling JSON schemas without hand-editing braces.
Turn input and output pairs into a clean, consistent few-shot prompt block.