An exact token count needs an encoding and a boundary

Two counters can disagree without either being broken. First ask which encoding they use and whether they count only the pasted text or the surrounding request as well.

Name the unit before using the number

Words are useful reading units; UTF-8 bytes measure an encoded representation. Tokens follow a particular encoding. There is no fixed word-to-token or character-to-token conversion that works for every text.

ToolRoof counts literal text with cl100k_base and reports both tokens and bytes. Exactness applies to that encoding, not to every model. Selecting a different model in your application does not change this counter.

The visible text is not the whole request

An API request may contain message roles, tool definitions, files and other material beyond the paragraph you pasted. A correct count for that paragraph can therefore be incomplete for the request.

Near a provider limit, use its appropriate request measurement or usage report. A fixed extra allowance is not a guarantee: the missing material depends on the request structure.

References: OpenAI — Plain-text counts versus complete request token counts

Do not silently rewrite text to improve the count

Spaces, line endings and character representation belong to the input. Strings that look alike may contain different code points. Keep the exact version you measured so comparisons can be reproduced.

The local counter does not normalize Unicode or replace user digits. Its limit is 20000 UTF-16 code units, not a promise about that many visible characters or tokens. Malformed text is rejected rather than silently changed.

Example: similar appearance, different representation

Under cl100k_base, the literal string hello world uses 11 UTF-8 bytes and 2 tokens. The phrase, its byte representation and its token segmentation are different quantities.

The single code point é, U+00E9, uses 1 token and 2 bytes. An e followed by U+0301 looks similar but uses 2 tokens and 3 bytes. This is not a ranking of languages or writing styles; it demonstrates why the original text and encoding belong beside a measurement.

References