tokc

tokc counts tokens using pretrainded models from HuggingFace.

Example usage

$ echo "The quick brown fox jumps over the lazy dog" | tokc -model mistralai/Mistral-Small-3.1-24B-Base-2503
11

Install

go install github.com/WinPooh32/tokc@latest