Author name: Tawanda Kembo

Tawanda Kembo is a dynamic investor, tech enthusiast, and fourth-generation entrepreneur. From writing his first software program at age 13 to founding Golix, and now leading investment funds PickTwo and Caloti, Tawanda merges digital innovation with strategic investments in the publishing and domain registration sectors. Drawing from diverse IT roles and entrepreneurial ventures, he's not only shaping the future of these industries but also sharing his wealth of knowledge as a sought-after speaker, consultant, and mentor.

Blog

What’s a token

A token is a group of characters, typically a common sequence of characters, which may or may not be part of a larger sequence. The definition of a token can vary because each model has its own unique tokenisation process.

Generally, a token can be as large as an entire English word. However, longer or less common words might be broken down into multiple tokens. On average, one token is about 4 characters, and roughly 100 tokens correspond to about 75 words.

Understanding tokens helps you better interact with LLMs, manage costs, and make the most efficient use of the model’s capabilities.

Scroll to Top