Token efficiency via BPE improves secrets scanning over entropy

The article proposes using Byte-Pair Encoding token efficiency to replace entropy as the primary filter after regex-based candidate extraction. Token efficiency measures how many tokens a BPE tokenizer needs to represent a string relative to the string’s length. Secrets that are rare in natural language break into many short tokens, giving a low token efficiency score. Everyday language or common code patterns produce fewer tokens, resulting in a higher score.

Adopting this method requires integrating a BPE tokenizer into the scanning pipeline and computing the token count for each regex match. The extra computation is modest because tokenization is a linear pass over the candidate string. No new rule sets or pattern libraries are needed; only the existing regex stage is kept. Engineers must decide on a threshold for token efficiency that separates secrets from benign text.

The approach may fail when a string is both rare and has high entropy but is not a secret, such as a randomly generated identifier used in a test suite. It also struggles if the tokenizer’s vocabulary has already absorbed certain secret-like patterns, causing them to tokenize efficiently and mask their rarity. In environments where the training data of the tokenizer differs greatly from the target codebase, the frequency assumptions behind BPE may not hold. Consequently, token efficiency works best when the tokenizer reflects the natural language and code patterns of the scanned repositories.

The idea originated from a community issue on the Gitleaks repository, showing that the concept emerged from practitioner feedback rather than a formal research study. With only one source describing the method, independent validation is still limited. Engineers should treat token efficiency as a complementary filter and continue to monitor false-positive rates in their own deployments.