Language Entropy

The Apius Language Entropy app measures how well any string matches the character patterns of a given language. Its custom search command computes bigram cross-entropy against trained language models: strings built from natural, language-typical character sequences receive low scores, while random-looking or machine-generated strings — hashes, tokens, encoded payloads, algorithmically generated names — stand out with high scores. Unlike plain Shannon entropy, which only measures character distribution within a string, cross-entropy scoring evaluates strings against real language statistics, reliably separating genuine words from random sequences built from the same characters. Scoring works with any language: the app ships with a Polish (default) and English bigram models and lets you build and upload models for other languages from simple frequency files through its web interface. Scoring the same field against multiple models also reveals which language a string most likely belongs to. Typical applications include security use cases such as detecting DGA domains, DNS tunneling, and C2 traffic hiding behind random-looking names, as well as broader data-quality and anomaly-detection scenarios: spotting machine-generated identifiers where human-given names are expected, flagging encoded or obfuscated content in logs, and filtering garbage values in text fields. The app includes the entropy streaming search command (usable in any SPL pipeline), a model-management page for converting bigram frequency files into ready-to-use probability models, and an interactive Entropy Lab for scoring individual strings or batch files, tuning thresholds with per-model summary statistics, and exporting results to CSV.