Zipf-Gramming: Scaling Byte N-Grams Up to Production Sized Malware Corpora

2025/11/17 by Raff, Edward, Curtin, Ryan R., Everett, Derek +2 · 1 citation
Computer Science · #Advanced Malware Detection Techniques #Cryptography and Security (cs.CR) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Mathematical Software (cs.MS) #Software Engineering Research #Software Testing and Debugging Techniques

paper · doi:10.48550/arxiv.2511.13808

openalex publication_date 2025/11/17 · openalex created_date 2025/11/20 · openalex updated_date 2026/07/28

Abstract

A classifier using byte n-grams as features is the only approach we have found fast enough to meet requirements in size (sub 2 MB), speed (multiple GB/s), and latency (sub 10 ms) for deployment in numerous malware detection scenarios. However, we've consistently found that 6-8 grams achieve the best accuracy on our production deployments but have been unable to deploy regularly updated models due to the high cost of finding the top-k most frequent n-grams over terabytes of executable programs. Because the Zipfian distribution well models the distribution of n-grams, we exploit its properties to develop a new top-k n-gram extractor that is up to 35× faster than the previous best alternative. Using our new Zipf-Gramming algorithm, we are able to scale up our production training set and obtain up to 30% improvement in AUC at detecting new malware. We show theoretically and empirically that our approach will select the top-k items with little error and the interplay between theory and engineering required to achieve these results.

Citations

• EMBER2024 -- A Benchmark Dataset for Holistic Evaluation of Malware Classifiers
• Living off the Analyst: Harvesting Features from Yara Rules for Malware Detection
• Is Function Similarity Over-Engineered? Building a Benchmark
• Assemblage: Automatic Binary Dataset Construction for Machine Learning
• Semi-supervised Classification of Malware Families Under Extreme Class Imbalance via Hierarchical Non-Negative Matrix Factorization with Automatic Model Selection
• AVScan2Vec: Feature Learning on Antivirus Scan Data for Production-Scale Malware Corpora
• Efficient Malware Analysis Using Metric Embeddings
• Rank-1 Similarity Matrix Decomposition For Modeling Changes in Antivirus Consensus Through Time
• MOTIF: A Large Malware Reference Dataset with Ground Truth Family Labels
• SPRISS: Approximating Frequent k-mers by Sampling Reads, and Applications
• A Survey of Machine Learning Methods and Challenges for Windows Malware Classification
• Would a File by Any Other Name Seem as Malicious?
• KiloGrams: Very Large N-Grams for Malware Classification
• TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time
• Smooth q-Gram, and Its Applications to Detection of Overlaps among Long, Error-Prone Sequencing Reads
• On Normalized Compression Distance and Large Malware
• Deep Neural Network Based Malware Detection Using Two Dimensional Binary Program Features
• Zipf's law holds for phrases, not words
• Power-law distributions in empirical data
• Power laws, Pareto distributions and Zipf's law
• Handbook of Mathematical Functions With Formulas, Graphs and Mathematical Tables (National Bureau of Standards Applied Mathematics Series No. 55)

Cited by

• Intermediate N-Gramming: Deterministic and Fast N-Grams For Large N and Large Datasets

Related

• KiloGrams: Very Large N-Grams for Malware Classification
• Malware Classification Leveraging NLP & Machine Learning for Enhanced Accuracy
• Malware Detection by Eating a Whole EXE
• Deep Neural Network Based Malware Detection Using Two Dimensional Binary Program Features
• Packed Malware Detection Using Grayscale Binary-to-Image Representations