Time for aCTIon: Automated Analysis of Cyber Threat Intelligence in the Wild

2023/07/14 by Giuseppe Siracusano, Davide Sanvito, Siracusano, Giuseppe +15 · 6 citations
Computer Science · #Advanced Malware Detection Techniques #Cryptography and Security (cs.CR) #Cybercrime and Law Enforcement Studies #FOS: Computer and information sciences #Machine Learning (cs.LG) #Software Engineering Research

paper · pdf · doi:10.48550/arxiv.2307.10214

openalex publication_date 2023/07/14 · openalex created_date 2023/07/22 · openalex updated_date 2026/07/28

Abstract

Cyber Threat Intelligence (CTI) plays a crucial role in assessing risks and enhancing security for organizations. However, the process of extracting relevant information from unstructured text sources can be expensive and time-consuming. Our empirical experience shows that existing tools for automated structured CTI extraction have performance limitations. Furthermore, the community lacks a common benchmark to quantitatively assess their performance. We fill these gaps providing a new large open benchmark dataset and aCTIon, a structured CTI information extraction tool. The dataset includes 204 real-world publicly available reports and their corresponding structured CTI information in STIX format. Our team curated the dataset involving three independent groups of CTI analysts working over the course of several months. To the best of our knowledge, this dataset is two orders of magnitude larger than previously released open source datasets. We then design aCTIon, leveraging recently introduced large language models (GPT3.5) in the context of two custom information extraction pipelines. We compare our method with 10 solutions presented in previous work, for which we develop our own implementations when open-source implementations were lacking. Our results show that aCTIon outperforms previous work for structured CTI extraction with an improvement of the F1-score from 10%points to 50%points across all tasks.

Cited by

• CTIGuardian: A Few-Shot Framework for Mitigating Privacy Leakage in Fine-Tuned LLMs
• Large Language Models for Explainable Threat Intelligence
• Large Language Models for Security Operations Centers: A Comprehensive Survey
• A Decade-long Landscape of Advanced Persistent Threats: Longitudinal Analysis and Global Trends
• LLMs in the SOC: An Empirical Study of Human-AI Collaboration in Security Operations Centres
• ExCyTIn-Bench: Evaluating LLM agents on Cyber Threat Investigation

Related

• CTI-HAL: A Human-Annotated Dataset for Cyber Threat Intelligence Analysis
• Actionable Cyber Threat Intelligence using Knowledge Graphs and Large Language Models
• Towards a scalable AI-driven framework for data-independent Cyber Threat Intelligence Information Extraction
• A Threat Intelligence Event Extraction Conceptual Model for Cyber Threat Intelligence Feeds
• A Structured Cyber Threat Intelligence Dataset Using STIX 2.1 Entities and MITRE ATT&CK Mappings