On-Premise SLMs vs. Commercial LLMs: Prompt Engineering and Incident Classification in SOCs and CSIRTs

2025/11/18 by Almeida, Gefté, Pohlmann, Marcio, Severo, Alex +3
Computer Science · #68T01 #Artificial Intelligence (cs.AI) #Cryptography and Security (cs.CR) #FOS: Computer and information sciences #I.2 #Information and Cyber Security #Machine Learning (cs.LG) #Network Security and Intrusion Detection #Software System Performance and Reliability

paper · doi:10.48550/arxiv.2511.14908

openalex publication_date 2025/11/18 · openalex created_date 2025/11/23 · openalex updated_date 2026/07/28

Abstract

In this study, we evaluate open-source models for security incident classification, comparing them with proprietary models. We utilize a dataset of anonymized real incidents, categorized according to the NIST SP 800-61r3 taxonomy and processed using five prompt-engineering techniques (PHP, SHP, HTP, PRP, and ZSL). The results indicate that, although proprietary models still exhibit higher accuracy, locally deployed open-source models provide advantages in privacy, cost-effectiveness, and data sovereignty.

Citations

• Temperature in SLMs: Impact on Incident Categorization in On-Premises Environments
• A Cost-Benefit Analysis of On-Premise Large Language Model Deployment: Breaking Even with Commercial LLM Services
• Advancing Autonomous Incident Response: Leveraging LLMs and Cyber Threat Intelligence
• Less Data, More Security: Advancing Cybersecurity LLMs Specialization via Resource-Efficient Domain-Adaptive Continuous Pre-training with Minimal Tokens
• Llama-3.1-FoundationAI-SecurityLLM-Base-8B Technical Report

Related

• Evaluating the efficacy of LLM Safety Solutions : The Palit Benchmark Dataset
• An Ethically Grounded LLM-Based Approach to Insider Threat Synthesis and Detection
• Unveiling the Landscape of LLM Deployment in the Wild: An Empirical Study
• Towards an Automated Test of LLM Security Knowledge
• aiXamine: Simplified LLM Safety and Security