Specification and Evaluation of Multi-Agent LLM Systems -- Prototype and Cybersecurity Applications

2025/06/12 by Felix Härer, Härer, Felix · 2 citations
Computer Science · Social Sciences · #68T01 #Access Control and Trust #Artificial Intelligence (cs.AI) #Cryptography and Security (cs.CR) #FOS: Computer and information sciences #I.2.1 #Mobile Agent-Based Network Management #Multi-Agent Systems and Negotiation

paper · pdf · doi:10.48550/arxiv.2506.10467

openalex publication_date 2025/06/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent advancements in LLMs indicate potential for novel applications, e.g., through reasoning capabilities in the latest OpenAI and DeepSeek models. For applying these models in specific domains beyond text generation, LLM-based multi-agent approaches can be utilized that solve complex tasks by combining reasoning techniques, code generation, and software execution. Applications might utilize these capabilities and the knowledge of specialized LLM agents. However, while many evaluations are performed on LLMs, reasoning techniques, and applications individually, their joint specification and combined application is not explored well. Defined specifications for multi-agent LLM systems are required to explore their potential and their suitability for specific applications, allowing for systematic evaluations of LLMs, reasoning techniques, and related aspects. This paper reports the results of exploratory research to specify and evaluate these aspects through a multi-agent system. The system architecture and prototype are extended from previous research and a specification is introduced for multi-agent systems. Test cases involving cybersecurity tasks indicate feasibility of the architecture and evaluation approach. In particular, the results show the evaluation of question answering, server security, and network security tasks that were completed correctly by agents with LLMs from OpenAI and DeepSeek.

Citations

• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• Graph-constrained Reasoning: Faithful Reasoning on Knowledge Graphs with Large Language Models
• Constrained Reasoning Chains for Enhancing Theory-of-Mind in Large Language Models
• CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligence
• SECURE: Benchmarking Large Language Models for Cybersecurity
• When LLMs Meet Cybersecurity: A Systematic Literature Review
• CyberMetric: A Benchmark Dataset based on Retrieval-Augmented Generation for Evaluating LLMs in Cybersecurity Knowledge
• Large Language Model based Multi-Agents: A Survey of Progress and Challenges
• Conceptual Model Interpreter for Large Language Models
• ReCEval: Evaluating Reasoning Chains via Correctness and Informativeness
• Self-Consistency Improves Chain of Thought Reasoning in Language Models
• Attention Is All You Need

Cited by

• Failure Modes in LLM Systems: A System-Level Taxonomy for Reliable AI Applications
• Bridging LLM Planning Agents and Formal Methods: A Case Study in Plan Verification

Related

• The Evolution of Agentic AI in Cybersecurity: From Single LLM Reasoners to Multi-Agent Systems and Autonomous Pipelines
• A Survey on LLM-based Multi-Agent System: Recent Advances and New Frontiers in Application
• Beyond Self-Talk: A Communication-Centric Survey of LLM-Based Multi-Agent Systems
• LLM-based Multi-Agent Systems: Techniques and Business Perspectives
• Seven Security Challenges in Cross-domain Multi-agent LLM Systems