NetPrompt: Evaluation of LLMs as Network Intrusion Detection System

Pratyay Kumar, Abu Saleh Md Tayeen, Qixu Gong, Jiefei Liu, Satyajayant Misra,
Huiping Cao, Jayashree Harikumar

MILCOM 2025 (Accepted and Presented)

Guide to use LLMs as NIDS

Welcome to the NetPrompt repository for Evaluating Large Language Models (LLMs) as Network Intrusion Detection System (NIDS). This repository contains the Processed Datasets and Prompts: Zer-Shot (ZS), Few-Shot-p (FS) where p ∈ {1, 2, 3}, and Chain-of-Thought (CoT).

1. 🚀Environment Setting

1. Create and activate conda environment:

   conda create -n netprompt python=3.9.21
   conda activate netprompt

2. Install dependencies:

   pip install -r requirements.txt

3. Configure API keys:

   • HuggingFace Token (for LLaMA):
     • Edit llm/llama.py at line 9, replace "******" with your token
     • Edit main_cot.py at line 44, replace "******" with your token
   • Gemini API Key: Edit main.py at line 100, replace "*****" with your API key

4. Download data:

   • Download the data folder from data and place it at ./data in the repository root.

2. 🔬 Qick Start to Reproducing the Results Shown in the Paper

To reproduce the results, run the following bash scripts from the repository root:

./scripts/llama.sh
./scripts/qwen.sh
./scripts/gemini.sh

Note: Running all experiments end-to-end is compute/time intensive. On our hardware (AMD EPYC 7313 16-Core processor, 503 GB of RAM, and a single NVIDIA A100 GPU with 40 GB VRAM), completing the full set took more than 1 month.

For individual experiments:

• Run main.py for Zero-Shot (ZS) and Few-Shot (FS) experiments
• Run main_cot.py for Chain-of-Thought (CoT) experiments

See the Manual Reproduction section for detailed CLI examples and options.

2.2 Estimate running time for above bash files (better configuration might use less running time):

Average inference time (in seconds) to process each sample on the CICIDS2017 dataset.

Model	Inference Time
Baseline ML Model	0.000000874

Model	ZS	FS-1	FS-2	FS-3
Qwen2.5-7B-Instruct	0.1047	0.1942	0.2691	4.5001
Llama-3.1-8B-Instruct	0.1452	0.1976	0.2549	4.1652
Gemini-2.0-Flash	0.4250	0.6035	0.7717	0.8312

3. Project Structure

• data: Processed version of CICIDS2017 and CICDDoS2019 datasets.
• llm: Wrapper classes for the LLM models used in our experiments.
• prompt: All prompt templates used for evaluation, including ZS, FS, and CoT formats.
• scripts: Shell scripts to reproduce all experiments.
• models: Pretrained baseline MLP models used for comparative evaluation.
• tables: Contains additional results with Precision and Recall (not included in the main body of the paper).

4. Manual Reproduction (Detailed Steps)

4.1 General format for Zero-Shot & Few-Shot prompting:

python main.py -m <model_name> -d <data_year> -p <prompt_type> [-e <fewshot_example_number>]

• -m/--model_name: gemini, llama or qwen
• -d/--data_year: 2017 or 2019
• -p/--prompt_type: zeroshot or fewshot
• -e/--num_examples: Requires for fewshot prompting only

4.2 General format for Chain-of-Thought prompting:

python main_cot.py --help
usage: main_cot.py [-h] [--model_name {qwen,llama}] [--data_year {2017,2019}] [--prompt_name PROMPT_NAME] [--attack_desc_enable] [--num_examples NUM_EXAMPLES] [--feature_type FEATURE_TYPE]

• -h/--help: show help message and exit
• --model_name: Model to use - llama or qwen
• -d/--data_year: 2017 or 2019
• --prompt_name: PROMPT_NAME - Name of the prompt strategy to use
• --attack_desc_enable: Enable attack descriptions in prompt
• --num_examples: NUM_EXAMPLES - Number of examples to include in prompt
• feature_type: FEATURE_TYPE

4.3 Baseline MLP Model

Train the baseline MLP model:

python train_model.py -m <data_year>

• -m/--model: 2017 (CICIDS2017) or 2019 (CICDDoS2019)

Example:

python train_model.py -m 2017

Test the baseline MLP model:

python test_ml.py -m <data_year>

• -m/--model: 2017 (CICIDS2017) or 2019 (CICDDoS2019)

Example:

python test_ml.py -m 2019

4.4 Extend NetPrompt

To add a new LLM:

• Define a new class under llm/
• Add the model in main.py

To add new prompts:

• Add your prompt under prompt/
• Reference it in main.py or main_cot.py

Citation

This work has been accepted and presented at MILCOM 2025. The BibTeX citation will be added here once the proceedings are published.

License

This project is licensed under the GNU General Public License v2.0 - see the LICENSE file for details.