Update README.md
Browse files
README.md
CHANGED
@@ -1,3 +1,51 @@
|
|
1 |
-
---
|
2 |
-
license: mit
|
3 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
---
|
2 |
+
license: mit
|
3 |
+
datasets:
|
4 |
+
- trendmicro-ailab/Primus-FineWeb
|
5 |
+
language:
|
6 |
+
- en
|
7 |
+
base_model:
|
8 |
+
- meta-llama/Llama-3.1-8B-Instruct
|
9 |
+
pipeline_tag: text-generation
|
10 |
+
library_name: transformers
|
11 |
+
tags:
|
12 |
+
- cybersecurity
|
13 |
+
- pretraining
|
14 |
+
---
|
15 |
+
|
16 |
+
# Primus: A Pioneering Collection of Open-Source Datasets for Cybersecurity LLM Training
|
17 |
+
|
18 |
+
<img src="https://i.imgur.com/PtqeTZw.png" alt="Llama-Primus-Merged Overview" width="60%">
|
19 |
+
|
20 |
+
> TL;DR: Llama-Primus-Base is a foundation model based on Llama-3.1-8B-Instruct, continually pre-trained on Primus-Seed (0.2B) and Primus-FineWeb (2.57B). Primus-Seed is a high-quality, manually curated cybersecurity text dataset, while Primus-FineWeb consists of cybersecurity texts filtered from FineWeb. By pretraining on such a large-scale cybersecurity corpus, it achieves a 🚀**15.88%** improvement in aggregated scores across multiple cybersecurity benchmarks, demonstrating the effectiveness of cybersecurity-specific pretraining.
|
21 |
+
|
22 |
+
**🔥 For more details, please refer to the paper: [[📄Paper]](https://arxiv.org/abs/2502.11191).**
|
23 |
+
|
24 |
+
## Introduction
|
25 |
+
|
26 |
+
Large Language Models (LLMs) have demonstrated remarkable versatility in recent years, with promising applications in specialized domains such as finance, law, and biomedicine. However, in the domain of cybersecurity, we noticed a lack of open-source datasets specifically designed for LLM pre-training—even though much research has shown that LLMs acquire their knowledge during pre-training. To fill this gap, we present a collection of datasets covering multiple stages of cybersecurity LLM training, including pre-training (_Primus-Seed_ and _Primus-FineWeb_), instruction fine-tuning (_Primus-Instruct_), and reasoning data for distillation (_Primus-Reasoning_). Based on these datasets and Llama-3.1-8B-Instruct, we developed _Llama-Primus-Base_, _Llama-Primus-Merged_, and _Llama-Primus-Reasoning_. This model card is **Llama-Primus-Base**.
|
27 |
+
|
28 |
+
> **Note:** No TrendMicro customer information is included.
|
29 |
+
|
30 |
+
## Cybersecurity Benchmark Results
|
31 |
+
|
32 |
+
| **Metric** | **Llama-3.1-8B-Instruct** | **Llama-Primus-Base** |
|
33 |
+
|---------------------------------|---------------------------|------------------------------|
|
34 |
+
| **CISSP (Exams in book)** | 0.7073 | **0.7230** |
|
35 |
+
| **CTI-Bench (MCQ)** | 0.6420 | **0.6676** |
|
36 |
+
| **CTI-Bench (CVE → CWE)** | 0.5910 | **0.6780** |
|
37 |
+
| **CTI-Bench (CVSS, _lower is better_)** | 1.2712 | **1.0912** |
|
38 |
+
| **CTI-Bench (ATE)** | 0.2721 | **0.3140** |
|
39 |
+
| **CyberMetric (500)** | 0.8560 | **0.8660** |
|
40 |
+
| **SecEval** | 0.4966 | **0.5007** |
|
41 |
+
| **_Agg._** | 2.29 | **2.66** ↑**15.88%** 🔥 |
|
42 |
+
|
43 |
+
CTI-Bench (CVSS) is scored using Mean Absolute Deviation (_lower is better_), CTI-ATE uses F1 score, and the others use accuracy. The aggregate score (_Agg._) is the sum of all benchmarks, with CTI-Bench (CVSS) negated.
|
44 |
+
|
45 |
+
References:
|
46 |
+
- **CyberMetric**: [CyberMetric: A Benchmark Dataset based on Retrieval-Augmented...](https://arxiv.org/abs/2402.07688)
|
47 |
+
- **CtiBench**: [CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligence](https://arxiv.org/abs/2406.07599)
|
48 |
+
- **SecEval**: [SecEval: A Comprehensive Benchmark for Evaluating Cybersecurity Knowledge of Foundation Models](https://xuanwuai.github.io/SecEval/)
|
49 |
+
|
50 |
+
## License
|
51 |
+
This model is based on the MIT license, but you must also comply with the Llama 3.1 Community License Agreement.
|