--- license: mit language: - en - ja base_model: - trend-cybertron/Llama-Primus-Nemotron-70B-Base pipeline_tag: text-generation extra_gated_fields: Affiliation: text Country: country I want to use this model for: type: select options: - Research - Commercial - label: Other value: other Job title: type: select options: - Student - Research graduate - AI researcher - AI developer/engineer - Cybersecurity researcher - Reporter - Other geo: ip_location library_name: transformers datasets: - trendmicro-ailab/Primus-FineWeb - trendmicro-ailab/Primus-Instruct tags: - cybersecurity --- # Llama-Primus-Nemotron-70B-Instruct Llama-Primus-Nemorton - [Introduction](#introduction) - [Benchmark Results](#benchmark-results) - [Training Datasets](#training-datasets) - [Acknowledgments](#acknowledgments) ## Introduction The **Llama-Primus-Nemotron** series builds upon `nvidia/Llama-3.1-Nemotron-70B-Instruct` through continued training. Following the same methodology as described in the [Primus paper](https://arxiv.org/abs/2502.11191), we first performed pre-training on large-scale cybersecurity corpora (over **10B** tokens) to obtain **Llama-Primus-Nemotron-Base**. We then conducted supervised-finetuning and applied [DELLA](https://arxiv.org/abs/2406.11617) to merge with the original Nemotron, resulting in **Llama-Primus-Nemotron-70B-Instruct**. _Llama-Primus-Nemotron-70B-Instruct_ achieves an **18.18%** improvement in aggregate scores across several public cybersecurity benchmarks, while maintaining **same** performance in general-purpose instruction following benchmark (Arena Hard). --- ## Benchmark Results - [Cybersecurity](#cybersecurity) - [General Chat Performance](#general-chat-performance) - [Safety & Toxicity](#safety--toxicity) ### Cybersecurity | **Metric** (5-shot, w/ chat template) | **Llama-3.1-Nemotron-70B-Instruct** | **Llama-Primus-Nemotron-70B-Instruct** | |-------------------------------------------|-------------------------------------|----------------------------------------| | **CTI-Bench (MCQ)** | 0.6320 | 0.7148 | | **CTI-Bench (CVE → CWE)** | 0.6020 | 0.6770 | | **CTI-Bench (CVSS, _lower is better_)** | 1.4523 | 1.2469 | | **CTI-Bench (ATE)** | 0.4284 | 0.5039 | | **CyberMetric (500)** | 0.9240 | 0.9280 | | **SecEval** | 0.6875 | 0.7095 | | **CISSP (Exam Questions)** | 0.8428 | 0.8625 | | **_Aggregate_** | 2.6644 | 3.1488 **↑18.18%** 🔥 | CTI-Bench(CVSS) is scored using Mean Absolute Deviation (_lower is better_), CTI-ATE uses F1 score, and the others use accuracy. The aggregate score (_Agg._) is the sum of all benchmarks, with CTI-Bench(CVSS) negated. References: - **CyberMetric**: [CyberMetric: A Benchmark Dataset based on Retrieval-Augmented...](https://arxiv.org/abs/2402.07688) - **CTI-Bench**: [CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligence](https://arxiv.org/abs/2406.07599) - **SecEval**: [SecEval: A Comprehensive Benchmark for Evaluating Cybersecurity Knowledge of Foundation Models](https://xuanwuai.github.io/SecEval/) ### General Chat Performance | **Metric** | **Llama-3.1-Nemotron-70B-Instruct** | **Llama-Primus-Nemotron-70B-Instruct** | |-----------------|-------------------------------------|----------------------------------------| | **Arena Hard** | 85.1 | 85.8 | Reference: - [**Arena Hard**](https://github.com/lmarena/arena-hard-auto) ### Safety & Toxicity | **Metric** | **Llama-3.1-Nemotron-70B-Instruct** | **Primus-Labor-70B (Llama-3.1-Nemotron-70B-Instruct)** 🔥 | |-------------------------------------------|-------------------------------------|-----------------------------------------------------------| | **dan (Jailbreak)** | 43.14% | 61.96% | | **encoding (Jailbreak)** | 93.37% | 96.87% | | **goodside (Hallucination / Prompt Injection)** | 75.00% | 72.50% | | **latentinjection (Prompt Injection)** | 62.46% | 70.35% | | **leakreplay (Copyright)** | 88.23% | 92.43% | | **malwaregen (Disallowed content)** | 18.99% | 25.84% | | **realtoxicityprompts (Disallowed content)** | 97.55% | 98.25% | | **snowball (Hallucination)** | 100.00% | 100.00% | | **xss (Prompt Injection)** | 81.67% | 100.00% | | **XSTest (Over Refusal)** | 94.40% | 97.20% | References: - **Garak**: [Garak Repository](https://github.com/leondz/garak) - **XSTest**: [XSTest Repository](https://github.com/paul-rottger/exaggerated-safety) --- ## Training Datasets #### Pre-training: - **Primus-Seed-V2 (0.417B):** An enhanced version of [Primus-Seed](https://huggingface.co/datasets/trendmicro-ailab/Primus-Seed), enriched with blogs, news, books, websites, Wikipedia, MITRE and Trend Micro knowledge. - **Primus-FineWeb (2.57B):** Cybersecurity text filtered from FineWeb-edu-score-2. [Link](https://huggingface.co/datasets/trendmicro-ailab/Primus-FineWeb) - **Primus-Nemotron-CC (7.6B):** Cybersecurity text filtered from Nemotron-CC. #### SFT: - **Primus-Instruct**: [LINK](https://huggingface.co/datasets/trendmicro-ailab/Primus-Instruct) > **Note:** Datasets *Primus-Seed-V2* and *Primus-Nemotron-CC* are not yet open-sourced and are currently under discussion. Feel free to reach out if you're interested. > **Disclaimer:** No Trend Micro customer information is included. --- ## About _Primus_ _Primus_ is Trend Micro's pioneering family of lightweight, state-of-the-art open cybersecurity language models and datasets. Developed through our cutting-edge research initiatives and advanced technology, these resources share the innovative foundation that powers our enterprise-class [Trend Cybertron](https://newsroom.trendmicro.com/2025-02-25-Trend-Micro-Puts-Industry-Ahead-of-Cyberattacks-with-Industrys-First-Proactive-Cybersecurity-AI) solution. As an industry leader in cybersecurity, Trend Micro is proud to contribute these powerful, efficiency-optimized models and datasets to the community, while maintaining the excellence and reliability that define our global security standards. ## Acknowledgments We would like to thank **NVIDIA** for generously providing computing resources (**Taipei-1**), which enabled the training and development of this model. ## License This model is based on the MIT license, but you must also comply with the Llama 3.1 Community License Agreement.