esa-sceva/llama3-satcom-8b

LLama3 SatCom 8B

LLama3 SatCom 8B is a fine-tuned open Large Language Model (LLM) developed under the ESA ARTES programme as part of the SatcomLLM / SCEVA (SatCom Expert Virtual Assistant) project.
It is designed to support satellite communications (SatCom) experts, engineers, and mission planners through domain-specialised reasoning, question answering, and document-based assistance.

---

Model Description

• Base model: meta-llama/Llama-3.1-8B-Instruct
• Fine-tuning type: Instruction fine-tuning (IFT)
• Training data: Domain-specific question–answer datasets (manual, synthetic, and multiple-choice)
• Architecture: Decoder-only transformer, 8 billion parameters
• Languages: English
• License: LLama-3.1 Communiy License Agreement

The model has been fine-tuned on curated SatCom-related corpora to enhance its understanding of technical language, protocols, and reasoning processes common to satellite communications, including 5G/6G non-terrestrial networks, link budget evaluation, and mission engineering tasks.

---

Training Datasets

Dataset	Description
esa-sceva/satcom-synth-qa	Synthetic QA data generated via agentic pipelines using large teacher models
esa-sceva/satcom-synth-qa-cot	Chain-of-thought annotated QA used to improve reasoning depth and factual traceability

---

Intended Use

Primary use cases:

• Technical Q&A and reasoning on SatCom systems
• Support for link budget and RF engineering questions
• Guidance for 5G/6G NTN (Non-Terrestrial Network) operations
• Mission design, planning, and anomaly detection support
• Educational and research use within the SatCom sector

Intended users:

• ESA engineers and project officers
• SatCom and aerospace researchers
• SMEs and technical operators in satellite communication
• Academic and educational users

---

Limitations

• The model does not access real-time mission data or proprietary ESA documents.
• Answers are based on training data and may require expert validation for operational use.
• It should not be relied upon for flight-critical or safety-critical decisions.
• Limited context window (base 8B configuration) may constrain long-document reasoning.

---

Technical Details

Parameter	Value
Base Model	Llama 3.1 8B Instruct
Parameters	8 billion
Context length	8k tokens
Precision	bfloat16 / fp16
Framework	Lit-GPT (Lightning AI)
Training infra	EuroHPC MareNostrum5 + AWS EC2
Optimisation	LoRA fine-tuning, cosine LR schedule

---

Evaluation

Evaluation Datasets

The model was benchmarked on both general-purpose and domain-specific QA tasks. Regarding Satcom-specific datasets:

Dataset	Subset	Description
esa-sceva/satcom-qa	Open SatCom QA	Conceptual and reasoning-based questions on SatCom workflows, regulations, and mission/system design
	Math SatCom QA	Quantitative and formula-based questions derived from system design and orbital mechanics topics
esa-sceva/satcom-mcqa	Open MCQA	Conceptual multiple-choice questions on RF systems, communication protocols, and architecture
	Math MCQA	Numerical and link-budget-focused multiple-choice questions testing applied calculations

Results

Evaluation Subset	Metric	Base Model	LLama3 SatCom 8B	Notes
Math SatCom MCQA	Accuracy	0.614	0.648	Enhanced quantitative reasoning and improved accuracy on link budget and RF propagation problems
Open SatCom MCQA	Accuracy	0.896	0.901	Increased precision on domain-specific conceptual questions in satellite communication
Math SatCom QA	LLM-as-a-judge	0.651	0.686	Stronger analytical reasoning and consistency in multi-step mathematical problem solving
Open SatCom QA	LLM-as-a-judge	0.836	0.844	Better grasp of SatCom terminology, architecture, and system-level design reasoning
MMLU	Accuracy	0.6810	0.6823	Maintains competitive general reasoning performance across diverse open-domain topics

Results indicate strong understanding of satellite-specific terminology and calculations, particularly in RF link analysis and 5G NTN configuration reasoning.

---