EC-RAFT: Automated Generation of Clinical Trial Eligibility Criteria

Model Description

EC-RAFT is a fine-tuned Retrieval-Augmented Fine-Tuning (RAFT) model based on LLaMA-3.1-8B-Instruct architecture.
It is designed to automatically generate structured, high-quality clinical trial eligibility criteria (EC) directly from trial titles and descriptions.

EC-RAFT integrates domain-specific retrieval with synthesized intermediate reasoning steps, enabling it to produce clinically relevant and contextually appropriate EC sets.

Fine-tuning Details

• Original Model: LLaMA-3.1-8B-Instruct
• Datasets used for fine-tuning:
  • ClinicalTrials.gov (267,347 trials, 2000–2024) biodatlab/ec-raft-dataset
  • Retrieval corpus constructed using SciNCL model
  • Intermediate reasoning steps R generated using Gemini-1.5-flash-002
  • Fine-tuning method:
    • Retrieval-Augmented Fine-Tuning (RAFT)
    • Low-Rank Adaptation (LoRA)

Model Performance

Evaluated on a held-out ClinicalTrials.gov test split:

Metric	Score
BERTScore (semantic similarity)	86.23
Precision (LLM-guided evaluation)	78.84%
Recall (LLM-guided evaluation)	75.89%
Mean LLM-as-a-Judge Score (0–3)	1.7150
Mean Pair-BERTScore	67.76

• Outperforms zero-shot LLaMA-3.1 and Gemini-1.5-flash baselines
• Outperforms fine-tuned LLaMA and Meditron baselines
• Clinically validated: LLM-as-a-Judge scores highly correlated with human physician evaluation

Intended Use

• Assist researchers, trial designers, and sponsors in drafting clinical trial eligibility criteria.
• Automate EC generation to reduce manual effort and improve consistency.
• Support clinical trial design transparency and quality.
• Enable integration with trial registry platforms, clinical trial matching systems, and EC recommendation tools.

Limitations

• Requires human validation of generated EC before clinical use.
• Trained on public ClinicalTrials.gov data — may not generalize well to:
  • Rare or novel diseases
  • Specialized or non-standard trial designs
  • Non-public trial data
• Optimized for English-language clinical trials.
• As with any LLM-based system, risks include hallucination, subtle errors, and domain shifts.
• Evaluation metrics (BERTScore, LLM-as-a-Judge) are proxies — not full substitutes for domain expert review.

Acknowledgments

This model was developed using resources provided by:

• RAVIS Technology for feedback and collaboration.
• Faculty of Medicine Ramathibodi Hospital
• NSTDA Supercomputer Center (ThaiSC), Project #pv814001

We also acknowledge the contributions of the broader open-source community whose tools and prior works on RAFT, SciNCL, LoRA, LLaMA-3, and biomedical NLP made this project possible.