datasets:
- semaj83/ctmatch
language:
- en
metrics:
- f1
pipeline_tag: text-classification
tags:
- medical
widget:
- text: >-
Patient is a 45-year-old man with a history of anaplastic astrocytoma of
the spine complicated by severe lower extremity weakness and urinary
retention s/p Foley catheter, high-dose steroids, hypertension, and
chronic pain. Therapy included field radiation t10-l1 followed by 11
cycles of temozolomide 7 days on and 7 days off. This was followed by
CPT-11 Weekly x4 with Avastin Q2 weeks/ 2 weeks rest and repeat cycle.
[SEP] eligible ages (years): 18.0-99.0, Low-Grade Astrocytoma, Nos
Histologically or cytologically confirmed low-grade astrocytoma that has
progressed, recurred, or persisted after initial therapy, including
radiotherapy Previously treated with at least 1 prior standard therapy
(e.g., radiotherapy, chemotherapy, immunotherapy, or cytodifferentiating
agent)
- text: >-
Patient is a 45-year-old man with a history of anaplastic astrocytoma of
the spine complicated by severe lower extremity weakness and urinary
retention s/p Foley catheter, high-dose steroids, hypertension, and
chronic pain. Therapy included field radiation t10-l1 followed by 11
cycles of temozolomide 7 days on and 7 days off. This was followed by
CPT-11 Weekly x4 with Avastin Q2 weeks/ 2 weeks rest and repeat cycle.
[SEP] eligible ages (years): 21.0-80.0, Muscle Spasticity Healthy Adult
patients with selective corticospinal tract dysfunction Minimum age 21
years; maximum age 80 years Moderate severity of weakness (greater than or
equal to MRC Grade 4) Adult normal volunteers Severe weakness with
inability to maintain voluntary contractions Significant sensory
impairment For TMS studies only: pregnancy, implanted devices such as
pacemakers, medication pumps or defibrillators, metal in the cranium
except the mouth, intracardiac lines, history of seizures
Model Card for semaj83/scibert_finetuned_ctmatch
This model can be used for classifying "<topic> [SEP] <clinical trial document>" pairs into 3 classes, 0, 1, 2, or not relevant, partially relevant, and relevant.
Model Details
Fine-tuned from 'allenai/scibert_scivocab_uncased' on triples of labelled topic, documents, relevance labels. These triples were processed using ctproc, collated from the openly available TREC22 Precision Medicine and CSIRO datasets here: https://huggingface.co/datasets/semaj83/ctmatch_classification
Model Description
Transformer model with linear sequence classification head, trained with cross-entropy on ~30k triples and evaluated using f1.
- Developed by: James Kelly
- Model type: SequenceClassification
- Language(s) (NLP): English
- License: MIT
- Finetuned from model:
allenai/scibert_scivocab_uncased
Model Sources
- Repository: https://github.com/semajyllek/ctmatch
- Paper [optional]: [More Information Needed]
Uses
Direct Use
[More Information Needed]
Downstream Use
ctmatch IR pipeline for matching large set of clinical trials documents to text topic.
Bias, Risks, and Limitations
Please see dataset sources for information on patient descriptions (topics), constructed by medical professionals for these datasets. No personal health information about real individuals is contained in the related dataset. Links in dataset location on hub.
The claissifier model performs much better on deciding if a pair is 0 - not relevant, than differentiating between 1, partially relevant, and 2, relevant, though this is still an important clinical task.
How to Get Started with the Model
Use the code below to get started with the model.
from transformers import AutoTokenizer, AutoModelForSequenceClassification
tokenizer = AutoTokenizer.from_pretrained("semaj83/scibert_finetuned_ctmatch")
model = AutoModelForSequenceClassification.from_pretrained("semaj83/scibert_finetuned_ctmatch")
Training Details
see notebook in ctmatch repo.
Training Data
https://huggingface.co/datasets/semaj83/ctmatch
Preprocessing
If using ctmatch labelled dataset, using the tokenizer alone is sufficient. If using raw topic and/or clinical trial documents, you may need to use ctproc or another method to extract relevant fields and preprocess text.
Training Hyperparameters
max_sequence_length=512 batch_size=8 padding='max_length' truncation=True learning_rate=2e-5 train_epochs=5 weight_decay=0.01 warmup_steps=500 seed=42 splits={"train":0.8, "val":0.1} use_trainer=True fp16=True early_stopping=True
Evaluation
sklearn classifier table on random test split:
precision recall f1-score support
0 0.88 0.93 0.90 5430
1 0.56 0.56 0.56 1331
2 0.65 0.49 0.56 1178
accuracy 0.80 7939
macro avg 0.70 0.66 0.67 7939
weighted avg 0.79 0.80 0.79 7939
Model Card Authors
James Kelly