|
--- |
|
datasets: |
|
- semaj83/ctmatch |
|
language: |
|
- en |
|
metrics: |
|
- f1 |
|
pipeline_tag: text-classification |
|
tags: |
|
- medical |
|
widget: |
|
- text: "Patient is a 45-year-old man with a history of anaplastic astrocytoma of the spine complicated by severe lower extremity weakness and urinary retention s/p Foley catheter, high-dose steroids, hypertension, and chronic pain. Therapy included field radiation t10-l1 followed by 11 cycles of temozolomide 7 days on and 7 days off. This was followed by CPT-11 Weekly x4 with Avastin Q2 weeks/ 2 weeks rest and repeat cycle. [SEP] eligible ages (years): 18.0-99.0, Low-Grade Astrocytoma, Nos Histologically or cytologically confirmed low-grade astrocytoma that has progressed, recurred, or persisted after initial therapy, including radiotherapy Previously treated with at least 1 prior standard therapy (e.g., radiotherapy, chemotherapy, immunotherapy, or cytodifferentiating agent)" |
|
- text: "Patient is a 45-year-old man with a history of anaplastic astrocytoma of the spine complicated by severe lower extremity weakness and urinary retention s/p Foley catheter, high-dose steroids, hypertension, and chronic pain. Therapy included field radiation t10-l1 followed by 11 cycles of temozolomide 7 days on and 7 days off. This was followed by CPT-11 Weekly x4 with Avastin Q2 weeks/ 2 weeks rest and repeat cycle. [SEP] eligible ages (years): 21.0-80.0, Muscle Spasticity Healthy Adult patients with selective corticospinal tract dysfunction Minimum age 21 years; maximum age 80 years Moderate severity of weakness (greater than or equal to MRC Grade 4) Adult normal volunteers Severe weakness with inability to maintain voluntary contractions Significant sensory impairment For TMS studies only: pregnancy, implanted devices such as pacemakers, medication pumps or defibrillators, metal in the cranium except the mouth, intracardiac lines, history of seizures" |
|
--- |
|
|
|
# Model Card for semaj83/scibert_finetuned_ctmatch |
|
|
|
|
|
This model can be used for classifying "\<topic\> [SEP] \<clinical trial document\>" pairs into 3 classes, 0, 1, 2, or not relevant, partially relevant, and relevant. |
|
|
|
## Model Details |
|
|
|
Fine-tuned from 'allenai/scibert_scivocab_uncased' on triples of labelled topic, documents, relevance labels. |
|
These triples were processed using ctproc, collated from the openly available TREC22 Precision Medicine and CSIRO datasets here: |
|
https://huggingface.co/datasets/semaj83/ctmatch_classification |
|
|
|
### Model Description |
|
|
|
Transformer model with linear sequence classification head, trained with cross-entropy on ~30k triples and evaluated using f1. |
|
|
|
|
|
|
|
- **Developed by:** James Kelly |
|
- **Model type:** SequenceClassification |
|
- **Language(s) (NLP):** English |
|
- **License:** MIT |
|
- **Finetuned from model:** `allenai/scibert_scivocab_uncased` |
|
|
|
### Model Sources |
|
|
|
- **Repository:** https://github.com/semajyllek/ctmatch |
|
- **Paper [optional]:** [More Information Needed] |
|
|
|
## Uses |
|
|
|
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. --> |
|
|
|
### Direct Use |
|
|
|
|
|
|
|
[More Information Needed] |
|
|
|
### Downstream Use |
|
|
|
ctmatch IR pipeline for matching large set of clinical trials documents to text topic. |
|
|
|
|
|
|
|
## Bias, Risks, and Limitations |
|
|
|
Please see dataset sources for information on patient descriptions (topics), constructed by medical professionals for these datasets. |
|
No personal health information about real individuals is contained in the related dataset. |
|
Links in dataset location on hub. |
|
|
|
The claissifier model performs much better on deciding if a pair is 0 - not relevant, than differentiating between 1, partially relevant, and 2, relevant, |
|
though this is still an important clinical task. |
|
|
|
|
|
## How to Get Started with the Model |
|
|
|
Use the code below to get started with the model. |
|
|
|
``` |
|
|
|
from transformers import AutoTokenizer, AutoModelForSequenceClassification |
|
|
|
tokenizer = AutoTokenizer.from_pretrained("semaj83/scibert_finetuned_ctmatch") |
|
|
|
model = AutoModelForSequenceClassification.from_pretrained("semaj83/scibert_finetuned_ctmatch") |
|
|
|
``` |
|
|
|
## Training Details |
|
|
|
see notebook in ctmatch repo. |
|
|
|
|
|
|
|
### Training Data |
|
|
|
https://huggingface.co/datasets/semaj83/ctmatch |
|
|
|
|
|
#### Preprocessing |
|
|
|
If using ctmatch labelled dataset, using the tokenizer alone is sufficient. If using raw topic and/or clinical trial documents, |
|
you may need to use ctproc or another method to extract relevant fields and preprocess text. |
|
|
|
|
|
#### Training Hyperparameters |
|
|
|
|
|
`max_sequence_length=512 |
|
batch_size=8 |
|
padding='max_length' |
|
truncation=True |
|
learning_rate=2e-5 |
|
train_epochs=5 |
|
weight_decay=0.01 |
|
warmup_steps=500 |
|
seed=42 |
|
splits={"train":0.8, "val":0.1} |
|
use_trainer=True |
|
fp16=True |
|
early_stopping=True |
|
` |
|
|
|
|
|
## Evaluation |
|
|
|
sklearn classifier table on random test split: |
|
|
|
``` |
|
|
|
precision recall f1-score support |
|
|
|
0 0.88 0.93 0.90 5430 |
|
1 0.56 0.56 0.56 1331 |
|
2 0.65 0.49 0.56 1178 |
|
|
|
accuracy 0.80 7939 |
|
macro avg 0.70 0.66 0.67 7939 |
|
weighted avg 0.79 0.80 0.79 7939 |
|
|
|
``` |
|
|
|
|
|
## Model Card Authors |
|
|
|
James Kelly |
|
|
|
## Model Card Contact |
|
|
|
[email protected] |
