Johnyquest7/agentic_thyroid_model

Agentic Thyroid ResNet-18 — Ultrasound Nodule Malignancy Classifier

⚠️ RESEARCH USE ONLY — NOT FOR CLINICAL USE. This model is a research artifact trained on a single retrospective dataset and is validated only one external dataset. It must not be used for diagnosis, screening, or any clinical decision-making. Multiple external, prospective validation is required before any clinical consideration.

A ResNet-18 binary classifier that predicts the probability that a cropped thyroid ultrasound nodule image is malignant (positive class) vs benign. Built for a reproducible, publication-oriented experiment with proper calibration and a sensitivity-prioritized, validation-locked decision threshold.

• Backbone: timm ResNet-18, A1 ImageNet-1k recipe (resnet18.a1_in1k), full fine-tune
• Selected by: validation AUROC (14-trial sweep; winner val AUROC 0.9756)
• Calibration: temperature scaling (T = 0.5646), fit on validation
• Locked threshold: 0.7113 (highest-specificity threshold with validation sensitivity ≥ 0.95)

Cite this model

Thomas J. agentic_thyroid_model [software]. Revision e4f94ea. Hugging Face; 2026. doi:10.57967/hf/9282

Intended use

• Intended: methodological research, benchmarking, and as a baseline for thyroid ultrasound malignancy classification studies.
• Out of scope: any clinical, diagnostic, triage, or screening use; use on images acquired/preprocessed differently from the training data without re-validation; use on non-thyroid or non-ultrasound images.

Dataset

• Source: Johnyquest7/TN5000-thyroid-nodule-classification (derived from TN5000; nodule ROI cropped to 224×224 RGB PNG).
• Splits (kept strictly separate): Train 3,500 · Valid 500 · Test 1,000.
• Labels: 0 = Benign, 1 = Malignant (positive class = Malignant).
• Class balance: ~70–75% malignant in every split (mild imbalance). See data_exploration_report.md: 0 corrupt images, 0 cross-split pixel duplicates, 0 filename-ID overlaps → no detectable leakage.

Label definitions

Label	Class	Meaning
0	Benign	Non-malignant thyroid nodule
1	Malignant	Malignant thyroid nodule (positive class)

Preprocessing (locked — configs/preprocess.json)

Deterministic eval/inference path (no augmentation):

1. Resize to 224×224 (bicubic; the timm A1 data config).
2. ToTensor().
3. Normalize with ImageNet mean [0.485, 0.456, 0.406], std [0.229, 0.224, 0.225].

Grayscale ultrasound images are loaded as 3-channel RGB.

Model architecture

ResNet-18 with a single-logit binary head (num_classes=1); sigmoid → probability of malignancy. ~11.2M parameters. Trained from ImageNet-1k weights (full fine-tune).

Training procedure (final / winning config)

Setting	Value
Backbone	timm:resnet18.a1_in1k, full fine-tune
Loss	Focal loss (γ=1.0, α=0.5)
Class-imbalance handling	none beyond focal (mild imbalance; preserves calibration)
Augmentation	medical_default: hflip, mild affine (rot ≤10°, translate 5%, scale 0.9–1.1), mild brightness/contrast (±15%), light Gaussian blur (p=0.2)
Optimizer	AdamW, lr 2e-4, weight decay 1e-4
Scheduler	Cosine annealing, 2-epoch warmup
Batch size	32
Epochs	≤40, early stopping on val AUROC (patience 8)
Mixed precision	yes (fp16 autocast)
Seed	42 (strict determinism; CUBLAS_WORKSPACE_CONFIG=:4096:8, cuDNN deterministic)
Best epoch	6

Augmentations were chosen to be medically plausible for B-mode ultrasound (no vertical flip, no large rotation/shear, no aggressive crop, no color/HSV jitter), informed by MediAug (arXiv:2504.18983) and thyroid-US best practice. The augmentation ablation confirmed medical_default (0.9712–0.9756 val AUROC) outperforms both flip_only (0.9637) and medical_strong (0.9609).

Environment: torch 2.12.0+cu130, torchvision 0.27.0+cu130, timm 1.0.27, scikit-learn 1.9.0, numpy 2.4.6, NVIDIA A10G (CUDA 13.0, cuDNN 9.2). Full versions in configs/env_info.json.

Validation threshold strategy

After selecting the model by validation AUROC and calibrating on validation, the decision threshold was chosen on the validation set as the highest-specificity threshold achieving sensitivity ≥ 0.95 (clinically sensitivity-prioritized). Youden's J is reported as a secondary reference. The threshold (0.7113) was locked before the test set was touched.

• Validation @ locked threshold: sensitivity 0.952, specificity 0.896.

Calibration results

Temperature scaling (T=0.5646) on validation reduced validation ECE from 0.0833 → 0.0308 and Brier from 0.0592 → 0.0525 with AUROC unchanged (monotonic). On the test set, calibrated ECE is 0.0314 (well calibrated). Reliability diagrams: results/figures/valid_calibration.png, results/figures/test_calibration.png.

Files in this repo

final_model.pt                 # locked weights + backbone/preprocess metadata
configs/final_config.yaml      # single-command training config
configs/preprocess.json        # locked preprocessing
configs/calibration.json       # temperature scaling parameter + before/after metrics
configs/threshold.json         # locked decision threshold + selection method
configs/env_info.json          # exact package/hardware/CUDA versions
train.py  evaluate.py  evaluate_external.py  explore_data.py  sweep.py  finalize.py
thyroid_lib.py                 # shared preprocessing/model/calibration/metrics
requirements.txt
data_exploration_report.md     # full data audit (counts, leakage, intensity, grids)
LOG.md                         # chronological experiment log
results/                       # figures, tables (incl. CIs + sweep leaderboard), per-image CSVs. Not with TTA

Reproduce

pip install -r requirements.txt
python explore_data.py --dataset_id Johnyquest7/TN5000-thyroid-nodule-classification
python train.py    --config configs/final_config.yaml
python evaluate.py --split test --config configs/final_config.yaml

Evaluate a future external dataset with the same preprocessing/calibration/threshold:

python evaluate_external.py \
  --model_repo Johnyquest7/agentic_thyroid_model \
  --data_dir /path/to/external_dataset \
  --output_dir external_results
# external dataset = folder with Benign/ Malignant/ subfolders, OR add --csv labels.csv

Limitations, bias, and leakage concerns

• Single-source dataset Performance on data from other scanners, institutions, or populations is unknown and likely lower.
• Cropped-ROI inputs. The model expects nodule-cropped images like TN5000; whole-frame ultrasound will be out of distribution.
• Sensitivity gap at deployment threshold. The threshold targets ≥0.95 sensitivity on validation; on the test set sensitivity was 0.904 — i.e. the operating point does not perfectly transfer, and ~10% of malignant nodules were missed at this threshold. Threshold re-calibration on local data is advisable before any use.
• Label/selection bias. Labels and cohort composition reflect the source dataset's referral and pathology-confirmation process.
• Leakage checks were exhaustive within the available signal (exact pixel hashing + filename-ID overlap, all zero) but cannot rule out near-duplicate or same-patient-different-image leakage if such structure exists in the source. But the original article describing the dataset states that "Only one representative image among the images of the same perspective of the patient is reserved, to prevent the risk of data leakage and ensure the dataset can be partitioned at the patient level;"

⚠️ External validation required before clinical use

At present this model has only one external validation. This model requires independent, ideally prospective, multi-site external validation before any clinical consideration. It is released for research reproducibility only.

Citation

Dataset: TN5000 (Yu et al., Scientific Data, 2025). Augmentation guidance: MediAug, arXiv:2504.18983.

Generated by ML Intern

This model repository was generated by ML Intern, an agent for machine learning research and development on the Hugging Face Hub.

• Try ML Intern: https://smolagents-ml-intern.hf.space
• Source code: https://github.com/huggingface/ml-intern

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = 'Johnyquest7/agentic_thyroid_model'
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

For non-causal architectures, replace AutoModelForCausalLM with the appropriate AutoModel class.