Hi Nicolay,
Initially, we aimed to chain together CNNs, Vision Transformers and GNNs into a unified pipeline for feature extraction across modalities. As we dug deeper, however, do we think that perhaps developing and fine-tuning a medicine-capable foundation model—one pre-trained on vast, multimodal cardiac data—would give us a far more flexible core for the Digital Heart Model?
In our current approach:
Foundation Model Pre‑training
We pre‑train an extensive, multimodal network on imaging (MRI/CT, histopathology) and ECG waveforms. This provides us with rich spatial, textural, and temporal representations in a single model.
Fine‑tuning for Clinical Tasks
From that base, we fine‑tune on specific tasks—event prediction, arrhythmia classification, segmentation—to achieve our target AUCs and Dice scores. This strategy leverages transfer learning to enhance performance, particularly when labelled clinical data are scarce.
Integration with Physics‑based Simulation
Finally, we will couple the fine-tuned AI core with a physical heart model, solving biomechanical and hemodynamic equations, to enable virtual interventions and personalised simulations.
We are finalising a detailed framework diagram and a preprint that outlines each of these steps, providing scientific rationale, training regimes, and fusion strategies. We will be publishing these diagrams and full results—including model checkpoints and code—on Hugging Face and arXiv.org in our upcoming posts. Stay tuned there for the deep dive! 🚀
Happy to dive into any specific aspect as we roll things out.
Best,
Volodymyr