sCellTransformer

sCellTransformer (sCT) is a long-range foundation model designed for zero-shot prediction tasks in single-cell RNA-seq and spatial transcriptomics data. It processes raw gene expression profiles across multiple cells to predict discretized gene expression levels for unseen cells without retraining. The model can handle up to 20,000 protein-coding genes and a bag of 50 cells in the same sample. This ability (around a million-gene expressions tokens) allows it to learn cross-cell relationships and capture long-range dependencies in gene expression data, and to mitigate the sparsity typical in single-cell datasets.

sCT is trained on a large dataset of single-cell RNA-seq and finetuned on spatial transcriptomics data. Evaluation tasks include zero-shot imputation of masked gene expression, and zero-shot prediction of cell types.

Developed by: InstaDeep

Model Sources

• Repository: Nucleotide Transformer
• Paper: A long range foundation model for zero-shot predictions in single-cell and spatial transcriptomics data

How to use

Until its next release, the transformers library needs to be installed from source with the following command in order to use the models. PyTorch should also be installed.

pip install --upgrade git+https://github.com/huggingface/transformers.git
pip install torch

A small snippet of code is given here in order to infer with the model from random input.

import torch
from transformers import AutoModel

model = AutoModel.from_pretrained(
    "InstaDeepAI/sCellTransformer",
    trust_remote_code=True,
)
num_cells = model.config.num_cells
dummy_gene_expressions = torch.randint(0, 5, (1, 19968 * num_cells))
torch_output = model(dummy_gene_expressions)

A more concrete example is provided in the notebook example on one of the downstream evaluation dataset.

Training data

The model was trained following a two-step procedure: pre-training on single-cell data, then finetuning on spatial transcriptomics data. The single-cell data used for pre-training, comes from the Cellxgene Census collection datasets used to train the scGPT models. It consists of around 50 millions cells and approximately 60,000 genes. The spatial data comes from both the human breast cell atlas and the human heart atlas.

Training procedure

As detailed in the paper, the gene expressions are first binned into a pre-defined number of bins. This allows the model to better learn the distribution of the gene expressions through sparsity mitigation, noise reduction, and extreme-values handling. Then, the training objective is to predict the masked gene expressions in a cell, following a BERT-like style training.

BibTeX entry and citation info

@misc{joshi2025a,
title={A long range foundation model for zero-shot predictions in single-cell and
spatial transcriptomics data},
author={Ameya Joshi and Raphael Boige and Lee Zamparo and Ugo Tanielian and Juan Jose
Garau-Luis and Michail Chatzianastasis and Priyanka Pandey and Janik Sielemann and
Alexander Seifert and Martin Brand and Maren Lang and Karim Beguir and Thomas PIERROT},
year={2025},
url={https://openreview.net/forum?id=VdX9tL3VXH}
}