Convolutional Set Transformer

Highlights:

• CST is a novel deep learning architecture for processing image sets of arbitrary cardinality that are visually heterogeneous yet share high-level semantics (e.g., a common category, scene, or concept).
• CST is general-purpose and supports a broad range of applications, including set-based classification tasks and Set Anomaly Detection.
• In the domain of image-set processing, CST outperforms existing set-learning approaches such as Deep Sets and Set Transformer. Unlike these methods, which are inherently opaque, CST is fully compatible with standard CNN explainability tools, including Grad-CAM.
• While Deep Sets and Set Transformer are typically trained from scratch, CST supports Transfer Learning: it can be pre-trained on large-scale datasets and then effectively adapted to diverse downstream tasks. We publicly release CST-15, the first set-learning backbone pre-trained on ImageNet.

Code and Pre-trained Models:
We release the cstmodels Python package (pip install cstmodels) which provides reusable Keras 3 layers for building CST architectures, and an easy interface to load CST-15 pre-trained on ImageNet in just two lines of code:

from cstmodels import CST15
model = CST15(pretrained=True)

Documentation is available here. Tutorial notebooks are provided in the GitHub repo of the project.

Example Application of the CST:
Set Anomaly Detection is a binary classification task meant to identify images in a set that are anomalous or inconsistent with the majority of the set. Here, the notion of anomaly is context-dependent: the same image may be considered anomalous in one set but not in another, depending on the surrounding context. The Figure below shows two image sets derived from the CelebA dataset (Liu et al., 2015). In each set, a majority of normal images share two attributes ("wearing hat" and "smiling" in the first, "no beard" and "attractive" in the second), while a minority lack these attributes and are thus anomalous. After training a CST and a Set Transformer (Lee et al., 2019) on CelebA for Set Anomaly Detection, we evaluate the explainability of their predictions by overlaying Grad-CAMs on anomalous images. CST explanations correctly highlight the anomalous regions, whereas Set Transformer (ST) explanations fail to provide meaningful insights. Want to dive deeper? Check out our paper!