|
|
--- |
|
|
license: mit |
|
|
language: |
|
|
- en |
|
|
pipeline_tag: image-segmentation |
|
|
tags: |
|
|
- biology |
|
|
datasets: |
|
|
- HakaiInstitute/mussel-gooseneck-seg-rgb-1024-1024 |
|
|
--- |
|
|
|
|
|
# Mussel-Gooseneck-RGB: Multi-Species Segmentation Model for RGB Drone Imagery |
|
|
|
|
|
**Model Type:** ONNX Semantic Segmentation |
|
|
**Application:** Mussel and gooseneck barnacle detection in high-resolution RGB aerial imagery |
|
|
**Input:** 3-band RGB imagery (Red, Green, Blue) |
|
|
**Output:** Multi-class segmentation mask (background, mussels, gooseneck barnacles) |
|
|
|
|
|
## Model Description |
|
|
|
|
|
The Mussel-Gooseneck-RGB model is a deep learning semantic segmentation model specifically trained for detecting and differentiating between mussels and gooseneck barnacles in RGB drone imagery. This model enables detailed intertidal monitoring and research by providing species-specific segmentation maps for marine habitat assessment. |
|
|
|
|
|
**Key Features:** |
|
|
- Multi-species classification (mussels vs. gooseneck barnacles) |
|
|
- Optimized for standard RGB imagery from drones |
|
|
- ImageNet-pretrained normalization statistics |
|
|
- Efficient ONNX format for cross-platform deployment |
|
|
- Designed for intertidal zone monitoring |
|
|
|
|
|
## Model Details |
|
|
|
|
|
- **Version:** 20250725 |
|
|
- **Input Channels:** 3 (RGB) |
|
|
- **Input Size:** Dynamic tiling (recommended: 2048x2048 tiles) |
|
|
- **Normalization:** Standard (ImageNet statistics) |
|
|
- **Output:** Multi-class segmentation (0: background, 1: mussels, 2: gooseneck barnacles) |
|
|
- **Format:** ONNX |
|
|
|
|
|
### Normalization Parameters |
|
|
|
|
|
The model expects input images to be normalized using ImageNet statistics: |
|
|
|
|
|
```json |
|
|
{ |
|
|
"mean": [0.485, 0.456, 0.406], |
|
|
"std": [0.229, 0.224, 0.225], |
|
|
"max_pixel_value": 255.0 |
|
|
} |
|
|
``` |
|
|
|
|
|
## Usage |
|
|
|
|
|
### 1. Using kelp-o-matic CLI (recommended) |
|
|
|
|
|
For command-line usage: |
|
|
|
|
|
```bash |
|
|
# Install kelp-o-matic |
|
|
pip install git+https://github.com/HakaiInstitute/kelp-o-matic@dev |
|
|
|
|
|
# List available models |
|
|
kom list-models |
|
|
|
|
|
# Run segmentation on intertidal RGB drone imagery |
|
|
kom segment \ |
|
|
--model mussel-gooseneck-rgb \ |
|
|
--input /path/to/intertidal_rgb_image.tif \ |
|
|
--output /path/to/species_segmentation.tif \ |
|
|
--batch-size 8 \ |
|
|
--crop-size 2048 \ |
|
|
--blur-kernel 3 \ |
|
|
--morph-kernel 3 |
|
|
|
|
|
# Use specific model version |
|
|
kom segment \ |
|
|
--model mussel-gooseneck-rgb \ |
|
|
--version 20250725 \ |
|
|
--input image.tif \ |
|
|
--output result.tif |
|
|
|
|
|
# For high-resolution intertidal surveys |
|
|
kom segment \ |
|
|
--model mussel-gooseneck-rgb \ |
|
|
--input high_res_intertidal_survey.tif \ |
|
|
--output result.tif \ |
|
|
--batch-size 4 \ |
|
|
--crop-size 1024 |
|
|
``` |
|
|
|
|
|
### 2. Using kelp-o-matic Python API |
|
|
|
|
|
The easiest way to use this model is through the kelp-o-matic package: |
|
|
|
|
|
```python |
|
|
from kelp_o_matic import model_registry |
|
|
|
|
|
# Load the model (automatically downloads if needed) |
|
|
model = model_registry["mussel-gooseneck-rgb"] |
|
|
|
|
|
# Process a large intertidal survey image with automatic tiling |
|
|
model.process( |
|
|
input_path="path/to/your/intertidal_rgb_image.tif", |
|
|
output_path="path/to/output/species_segmentation.tif", |
|
|
batch_size=8, # Higher batch size for RGB |
|
|
crop_size=2048, |
|
|
blur_kernel_size=3, # Gentle post-processing |
|
|
morph_kernel_size=3, # Morphological operations |
|
|
) |
|
|
|
|
|
# For more control, use the predict method directly |
|
|
import rasterio |
|
|
import numpy as np |
|
|
|
|
|
with rasterio.open("intertidal_image.tif") as src: |
|
|
# Read a 2048x2048 tile (3 bands: RGB) |
|
|
tile = src.read(window=((0, 2048), (0, 2048))) # Shape: (3, 2048, 2048) |
|
|
tile = np.transpose(tile, (1, 2, 0)) # Convert to HWC |
|
|
|
|
|
# Add batch dimension and predict |
|
|
batch = np.expand_dims(tile, axis=0) # Shape: (1, 2048, 2048, 3) |
|
|
batch = np.transpose(batch, (0, 3, 1, 2)) # Convert to BCHW |
|
|
|
|
|
# Run inference (preprocessing handled automatically) |
|
|
predictions = model.predict(batch) |
|
|
|
|
|
# Post-process to get final segmentation |
|
|
segmentation = model.postprocess(predictions) |
|
|
# Result contains class labels: 0=background, 1=mussels, 2=gooseneck barnacles |
|
|
``` |
|
|
|
|
|
### 3. Direct ONNX Runtime Usage |
|
|
|
|
|
```python |
|
|
import numpy as np |
|
|
import onnxruntime as ort |
|
|
from huggingface_hub import hf_hub_download |
|
|
from PIL import Image |
|
|
|
|
|
# Download the model |
|
|
model_path = hf_hub_download(repo_id="HakaiInstitute/mussel-gooseneck-rgb", filename="model.onnx") |
|
|
|
|
|
# Load the model |
|
|
session = ort.InferenceSession(model_path) |
|
|
|
|
|
# ImageNet normalization parameters |
|
|
mean = np.array([0.485, 0.456, 0.406]) |
|
|
std = np.array([0.229, 0.224, 0.225]) |
|
|
|
|
|
# Preprocess your RGB image |
|
|
def preprocess(image): |
|
|
""" |
|
|
Preprocess RGB image for model input |
|
|
image: numpy array of shape [height, width, 3] with pixel values 0-255 |
|
|
""" |
|
|
# Normalize to 0-1 |
|
|
image = image.astype(np.float32) / 255.0 |
|
|
|
|
|
# Apply ImageNet normalization |
|
|
image = (image - mean) / std |
|
|
|
|
|
# Reshape to model input format [batch, channels, height, width] |
|
|
image = np.transpose(image, (2, 0, 1)) # HWC to CHW |
|
|
image = np.expand_dims(image, axis=0) # Add batch dimension |
|
|
|
|
|
return image |
|
|
|
|
|
# Load and preprocess image |
|
|
image = np.array(Image.open("intertidal_drone_image.jpg")) |
|
|
preprocessed = preprocess(image) |
|
|
|
|
|
# Run inference |
|
|
input_name = session.get_inputs()[0].name |
|
|
output = session.run(None, {input_name: preprocessed}) |
|
|
|
|
|
# Postprocess to get multi-class mask |
|
|
logits = output[0] |
|
|
prediction = np.argmax(logits, axis=1).squeeze(0).astype(np.uint8) |
|
|
# Result: 0=background, 1=mussels, 2=gooseneck barnacles |
|
|
``` |
|
|
|
|
|
### 4. Using HuggingFace Hub Integration |
|
|
|
|
|
```python |
|
|
from huggingface_hub import hf_hub_download |
|
|
import onnxruntime as ort |
|
|
|
|
|
# Download and load model |
|
|
model_path = hf_hub_download( |
|
|
repo_id="HakaiInstitute/mussel-gooseneck-rgb", |
|
|
filename="model.onnx", |
|
|
cache_dir="./models" |
|
|
) |
|
|
|
|
|
session = ort.InferenceSession(model_path) |
|
|
# ... continue with preprocessing and inference as above |
|
|
``` |
|
|
|
|
|
## Installation |
|
|
|
|
|
### For kelp-o-matic usage: |
|
|
|
|
|
```bash |
|
|
# Via pip |
|
|
pip install git+https://github.com/HakaiInstitute/kelp-o-matic@dev |
|
|
``` |
|
|
|
|
|
### For direct ONNX usage: |
|
|
|
|
|
```bash |
|
|
pip install onnxruntime huggingface-hub numpy pillow |
|
|
# For GPU support: |
|
|
pip install onnxruntime-gpu |
|
|
``` |
|
|
|
|
|
## Input Requirements |
|
|
|
|
|
- **Image Format:** 3-band RGB raster (JPEG, PNG, GeoTIFF) |
|
|
- **Band Order:** Red, Green, Blue |
|
|
- **Pixel Values:** Standard 8-bit (0-255 range) |
|
|
- **Spatial Resolution:** Optimized for high-resolution drone imagery (cm-level) |
|
|
- **Scene Type:** Intertidal zones with exposed mussel beds and barnacle colonies |
|
|
|
|
|
## Output Format |
|
|
|
|
|
- **Type:** Single-band raster with class labels |
|
|
- **Values:** |
|
|
- 0: Background (water, rocks, other substrates) |
|
|
- 1: Mussels (typically blue mussels, *Mytilus* spp.) |
|
|
- 2: Gooseneck barnacles (*Pollicipes* spp.) |
|
|
- **Format:** Matches input raster format and projection |
|
|
- **Spatial Resolution:** Same as input |
|
|
|
|
|
**Note:** The model outputs class probabilities, but kelp-o-matic automatically applies argmax to convert these to discrete class labels. |
|
|
|
|
|
## Performance Notes |
|
|
|
|
|
- **Dynamic Tile Size:** Supports flexible tile sizes (recommended: 2048x2048 or 1024x1024) |
|
|
- **Batch Size:** Start with 4, adjust based on available GPU memory |
|
|
|
|
|
## Large Image Processing |
|
|
|
|
|
For processing large geospatial images, the kelp-o-matic package handles: |
|
|
|
|
|
- **Automatic Tiling:** Splits large images into manageable tiles |
|
|
- **Overlap Handling:** Uses overlapping tiles to avoid edge artifacts |
|
|
- **Memory Management:** Processes tiles in batches to manage memory usage |
|
|
- **Geospatial Metadata:** Preserves coordinate reference system and geotransforms |
|
|
- **Post-processing:** Optional median filtering and morphological operations |
|
|
|
|
|
## Citation |
|
|
|
|
|
If you use this model in your research, please cite: |
|
|
|
|
|
```bibtex |
|
|
@software{Denouden_Kelp-O-Matic, |
|
|
author = {Denouden, Taylor and Reshitnyk, Luba}, |
|
|
doi = {10.5281/zenodo.7672166}, |
|
|
title = {{Kelp-O-Matic}}, |
|
|
url = {https://github.com/HakaiInstitute/kelp-o-matic} |
|
|
} |
|
|
``` |
|
|
|
|
|
## License |
|
|
|
|
|
MIT License - see the [kelp-o-matic repository](https://github.com/HakaiInstitute/kelp-o-matic/blob/main/LICENSE) for details. |
|
|
|
|
|
## Related Resources |
|
|
|
|
|
- **Documentation:** [kelp-o-matic.readthedocs.io](https://kelp-o-matic.readthedocs.io) |
|
|
- **Source Code:** [github.com/HakaiInstitute/kelp-o-matic](https://github.com/HakaiInstitute/kelp-o-matic) |
|
|
- **Other Models:** Check the [Hakai Institute HuggingFace organization](https://huggingface.co/HakaiInstitute) for additional marine segmentation models |
|
|
|
|
|
## Contact |
|
|
|
|
|
For questions or issues: |
|
|
- Open an issue on the [GitHub repository](https://github.com/HakaiInstitute/kelp-o-matic/issues) |
|
|
- Contact: [Hakai Institute](https://www.hakai.org) |
