README.md

---
library_name: transformers
license: apache-2.0
datasets:
- ivrit-ai/crowd-transcribe-v5
- ivrit-ai/crowd-recital-whisper-training
- ivrit-ai/knesset-plenums-whisper-training
language:
- he
metrics:
- wer
base_model:
- openai/whisper-large-v3-turbo
pipeline_tag: automatic-speech-recognition
---

# Model Card for Model ID

This model is a Hebrew finetune (continued training) of the OpenAI Whisper Large v3 Turbo model.


## Model Details

### Model Description

- **Developed by:** ivrit-ai
- **Language(s) (NLP):** Hebrew
- **License:** Apache-2.0
- **Finetuned from model** openai/whisper-large-v3-turbo

## Bias, Risks, and Limitations

Language detection capability of this model has been degraded during training - it is intended for mostly-hebrew audio transcription.
Language token should be explicitly set to Hebrew.

Additionally, the tanslation task was not trained and also degraded. This model would not be able to translate in any reasonable capacity.

## How to Get Started with the Model

Please follow the original [model card](https://huggingface.co/openai/whisper-large-v3-turbo#usage) for usage details - replacing with this model name.
You can also fine other weight formats ad quantizations on the [ivrit ai](https://huggingface.co/ivrit-ai) HF page.

We created some simple example scripts using this model and weights for other indference runtimes.
Find those in the ["examples"](https://github.com/ivrit-ai/asr-training/tree/master/examples) folder within the training GitHub repo.

## Training Details

### Training Data

This model was trained on the following datasets:

- [ivrit-ai/crowd-transcribe-v5](https://huggingface.co/datasets/ivrit-ai/crowd-transcribe-v5) - Publicly accessible audio sources have beem crowd-transcribed segment-by-segment - ~300h
- [ivrit-ai/crowd-recital-whisper-training](https://huggingface.co/datasets/ivrit-ai/crowd-recital-whisper-training) - Crowd-sourced recording of Wikipedia atricle snippets. ~50h
- [ivrit-ai/knesset-plenums-whisper-training](https://huggingface.co/datasets/ivrit-ai/knesset-plenums-whisper-training) - A subset of a Knesset (Israeli house of representitives) plenum protocols. ~325h

### Training Procedure

This model is a weighted-average of the lowest eval loss checkpoints (From around the end of epoch 2) from two seprate runs with the same setup.
Training code can be found on the ivrit-ai Github [here](https://github.com/ivrit-ai/asr-training)

#### Preprocessing

The "Crowd Recital" and "Knesset" datasets contain timestamps and previous text following the Whisper expected inputs.
Timestamps were used from 40% of samples from those datasets, and 50% of the previous text was used.

The "Crowd Transcribe" datasets has no timestamps or previous text and this preprocessing only included melspec feature extraction and text encoding.

Preprocessing code can be found within the training code [repository](https://github.com/ivrit-ai/asr-training).

Datasets were interleaved with 0.15:0.8:0.05 ratio (knesset:crowd-transcribe:crowd-recital).

#### Training Hyperparameters

- **Training regime:** bf16 mixed precision with sdpa
- **Learning Rate:** 1e-5, Linear decay, 800 steps warmup for 3 epochs
- **Batch Size:** 32

#### Training Hardward / Duration

- **GPU Type:** 8 x Nvidia A40 machine
- **Duration:** ~9h run, stopped at 3 epochs

## Evaluation

Please refer to the [ivrit-ai/hebrew-transcription-leaderboard](https://huggingface.co/spaces/ivrit-ai/hebrew-transcription-leaderboard)