README.md

---
library_name: "tensorflow"

language:
- ary

tags:
- text-normalization
- darija
- moroccan-arabic
- character-level
- lstm
inference: false
license: apache-2.0
---
# Darija Text Normalization Model

This repository contains a **Sequence-to-Sequence LSTM** model trained to normalize Darija text.
The model converts noisy or informal Darija into a standardized format using character-level tokenization.

## Model Details

- **Architecture:** Encoder-Decoder LSTM (Sequence-to-Sequence)
- **Task:** Text Normalization
- **Language:** Darija (Moroccan Arabic)
- **Input Tokenizer:** Character-level
- **Target Tokenizer:** Character-level
- **Embedding Dimension:** 50
- **Latent Dimension (LSTM Units):** 128
- **Training Data:** Darija Open Dataset ([link](https://github.com/darija-open-dataset/dataset))
- **Saved Model Format:** Keras (.keras)
- **Tokenizers Format:** JSON (.json)
- **Parameters Format:** JSON (.json)

## Files in this Repository

- **darija-text-normalizer.keras:** The trained Keras model.
- **tokenizer_input.json:** JSON file for the input tokenizer configuration.
- **tokenizer_target.json:** JSON file for the target tokenizer configuration.
- **model_parameters.json:** Model parameters (such as max sequence lengths and vocabulary sizes).
- **config.yaml:** YAML file with complete model configuration details.
- **README.md:** This file, describing the model and its usage.

## How It Works

The model uses an Encoder-Decoder LSTM architecture. The encoder processes the input text into a context vector. The decoder then uses this context to generate normalized text one character at a time. This approach helps the model handle spelling variations and out-of-vocabulary characters.

## Example Usage

Below is an example of how to load the model and tokenizers and perform text normalization:

```python
import tensorflow as tf
from tensorflow.keras.preprocessing.text import tokenizer_from_json
import json
import numpy as np

# --- Load the Keras model ---
loaded_model = tf.keras.models.load_model("darija-text-normalizer.keras")
encoder_model = tf.keras.models.Model(loaded_model.input[0], loaded_model.layers[2].output)

# Get latent dimension from the model configuration
latent_dim_loaded = loaded_model.layers[3].get_config()['units']
decoder_state_input_h = tf.keras.layers.Input(shape=(latent_dim_loaded,))
decoder_state_input_c = tf.keras.layers.Input(shape=(latent_dim_loaded,))
decoder_states_inputs = [decoder_state_input_h, decoder_state_input_c]
decoder_embedding_layer = loaded_model.layers[2]
decoder_lstm_layer = loaded_model.layers[3]
decoder_dense_layer = loaded_model.layers[4]

decoder_embedding_inf = decoder_embedding_layer(loaded_model.input[1])
decoder_outputs_inf, state_h_inf, state_c_inf = decoder_lstm_layer(
    decoder_embedding_inf, initial_state=decoder_states_inputs
)
decoder_states_inf = [state_h_inf, state_c_inf]
decoder_outputs_inf = decoder_dense_layer(decoder_outputs_inf)
decoder_model = tf.keras.models.Model(
    [loaded_model.input[1]] + decoder_states_inputs,
    [decoder_outputs_inf] + decoder_states_inf
)

# --- Load Tokenizers ---
with open("tokenizer_input.json", 'r', encoding='utf-8') as f:
    tokenizer_input_config = json.load(f)
tokenizer_input = tokenizer_from_json(tokenizer_input_config)

with open("tokenizer_target.json", 'r', encoding='utf-8') as f:
    tokenizer_target_config = json.load(f)
tokenizer_target = tokenizer_from_json(tokenizer_target_config)

# --- Load Model Parameters ---
with open("model_parameters.json", 'r', encoding='utf-8') as f:
    model_params = json.load(f)
    max_input_len = model_params['max_input_len']
    max_target_len = model_params['max_target_len']

def normalize_text(input_text, encoder_model, decoder_model, input_tokenizer, target_tokenizer, max_target_len, max_input_len):
    """Normalizes input Darija text using the trained encoder-decoder model."""
    input_seq = input_tokenizer.texts_to_sequences([input_text])
    padded_input_seq = tf.keras.preprocessing.sequence.pad_sequences(input_seq, maxlen=max_input_len, padding='post')
    states_value = encoder_model.predict(padded_input_seq, verbose=0)

    target_seq = [target_tokenizer.word_index.get(target_tokenizer.oov_token, 0)]
    if target_seq[0] is None:
        target_seq = [0]
    target_seq = np.array(target_seq).reshape(1, 1)

    decoded_sentence = ''
    stop_condition = False
    while not stop_condition:
        output_tokens, h, c = decoder_model.predict([target_seq] + states_value, verbose=0)
        sampled_token_index = np.argmax(output_tokens[0, -1, :])
        sampled_char = target_tokenizer.index_word.get(sampled_token_index, '')

        if sampled_char and sampled_char != target_tokenizer.oov_token:
            decoded_sentence += sampled_char

        if sampled_char == '' or sampled_char == target_tokenizer.oov_token or len(decoded_sentence) > max_target_len:
            stop_condition = True

        target_seq = np.array([sampled_token_index]).reshape(1, 1)
        states_value = [h, c]

    return decoded_sentence

# --- Example ---
input_text = "kn-mchiw l-sou9"  # Example input (Darija for "We are going to the market")
print("Input text:", input_text)
print("Normalized text:", normalize_text(input_text, encoder_model, decoder_model, tokenizer_input, tokenizer_target, max_target_len, max_input_len))
```

## Installation Requirements

Make sure to install the following packages:

```bash
pip install tensorflow numpy pyyaml huggingface_hub
```

**Before running this script, please follow these steps to resolve the Hugging Face Hub push error:**

1.  **Get your Hugging Face API Token:**
    *   Go to [https://huggingface.co/settings/tokens](https://huggingface.co/settings/tokens) and create a new token with "write" role.
    *   **Replace `HF_API_TOKEN = "YOUR_HF_API_TOKEN"` in the code with your actual token.**

2.  **Enter your Hugging Face Username:**
    *   **Replace `HF_USERNAME = "YOUR_HF_USERNAME"` in the code with your actual Hugging Face username.** This is the username you use to log in to Hugging Face.

3.  **(Recommended) Login to Hugging Face CLI:**
    *   Open your terminal or command prompt.
    *   Run this command: `huggingface-cli login`
    *   Enter your Hugging Face API token when prompted. This securely configures Git to authenticate with Hugging Face.

4.  **Check your Internet Connection:**
    *   Ensure you have a stable internet connection.

**After completing these steps, run the Python script again.**

If you still encounter issues, double-check:

*   That you have correctly installed all required libraries (`tensorflow`, `numpy`, `pyyaml`, `huggingface_hub`).
*   That your API token is valid and has "write" permissions.
*   That your Hugging Face username is correctly entered.

If the error persists, it might be a more temporary network issue or a problem on the Hugging Face Hub side, although the above steps should resolve the most common causes of this error.