Indonesian Multitask Text Generation and Emotion Classification

This model provides a new refresher in the field of emotion-aware dialogue systems in Indonesian by creating the Indonesian Empathetic Dialogue Dataset and conducting multitask text generation and emotion classification training using pretrained idT5

Model Details

Model Description

Developed by: Adhitia Erfina, Tran Thi Oanh, Le-Hong Phuong
Funded by: xxxxxxxxxxxxxxxxxxx
Model type: Multitask Text Generation and Emotion Classification
Language(s) (NLP): Indonesia
Finetuned from model: muchad/idt5-base

Model Sources

Repository: https://github.com/adhitia17/Multitask-Generative-Dialogue-and-Emotion-Classification-with-Indonesian-Empathetic-Dialogue-Dataset
Paper: xxxxxxxxxxxxxxxxxxx

Uses

This model is designed for multitask text-to-text generation in Indonesian, specifically trained for:

Dialogue Response Generation: Given a user utterance prefixed with dialog:, the model generates a relevant conversational response.
Emotion Classification: Given a text prefixed with emosi:, the model predicts the underlying emotion expressed in the text.
Context Understanding/Summarization (if applicable based on your training data): Given a text prefixed with konteks:, the model can perform tasks related to understanding or summarizing the provided context. It's intended to be used directly via the transformers library in Python for applications requiring these specific capabilities in Indonesian.

Direct Use

import torch
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

repo_id = "adhitia17/idmt"

print(f"Loading tokenizer and model from {repo_id}...")
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForSeq2SeqLM.from_pretrained(repo_id)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)
print(f"Model loaded to device: {device}")

def generate_response(input_text, task_prefix):
    """Generates a response from the model for a given task."""
    full_input = f"{task_prefix}: {input_text}"
    print(f"\nInput ({task_prefix}): {full_input}")
    
    input_ids = tokenizer(full_input, return_tensors="pt").input_ids.to(device)
    
    outputs = model.generate(
        input_ids,
        max_length=256, 
        num_beams=5,
        early_stopping=True
    )
    
    decoded_output = tokenizer.decode(outputs[0], skip_special_tokens=True)
    print(f"Output: {decoded_output}")
    return decoded_output


print("\nInference examples complete.")

How to Get Started with the Model

Use the code below to get started with the model. [More Information Needed]

Training Details

Training Data

Translated facebook/empathetic_dialogues to Indonesian (81.005 rows)

Training Procedure

Multitask Text Generation and Emotion Classification using T5Base

Preprocessing

facebook/empathetic_dialogues translated to Indonesian using facebook/nllb-200-1.3B

Training Hyperparameters

Learning Rate: 1e-5
Weight Decay: 0.01
Token: 512
Batch: 64
Epochs: 40
Warm Up Steps : 500
Optimizer: Adam
Evaluation Metrics : BLEU + ROUGE (text generation) & Accuracy + F1 (emotion classification)

Evaluation

Translated facebook/empathetic_dialogues to Indonesian (12.044 rows)

Testing Data & Metrics

Testing Data

Translated facebook/empathetic_dialogues to Indonesian (10.945 rows)

Results

BLEU: 0.1071
ROUGE: 0.2264
Accuracy: 0.7064
F1: 0.7049

Technical Specifications

GPU

1x NVIDIA H100 with 80 GB HBM2e memory, and FP8 Tensor Core 3.958 TFLOPS.

Training Hours

±18 hours

Citation

xxxxxxxxxxxxxxxxxxx

adhitia17
/

idmt