๐ง Llama-3.1-KokoroChat-ScorePrediction: Japanese Counseling Dialogue Scoring Model
Llama-3.1-KokoroChat-ScorePrediction is a large-scale Japanese language model fine-tuned on the KokoroChat datasetโa collection of over 6,000 psychological counseling dialogues conducted via role-play between trained counselors.
Unlike response-generation models, this version is trained to predict client feedback scores based on the full counseling dialogue history.
๐ก Overview
- โ Task: Predict the overall counseling quality score as rated by the client
- โ Dataset: 6,589 dialogues with feedback scores between 0 and 100
- โ Data source: Text-based role-play by trained counselors
- โ
Base Model:
tokyotech-llm/Llama-3.1-Swallow-8B-Instruct-v0.3
โ๏ธ Usage Example
# -*- coding: utf-8 -*-
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "UEC-InabaLab/Llama-3.1-KokoroChat-ScorePrediction"
# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")
# Set pad_token_id
if tokenizer.pad_token_id is None:
tokenizer.pad_token = "[PAD]"
tokenizer.pad_token_id = tokenizer.convert_tokens_to_ids("[PAD]")
model.config.pad_token_id = tokenizer.pad_token_id
# Build dialogue input
messages = [
{"role": "system", "content": "ๅฟ็ใซใฆใณใปใชใณใฐใฎไผ่ฉฑใซใคใใฆใ็นๆฐใใคใใฆใใ ใใใ"},
{"role": "user", "content": "ใซใฆใณใปใฉใผ: ๏ผใใใซใกใฏใใใใใใ้กใใใใใพใใๆ้ใซใชใใพใใใใฏใใใใใฆใใใ ใใพใใ๏ผใ\nใฏใฉใคใขใณใ: ๏ผใใใซใกใฏใใใใใใ้กใใใใใพใใ๏ผใ\nใซใฆใณใปใฉใผ: ใใใซใกใฏใ็ธ่ซๅกใงใใไปๆฅใฏใ็ธ่ซใซๆฅใฆใใ ใใฃใฆใใใใใจใใใใใพใใใฏใใใซใๅนด้ฝขใจๆงๅฅใ็ธ่ซๅ
ๅฎนใๆใใฆใใ ใใใ\nใฏใฉใคใขใณใ: 40ไปฃ.ๅฅณๆงใไปไบใ่พใใใใใใใใใใใจใใใใใชใใ\nใซใฆใณใปใฉใผ: ใใใชใใงใใญใใใใฏใๅฐใใงใใญใไปไบใ่พใใใใฆใๆฌกใซไฝใใใใใๆขใใใฆใใใใจใใใใจใงใใใใใ\nใฏใฉใคใขใณใ: ใฏใใใใใงใใ\nใซใฆใณใปใฉใผ: ใใใชใใงใใญใไปใฎใไปไบใ่พใใใใจๆใใใใใฃใใใฏใชใใงใใใใใ\nใฏใฉใคใขใณใ: ่ทๅ ดใฎใใฏใใฉใๅซใง่พใใใใงใใ\nใซใฆใณใปใฉใผ: ใใใชใใงใใญใใใใฏๅซใชๆฐๆใกใซใชใใพใใญ่พใใใใชใใฎใ่ช็ถใชใใจใ ใจๆใใพใใฉใใชใใจใใใฃใใฎใใใ่ฉฑใใงใใ็ฏๅฒใงๆใใฆใใใ ใใพใใใ\nใฏใฉใคใขใณใ: ไธๅธใซไฟก้ ผใใฆใชใใจ่จใใใใใใใใซๅๆใใใๅ
่ผฉใใใคใใ็งใฎไปไบใฎ้ใซๅ
ฅใฃใฆใใพใใ\nใซใฆใณใปใฉใผ: ใใใชใใงใใญใใใใฏๅทใคใใพใใญไธๅธใๅๅใฎๆนใฏใๅใๆนใงใใใใใ\nใฏใฉใคใขใณใ: ใใใ้ใใพใใไธๅธใฏไธไบบใงใๅใไพกๅค่ฆณใใใฃใๅ
่ผฉใฎๅๅใ๏ผไบบใใพใใ\nใซใฆใณใปใฉใผ: ใใใชใใงใใญใ่ชใใฆใใใใชใใฎใฏๆฌๅฝใซใใคใใใใจใ ใจๆใใพใไปใฎ็ถๆ
ใฏใฉใฎใใใ็ถใใฆใใใฎใงใใใใใ\nใฏใฉใคใขใณใ: ๅ
ฅ็คพใใ10ๅนดใใใใฏ็ตใกใพใใญใ\nใซใฆใณใปใฉใผ: ใใใชใใงใใญใใจใฆใ้ทใๆ้่ทๅ ดใซ่ฒข็ฎใใใใฎใ ใจๆใใพใใใฎ้ใใใฏใใฉใฏใใฃใจ็ถใใฆใใใฎใงใใใใใ\nใฏใฉใคใขใณใ: ไปๆใใจใใใใงใใญใ\nใซใฆใณใปใฉใผ: ใใใชใใงใใญใใจใฆใ้้
ทใช็ฐๅขใ ใจๆใใพใใใฎ็ฐๅขใฎไธญใงใใ่ช่บซใฎๅฝนๅฒใๆใใใใจใใใฆใใใฎใฏใใจใฆใ็ซๆดพใงใไปไบใฎ้ใซๅ
ฅใใใใจใไธไบบๅใฎๆฑใใงใฏใชใใใใง่
นใ็ซใคใจๆใใพใใใใใใใงใใใใใ\nใฏใฉใคใขใณใ: ๆๅใฏ็งใๆ
ฃใใชใใฆใในใฐใใใ ใฃใใฎใงไปๆนใชใใชใจๆใฃใฆใใพใใใใใใๆใใใฆใใใใจใซ้ๅๆใๆใใใใใซใชใใใใใชใใใใใซๅฑ
ใฆใไปๆนใชใใชใจๆใใ่ซฆใใใใใซใชใใ่ปข่ทใๆใใใใใซใชใใพใใใ่จๆญฃ่ปข่ทใ่ใใใใใซใชใใพใใใใงใใใ\nใซใฆใณใปใฉใผ: ่จๆญฃใใใใจใใใใใพใใใใชใใงใใญใใไปไบใซๅฏพใใใ็ๆฏใงไธๅฏงใชๆงๅญใไผใใฃใฆใใพใ้ใซๅ
ฅใใใใใจใใฃใใใฃใฆใใพใใใใๅ
ทไฝ็ใซใฏใฉใใชใใจใงใใใใใ\nใฏใฉใคใขใณใ: ๆฅญ่
ใซใ้ใๆฏๆใๆใซๅไปใฎ็งใซใ้ใ้ ใใฆใใใใใใใฎใซใใพใใๅ
่ผฉใซใ้ใ้ ใใฆใ็ขบ่ชใใใฆใใใ็งใซๅฑใใใใใใคใฌๆ้คใใฆใใฎใซใ็ฎใฎๅใงใใพใใใใคใฌๆ้คใใใใพใใ่ใใใๆใงใๆชๅฃใ้ฐๅฃใ่จใใใใใใใพใใ\nใซใฆใณใปใฉใผ: ใใใชใใงใใญใใใใๆๅฐๆชใซๆใใพใโฆ่ใใใใจใใใงๆชๅฃใ้ฐๅฃใๅซใชๆฐๆใกใซใชใใพใใญใใใใชๆใฏใฉใใใฆใใใฎใงใใใใใ\nใฏใฉใคใขใณใ: ใฏใใๆชๅฃใ้ฐๅฃใฏ่ชๅใฎใใจใใใชใใฆใใ่ชๅใฎใใจ่จใใใฆใใใใใซๅใฏๆใฃใฆใใพใฃใใใๅฃฐใใใใฎใๅซใงใใใ\nใซใฆใณใปใฉใผ: ใใใๆฌๅฝใซๅซใชๆฐๆใกใซใชใใพใใญใใใใฃใใใจใใ็คพๅ
ใซ็ธ่ซใๅ ฑๅใงใใๆนใฏใใใฃใใใใพใใใ\nใฏใฉใคใขใณใ: ๆ่ฟใฏใใพใๅงใพใฃใใจๆใใใใใซใชใใใใใใชใใใใซใใใใจ็ฎใฎๅใฎๆฃ่
ใใใจๅคงใใชๅฃฐใง่ฉฑใใใใซใใใจๆฅฝใซใชใใพใใใไธๅธใ1ไบบใใใใชใใฎใงใๅใๅบใๅ ดๆใฏใชใใงใใๅฐใใชใฏใชใใใฏใชใฎใงใ้ฃใใใงใใ\nใซใฆใณใปใฉใผ: ใกใใปใผใธใ้ใชใฃใฆใใพใฃใฆใใฟใพใใใใ่ช่บซใงๅฏพ็ญใใจใฃใฆใใฃใใใใฎใงใใญ่ใใใชใใจใใใใจใงๆฅฝใซใชใใใฎใฏใ็ด ๆตใช็บ่ฆใ ใจๆใใพใใ\nใฏใฉใคใขใณใ: ใฟใใช็ฅใฃใฆใใฆใใๆ้ปใฎไบ่งฃใงใในใซใผใงใใญใใใใใจใใใใใพใใ\nใซใฆใณใปใฉใผ: ใชใใปใฉใๅฐใใชใฏใชใใใฏใ ใจใใใฃใใใ้ใ้ฃใใใงใใญ่พใใใใๆฐๆใกใใจใฆใไผใใฃใฆใใพใใ่พใใๅพไฝใใใใใใใใใใชใใใจใใใใจใงใใใๆฌกใฎไปไบใ่ฆใคใใฆใใ่พใใใใจใ่ใใงใใใใใ\nใฏใฉใคใขใณใ: ่พใใใพใงใซๅผใ็ถใใๅๅนดใปใฉๅฟ
่ฆใชใฎใงใใใใใใใใใจๆใใใใๅใใ ใใใใจๆใฃใฆใพใใ\nใซใฆใณใปใฉใผ: ใใใชใใงใใญใๅๅนดใจใใใฎใฏ้ทใๆใใฆใใพใใพใใใใใฎ็นใฏใฉใฎใใใซใ่ใใงใใใ\nใฏใฉใคใขใณใ: ็งใๅคใใฃใฆใใชใจๆใใพใใใใ็งใฏๅไป1ไบบใงใใฃใฆใใใฎใงใๆ
ฃใใใพใงใซๆ้ใใใใใฎใงใใใใไปๆนใชใใจ่ซฆใใฆใใพใใ\nใซใฆใณใปใฉใผ: ใใใชใใงใใญใ็็ฑใใใฃใฆๅใๅ
ฅใใฆใใฎใงใใญใใใพใงใ่ฉฑใ่ใใฆใใฆใ้้
ทใช่ทๅ ด็ฐๅขใงใ็ๆฏใซไธๅฏงใซใไปไบใใใใฆใใฆใ่ฒฌไปปๆใฎๅผทใๆนใชใฎใ ใชใจๆใใฆใใพใ่ฒใ
ใ่ฉฑใใใ ใใฃใไธญใงใไปๆฅไธ็ชใ่ฉฑใซใชใใใใใจใฏใชใใงใใใใใ\nใฏใฉใคใขใณใ: ๆฌกใซใใใใใใจใใฟใคใใใใ\nใซใฆใณใปใฉใผ: ใใใชใใงใใญใๆฌกใฎ่ทๅ ดใซ็งปใใใใซใใไปๆฅใฏไธ็ทใซ่ใใใใฆใใ ใใใญใ\nใฏใฉใคใขใณใ: ใ้กใใใใใพใใ\nใซใฆใณใปใฉใผ: ใใกใใใใใใใใใ้กใใใใใพใใใใใใใจใใจใฏใฉใใชใใจใงใใใใๅฅฝใใชใใจใใ็ตฆไธๅพ
้ใใใใใจใชใฉใๅธๆใฏใใใใงใใใ\nใฏใฉใคใขใณใ: ใใซใฟใคใ ใงใชใใฆใๅฎๅฎใใฆใใฆใใใใใใใใฆใใใใใชใจๆใใพใใ\nใซใฆใณใปใฉใผ: ใใใชใใงใใญใๅฎๅฎใจใใใฎใฏๅบๅฎ็ตฆใงใใจใใๆๅณๅใใงใใใใใ\nใฏใฉใคใขใณใ: ใใใงใใญใไปใซไฝใใใใพใใ๏ผ\nใซใฆใณใปใฉใผ: ๆๆใใใฉในๆญฉๅๅถใฎใใใช็ตฆไธๅฝขๆ
ใใใใใใ็ขบ่ชใใใฆใใใ ใใพใใๅบๅฎ็ตฆใฏๆๅใใใฏใฃใใใใใใฎใงๅฎๅฟใงใใพใใญใ\nใฏใฉใคใขใณใ: ใใใงใใญใ\nใซใฆใณใปใฉใผ: ใใไธ็นใใใใใใฏใฉใใชใใจใซๆใใใใฆใใใพใใงใใใใใ\nใฏใฉใคใขใณใ: ๆ่ฌใใใๆใจใใงใใใญใ่ชๅใๆฅฝใใใจๆใใใใ\nใซใฆใณใปใฉใผ: ๅคงไบใชใใจใงใใญๆ่ฌใใใใใๆฅฝใใใจๆใใใใไปไบใซใคใใใจใใขใใใผใทใงใณใใใใใใใงใใญใ\nใฏใฉใคใขใณใ: ใใใงใใญใใขใใใผใทใงใณ้ซใใ็ใ็ใใจๅใใใใงใใ\nใซใฆใณใปใฉใผ: ใใใงใใญใ็ด ๆตใชๅใๆนใ ใจๆใใพใใไปไบใงๆฅฝใใใจๆใใใฎใฏใฉใใชๆใงใใใใใ\nใฏใฉใคใขใณใ: ๆฃ่
ใใใ่ฉฑใใฆไธใใ่ฉฑใใใใฆใๆใๆฅฝใใใจๆใใพใใ\nใซใฆใณใปใฉใผ: ใใใชใใงใใญใใใฎใใใซใๆใใซใชใใฎใงใใญไปใพใงใฎใ่ฉฑใใใใใฃใฆใใใจใใ่ช่บซใฎไธญใงใใใฎใใใซๅใใใใใจใใๅธๆใฏใใใชใใฏใฃใใใใฆใใใใใซๆใใพใใใใใใใใงใใใใใ\nใฏใฉใคใขใณใ: ๅฝขใซใใใฎใใ้ฃใใๆใใพใใ\nใซใฆใณใปใฉใผ: ใใใชใใงใใญใๅฝขใซใใใฎใ้ฃใใใจใๆใใชใใงใใญใใใใใใฐใไปไธใใฆใใใ ใใใๅธๆใใใฃใฆใใใญใผใฏใผใฏใซ่กใฃใใใใใใใงๆขใใฆใฟใใฎใฏใใใใงใใใใใ\nใฏใฉใคใขใณใ: ใฏใใใใใใฆใฟใพใใใใใใจใใใใใพใใใ\nใซใฆใณใปใฉใผ: ใฏใใใใฒใใฃใฆใฟใฆใใ ใใๅ
ใปใฉใใฃใใใฃใฆใใใใขใใใผใทใงใณ้ซใใ้ซใใ็ใ็ใๅใใใใชๅ ดๆใ่ฆใคใใใใจใใๅฟๆดใใฆใใใพใใ"}
]
# Tokenize with chat template
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt"
).to(model.device)
attention_mask = inputs.ne(tokenizer.pad_token_id)
# Generate score prediction
outputs = model.generate(
inputs,
attention_mask=attention_mask,
pad_token_id=tokenizer.pad_token_id,
max_new_tokens=256
)
# Extract only the newly generated tokens
response = outputs[0][inputs.shape[-1]:]
predicted_score = tokenizer.decode(response, skip_special_tokens=True)
# Print clean score prediction
print(predicted_score)
๐ ๏ธ Fine-Tuning Details
Fine-tuning was performed using QLoRA with the following configuration:
- Quantization: 4-bit NF4 with bfloat16 computation
- LoRA target modules:
q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj - LoRA parameters:
r = 8lora_alpha = 16lora_dropout = 0.05
Dataset Split
- Training/Validation/Test ratio: 8:1:1
Hyperparameter Settings
- Optimizer:
adamw_torch_fused - Warm-up Steps:
100 - Learning Rate:
2e-4 - Epochs:
4 - Batch Size:
4
๐ Citation
If you use this model or dataset, please cite the following paper:
@inproceedings{qi2025kokorochat,
title = {KokoroChat: A Japanese Psychological Counseling Dialogue Dataset Collected via Role-Playing by Trained Counselors},
author = {Zhiyang Qi and Takumasa Kaneko and Keiko Takamizo and Mariko Ukiyo and Michimasa Inaba},
booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics},
year = {2025},
url = {https://github.com/UEC-InabaLab/KokoroChat}
}
๐ Related
- ๐ Dataset:
- ๐ค Response Generation Models:
- Llama-3.1-KokoroChat-Low: fine-tuned on 3,870 dialogues with client feedback scores < 70
- Llama-3.1-KokoroChat-High: fine-tuned on 2,601 dialogues with client feedback scores between 70 and 98
- Llama-3.1-KokoroChat-Full: fine-tuned on 6,471 dialogues with client feedback scores โค 98
- ๐ Paper: ACL 2025 Paper
- Downloads last month
- 16
Inference Providers
NEW
This model isn't deployed by any Inference Provider.
๐
Ask for provider support