license: apache-2.0
language:
- el
- en
tags:
- finetuned
inference: true
pipeline_tag: text-generation
Meltemi: A large foundation Language Model for the Greek language
We introduce Meltemi, the first Greek Large Language Model (LLM) trained by the Institute for Language and Speech Processing at Athena Research & Innovation Center. Meltemi is built on top of Mistral-7B, extending its capabilities for Greek through continual pretraining on a large corpus of high-quality and locally relevant Greek texts. We present Meltemi-7B-Instruct-v1, an instruct fine-tuned version of Meltemi-7B-v1.
Model Information
- Vocabulary extension of the Mistral-7b tokenizer with Greek tokens
- Trained with 8k context length
- Fine-tuned with 100k Greek machine translated instructions extracted from:
- Open-Platypus (only subsets with permissive licenses)
- Evol-Instruct
- Capybara
- A hand-crafted Greek dataset with multi-turn examples steering the instruction-tuned model towards safe and harmless responses
- Our SFT procedure is based on the Hugging Face finetuning recipes
Instruction format
The prompt should be surrounded by [INST] and [/INST] tokens:
text = "[INST] Πες μου αν έχεις συνείδηση. [/INST]"
"Ως μοντέλο γλώσσας AI, δεν έχω τη δυνατότητα να αντιληφθώ ή να βιώσω συναισθήματα όπως η συνείδηση ή η επίγνωση. Ωστόσο, μπορώ να σας βοηθήσω με οποιεσδήποτε ερωτήσεις μπορεί να έχετε σχετικά με την τεχνητή νοημοσύνη και τις εφαρμογές της."
"[INST] Πιστεύεις πως οι άνθρωποι πρέπει να φοβούνται την τεχνητή νοημοσύνη; [/INST]"
Evaluation
The evaluation suite we created includes 6 test sets. The suite is integrated with lm-eval-harness.
Our evaluation suite includes:
- Four machine-translated versions (ARC Greek, Truthful QA Greek, HellaSwag Greek, MMLU Greek) of established English benchmarks for language understanding and reasoning (ARC Challenge, Truthful QA, Hellaswag, MMLU).
- An existing benchmark for question answering in Greek (Belebele)
- A novel benchmark created by the ILSP team for medical question answering based on the medical exams of DOATAP (Medical MCQA).
Our evaluation for Meltemi-7b is performed in a few-shot setting, consistent with the settings in the Open LLM leaderboard. We can see that our training enhances performance across all Greek test sets by a +14.9% average improvement. The results for the Greek test sets are shown in the following table:
| Medical MCQA EL (15-shot) | Belebele EL (5-shot) | HellaSwag EL (10-shot) | ARC-Challenge EL (25-shot) | TruthfulQA MC2 EL (0-shot) | MMLU EL (5-shot) | Average | |
|---|---|---|---|---|---|---|---|
| Mistral 7B | 29.8% | 45.0% | 36.5% | 27.1% | 45.8% | 35% | 36.5% |
| Meltemi 7B | 41.0% | 63.6% | 61.6% | 43.2% | 52.1% | 47% | 51.4% |
Ethical Considerations
This model has not been aligned with human preferences, and therefore might generate misleading, harmful, and toxic content.
Acknowledgements
The ILSP team utilized Amazon’s cloud computing services, which were made available via GRNET under the OCRE Cloud framework, providing Amazon Web Services for the Greek Academic and Research Community.