Update README.md

README.md

@@ -12,90 +12,18 @@ pipeline_tag: text-generation
 # Meltemi: A large foundation Language Model for the Greek language
 
 We introduce Meltemi, the first Greek Large Language Model (LLM) trained by the Institute for Language and Speech Processing at Athena Research & Innovation Center.
-Meltemi is built on top of Mistral-7b and has been trained on a large corpus which includes high-quality Greek texts. We present Meltemi-7b, along with an instruction-tuned variant, Meltemi-Instruct-7b which can be used for chat applications. Updated versions with enhanced chat and translation capabilities will be released shortly under version 2.
-Additionally, in the near future, we will also release a Mixture-of-Experts foundation model (MeltemiX-8x7b), as well as chat models based on real chats with human feedback.
+Meltemi is built on top of [Mistral-7b](https://huggingface.co/mistralai/Mistral-7B-v0.1), extending its capabilities for Greek through continual pretraining on a large corpus of high-quality and locally relevant Greek texts. We present Meltemi-7B-Instruct-v1, an instruct fine-tuned version of Meltemi-7B-v1.
 
-Meltemi is the first open Greek Foundation Language Model, available for research and commercial purposes. It is built on top of [Mistral-7b](https://mistral.ai/news/announcing-mistral-7b/), extending its capabilities for Greek through continual pretraining on a large corpus of high-quality and locally relevant Greek texts.
-The training was performed on AWS infrastructure thanks to a GRNET grant.
-We release two models trained with 8k context length: Meltemi-7b-v1 (INSERT HF LINK) and Meltemi-Instruct-7b-v1 (INSERT HF LINK) under the [Apache 2.0 License](https://github.com/apache/.github/blob/main/LICENSE).
-To assess the capabilities of Meltemi we constructed a standardized LLM evaluation suite for the Greek language, integrated with [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness).
+# Model Information
 
-
-# Introduction
-
-Large foundation models have revolutionized the AI field, opening new opportunities for research and industry applications.
-However, the development and adoption of Large Foundation Models has been mostly limited to the English language, while other digitally under-represented languages lag behind, due to the fact that most open-source foundation models having been trained with mostly English monolingual data.
-Recently, there have been efforts to extend the capabilities of open LLMs to other languages (e.g., [LeoLM](https://laion.ai/blog/leo-lm/) for German, [Aguila](https://huggingface.co/projecte-aina/aguila-7b) for Spanish, etc.).
-This movement provides alternatives to commercial, siloed solutions for the local communities, as well as more fine-grained control for the development of safe and application-optimized models.
-We develop and release Meltemi with the current status quo in mind, seeking to alleviate these limitations for the Greek language. Meltemi is developed as a bilingual model, maintaining its capabilities for the English language, while being extended to understand and generate fluent text in Modern Greek using state-of-the-art techniques.
-
-
-# Continual pretraining
-
-The original version of Mistral-7b is trained on a large corpus of English text. The corpus for the publicly released versions is estimated to contain approximately 800 billion tokens.
-We extend the pretraining of Mistral-7b with added proficiency for the Greek language, by utilizing a large corpus consisting of approximately **40 billion tokens**. 
-
-This corpus includes 28.5 billion monolingual Greek tokens, constructed from publicly available resources. Additionaly, to mitigate catastrophic forgetting and ensure that the model has bilingual capabilities, we use additional sub-corpora with monolingual English texts (10.5 billion tokens) and Greek-English parallel data (600 million tokens).
-
-This corpus has been processed, filtered, and deduplicated to ensure data quality (a detailed description of our data processing pipeline will be published in our upcoming paper) and is outlined below:
-
-<br/>
-<br/>
-Table 1: Pretraining Corpora
-
-| Sub-corpus   | # Tokens         | Percentage |
-|----------|------------------|------------|
-| Greek    | 28,555,902,360   | 72.0%      |
-| English  | 10,478,414,033   | 26.4%      |
-| Parallel | 633,816,023      | 1.6%       |
-| **Total**    | **39,668,132,416**   |  **100%**       |
-<br/>
-<br/>
-
-
-Our pretraining procedure uses insights from works which are focused on continual pretraining for adapting English models to a non-latin script language (Chinese), such as [Fast and efficient pretraining](https://arxiv.org/pdf/2304.08177.pdf) and [CollosalAI](https://huggingface.co/hpcai-tech/Colossal-LLaMA-2-7b-base).
-Our pretraining strategy consists of the following three stages:
-
-1. Vocabulary extension of the Mistral-7b tokenizer with Greek tokens
-2. Greek embedding initialization and fine-tuning on 10% of the corpus (all other model parameters are kept frozen)
-3. Continual pretraining of the whole model using the full corpus
-
-We use the following hyperparameters and training settings for the continual pretraining stage:
-
-<br/>
-
-Table 2: Training settings
-
-| Training settings |         |
-|-----------------|---------|
-| Training steps  | 25340   |
-| Warmup steps    | 253     |
-| Batch size      | 512     |
-| Context length  | 8192    |
-| Optimizer       | AdamW   |
-| Learning rate   | 2.5e-5    |
-| Learning rate decay | Cosine down to 2.5e-6 |
-| Adam β          | (0.9, 0.95) |
-| Weight decay    | 0.0    |
-| DeepSpeed       | Zero Stage-2 |
-| Precision       | BF16    |
-| GPUs       | 8  x NVIDIA H100 (80GB)  |
-| Energy footprint | 2300 kWh |
-
-<br/>
-<br/>
-
-# Supervised fine-tuning
-
-
-To create Meltemi-Instruct-7b, we utilize approximately 100k Greek instructions, which include machine-translated versions of existing single-turn and multi-turn conversation datasets. In particular, we used the following:
-
-* [Open-Platypus](https://huggingface.co/datasets/garage-bAInd/Open-Platypus) (only subsets with permissive licenses)
-* [Evol-Instruct](https://huggingface.co/datasets/WizardLM/WizardLM_evol_instruct_V2_196k)
-* [Capybara](https://huggingface.co/datasets/LDJnr/Capybara)
-* A manually created Greek dataset with multi-turn examples steering the instruction-tuned model towards safe and harmless responses
-
-The model is trained on the resulting instructions using full Supervised Fine-Tuning (SFT). Our SFT procedure is based on the [finetuning recipes](https://github.com/huggingface/alignment-handbook) provided by huggingface. We are extending and improving the instruction tuning dataset to enhance the model's chat and translation capabilities.
+- Vocabulary extension of the Mistral-7b tokenizer with Greek tokens
+- Trained with 8k context length
+- Fine-tuned with 100k Greek machine translated instructions extracted from:
+  * [Open-Platypus](https://huggingface.co/datasets/garage-bAInd/Open-Platypus) (only subsets with permissive licenses)
+  * [Evol-Instruct](https://huggingface.co/datasets/WizardLM/WizardLM_evol_instruct_V2_196k)
+  * [Capybara](https://huggingface.co/datasets/LDJnr/Capybara)
+  * A hand-crafted Greek dataset with multi-turn examples steering the instruction-tuned model towards safe and harmless responses
+- Our SFT procedure is based on the [Hugging Face finetuning recipes](https://github.com/huggingface/alignment-handbook)
 
 
 # Evaluation
@@ -107,12 +35,7 @@ Our evaluation suite includes:
 * An existing benchmark for question answering in Greek ([Belebele](https://arxiv.org/abs/2308.16884))
 * A novel benchmark created by the ILSP team for medical question answering based on the medical exams of [DOATAP](https://www.doatap.gr) ([Medical MCQA](https://huggingface.co/datasets/ilsp/medical_mcqa_greek)).
 
-Our evaluation for Meltemi-7b is performed in a few-shot setting, consistent with the settings in the [Open LLM leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard). We can see that our training enhances performance across all Greek test sets by a **+14.9%** average improvement. The results for the Greek test sets are shown in Table 3 and Figure 1:
-
-<br/>
-<br/>
-
-Table 3: Evaluation of Meltemi-7b on the Greek LLM benchmark
+Our evaluation for Meltemi-7b is performed in a few-shot setting, consistent with the settings in the [Open LLM leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard). We can see that our training enhances performance across all Greek test sets by a **+14.9%** average improvement. The results for the Greek test sets are shown in the following table:
 
 |                | Medical MCQA EL (15-shot) | Belebele EL (5-shot) | HellaSwag EL (10-shot) | ARC-Challenge EL (25-shot) | TruthfulQA MC2 EL (0-shot) | MMLU EL (5-shot) | Average |
 |----------------|----------------|-------------|--------------|------------------|-------------------|---------|---------|
@@ -120,35 +43,10 @@ Table 3: Evaluation of Meltemi-7b on the Greek LLM benchmark
 | Meltemi 7B     | 41.0%          | 63.6%       | 61.6%        | 43.2%            | 52.1%             | 47%     | 51.4%   |
 
 
-<br/>
-<br/>
-
-![Alt text](./meltemi-mistral.png)
-
-Figure 1: Comparison of Meltemi-7b and Mistral-7b on Greek test sets
-
-<br/>
-<br/>
-
-# Try it yourself
-
-You can try the released models yourself in the [following link]().
-
-# Code availability
-
-All the training and fine-tuning scripts, as well as our lm-evaluation-harness fork will be made publicly available under a permissive license. 
-
 # Acknowledgements
 
 The ILSP team utilized Amazon’s cloud computing services, which were made available via GRNET under the [OCRE Cloud framework](https://www.ocre-project.eu/), providing Amazon Web Services for the Greek Academic and Research Community. 
 
-## Contributions
- 
-* Project lead: Vassilis Katsouros
-* Data acquisition and curation: Dimitris Roussis, Leon Voukoutis, Prokopis Prokopidis, Vassilis Papavassiliou
-* Model training: Leon Voukoutis, Dimitris Roussis
-* Model evaluation: Prokopis Prokopidis, Dimitris Roussis, Leon Voukoutis
-* Infrastructure: Sokratis Sofianopoulos, George Paraskevopoulos
-* Technical supervision: Nassos Katsamanis, Stelios Piperidis, Sokratis Sofianopoulos, George Paraskevopoulos
- 
-Special thanks to Sotiris Kotitsas, Petros Stavropoulos, Dimitris Pappas, Dimitris Galanis for their input during the design and development process. Special thanks to Olga Yannoutsou for her help in the translation of one of the evaluation datasets. And special thanks as well to all members of ILSP that participated in the internal evaluation.
+# Ethical Considerations
+
+This model has not been aligned with human preferences, and therefore might generate misleading, harmful, and toxic content.