Create README.md

README.md

@@ -0,0 +1,223 @@
+---
+language:
+- de
+- bg
+- cs
+- da
+- el
+- en
+- es
+- et
+- fi
+- fr
+- ga
+- hr
+- hu
+- it
+- lt
+- lv
+- mt
+- nl
+- pl
+- pt
+- ro
+- sl
+- sv
+- sk
+metrics:
+- accuracy
+- bleu
+pipeline_tag: text-generation
+library_name: transformers
+license: apache-2.0
+---
+# Model Card for Teuken-7B-base-v0.6
+
+Teuken-7B-base-v0.6 is a 7B parameter multilingual large language model (LLM) pre-trained with 6T tokens within the research project OpenGPT-X. 
+
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+- **Developed by:** Fraunhofer, Forschungszentrum Jülich, TU Dresden, DFKI
+- **Funded by:** German Federal Ministry of Economics and Climate Protection (BMWK) in the context of the OpenGPT-X project
+- **Model type:** Transformer based decoder-only model
+- **Language(s) (NLP):** bg, cs, da, de, el, en, es, et, fi, fr, ga, hr, hu, it, lt, lv, mt, nl, pl, pt, ro, sk, sl, sv
+- **Shared by:** OpenGPT-X
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+Teuken-7B-base-v0.6 is intended for commercial and research use in all official 24 European languages. Since Teuken-7B-base-v0.6
+focuses on covering all 24 EU languages, it renders more stable results across these languages and better reflects European values in its answers than English-centric models. It is therefore specialized for use in multilingual tasks.
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+The model is not intended for use in math and coding tasks.
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+Teuken-7B-base-v0.6 as a base model is not free from biases and hallucinations. It is therefore recommended to instruction tune it to fit it to the user's purposes and minimize biases and any risks arising. Finetuned models limiting risks and biases will appear soon after the release of the base model as a community effort. 
+
+## How to Get Started with the Model
+
+## Usage
+The model requires transformers, sentencepiece, and the torch library.
+After installation, here's an example of how to use the model:
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_name = "EuropeanLLM-Beta/Teuken-7B-base-v0.6"
+prompt = "Insert text here..."
+device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(model_name, trust_remote_code=True).to(device)
+inputs = tokenizer(prompt, return_tensors="pt")
+inputs = {k: v.to(device) for k, v in inputs.items()}  # Move inputs to the same device as the model
+output = model.generate(input_ids=inputs['input_ids'], max_new_tokens=1000, do_sample=True)
+result = tokenizer.decode(output.tolist())
+```
+
+This example demonstrates how to load the model and tokenizer, prepare input, generate text, and print the result.
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+Teuken-7B-base-v0.6 was pre-trained on 5.5 trillion tokens of data from publicly available sources. 
+
+The pretraining data has a cutoff of September 2023.
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+Transformer-based decoder-only model that has been trained based on the causal language modeling objective.
+
+
+#### Training Hyperparameters
+
+- **Training regime:** bf16 mixed precision <!--fp32, fp16 mixed precision, , bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+The model was evaluated in 21 languages on ARC, GSM8K, HellaSwag, TruthfulQA, Translation and MMLU. Results can be seen in the European LLM Leaderboard (https://huggingface.co/spaces/openGPT-X/european-llm-leaderboard).
+
+
+## Technical Specifications
+
+### Model Architecture and Objective
+
+| Hyper-Parameter            | Value    |
+|----------------------------|----------|
+| Training Objective         | CLM      |
+| Activation Function        | SwiGLU   |
+| Seq Length                 | 4096     |
+| Position Embeddings        | Rotary   |
+| Num Layers                 | 32       |
+| Hidden Size                | 4096     |
+| FFN Hidden Size            | 13440    |
+| Num Attention Heads        | 32       |
+| Head Dim                   | 128      |
+| Group Query Attention      | yes      |
+| Num Query Groups           | 2        |
+| Normalization              | RMSNorm  |
+| Learning rate              | 3e-4     |
+| Min learning rate          | 3e-5     |
+| Disable bias in linear     | yes      |
+| Hidden dropout             | 0.0      |
+| Attention dropout          | 0.0      |
+| Optimizer                  | AdamW    |
+| Beta1                      | 0.9      |
+| Beta2                      | 0.95     |
+| Sequence-parallelism      
+| Data-type                  | bf16     |
+| Recompute-activations      | yes      |
+| Distributed-optimizers     | yes      |
+| Model Initialization       |          |
+
+
+### Compute Infrastructure
+
+We trained our models on JUWELS Booster which consists of 936 compute nodes, each equipped with 4 NVIDIA A100 GPUs. The GPUs are hosted by AMD EPYC Rome CPUs. The compute nodes are connected with HDR-200 InfiniBand in a DragonFly+ topology. 
+
+#### Hardware
+
+The configuration of JUWELS Booster compute nodes is the following:
+
+    CPU: AMD EPYC 7402 processor; 2 sockets, 24 cores per socket, SMT-2 (total: 2×24×2 = 96 threads) in NPS-4 1 configuration
+
+    Memory: 512 GB DDR4-3200 RAM (of which at least 20 GB is taken by the system software stack, including the file system); 256 GB per socket; 8 memory channels per socket (2 channels per NUMA domain)
+
+    GPU: 4 × NVIDIA A100 Tensor Core GPU with 40 GB; connected via NVLink3 to each other
+
+    Network: 4 × Mellanox HDR200 InfiniBand ConnectX 6 (200 Gbit/s each), HCA
+
+    Periphery: CPU, GPU, and network adapter are connected via 2 PCIe Gen 4 switches with 16 PCIe lanes going to each device (CPU socket: 2×16 lanes). PCIe switches are configured in synthetic mode.
+
+#### Software
+
+https://github.com/OpenGPTX/Megatron-LM
+
+### Toxic Content
+
+This Language Model (LLM) may generate content that is inappropriate, offensive, or harmful. While the dataset has been heavily filtered to minimize such outputs,
+the model may still produce text that is biased or toxic due to the large scale and diverse nature of the data. 
+
+## Citation
+
+TODO
+
+
+**BibTeX:**
+
+TODO
+
+**APA:**
+
+TODO
+
+## Model Card Contact
+
+<div class="hf-card">
+    <h2>Contact Information</h2>
+    <p>You can reach out to the following model card contact:</p>
+    <ul>
+        <li>
+            <a href="https://huggingface.co/openGPT-X" target="_blank">OpenGPT-X</a> 
+            - <a href="mailto:[email protected]">[email protected]</a>
+        </li>
+    </ul>
+</div>
+
+# Team
+## Data Team
+Anirban Bhowmick (IAIS), Nicolo Brandizzi (IAIS), Lennard Helmer (IAIS), Benny Jörg Stein (IAIS), Karl-Heinz Sylla (IAIS), Pavel Denisov (IAIS), Qasid Saleem (IAIS), Johannes Leveling (IAIS), Hammam Abdelwahab (IAIS), Luzian Hahn (IIS), Farzad Naderi (IIS), Md Saiful Islam (IIS), Alexander Schwirjow (IIS), Pedro Ortiz Suarez (ex. DFKI), Malte Ostendorff (ex. DFKI)
+## Model-Training Team
+### Core contributors
+Mehdi Ali (IAIS), Michael Fromm (IAIS), Jan Ebert (FZJ), Chelsea John (FZJ), Lena Jurkschat (TUD), Alexander Weber (IAIS)
+### Contributors:
+Richard Rutmann (IAIS), Daniel Steinigen (IAIS), Lalith Manjunath (TUD), Carolin Penke (FZJ)
+## Evaluation Team
+### Core contributors
+Klaudia Thellmann (TUD), Alex Jude (IAIS), Jasper Buschhoff (IAIS)
+### Contributors:
+Shima Assadi (IIS), Fabio Barth (DFKI)
+## Management
+Joachim Köhler (IAIS), Nicolas Flores-Herr (IAIS), Stefan Kesselheim (FZJ), Andreas Herten (FZJ), Georg Rehm (DFKI), René Jäkel (TUD), Fabian Küch (IIS), Nicole Hildebrandt (IAIS), Ines Wendler (IAIS)