Create README.md

README.md

@@ -0,0 +1,119 @@
+---
+license: apache-2.0
+language:
+- en
+base_model:
+- baidu/ERNIE-4.5-300B-A47B-Base-PT
+---
+# ERNIE-4.5-300B-A47B-Base
+## ERNIE 4.5 Highlights
+The advanced capabilities of the ERNIE 4.5 models, particularly the MoE-based A47B and A3B series, are underpinned by several key technical innovations:
+- **Multimodal MoE Pretraining:** Our models are jointly trained on both textual and visual modalities to better capture the nuances of multimodal information and improve performance on tasks involving text generation, image understanding, and cross-modal reasoning. To achieve this without one modality hindering the learning of another, we designed a heterogeneous MoE structure, incorporated three-dimensional rotary embeddings, and employed router orthogonal loss and multimodal token-balanced loss. These architectural choices ensure that both modalities are effectively represented, allowing for mutual reinforcement during training.
+- **Scaling-Efficient Architecture and Infrastructure:** To train the large multimodal MoE models efficiently, we introduce a novel heterogeneous hybrid parallelism and multi-level load balancing strategy for efficient training of ERNIE 4.5 models. By using on-device expert parallelism, memory-efficient pipeline scheduling, and FP8 mixed precision, we achieve ideal pre-training performance. For inference, we propose a quantization method with collaborative parallelism among multiple experts to achieve lossless quantization. Built on PaddlePaddle, ERNIE 4.5 delivers high-performance inference across a wide range of hardware platforms.
+- **Modality-Specific Post-training:** To meet the diverse requirements of real-world applications, we fine-tuned variants of the pretrained model for specific modalities. Our LLMs are optimized for general-purpose language understanding and generation. The VLMs focuses on visual-language understanding and supports both thinking and no-thinking mode. Each model employed a combination of Supervised Fine-tuning (SFT), Direct Preference Optimization (DPO) or a modified reinforcement learning method named Unified Preference Optimization (UPO) for post-training, using targeted datasets aligned with its intended usage scenario.
+
+To ensure the stability of multimodal joint training, we adopt a staged training strategy. In the first and second stage, we train only the text-related parameters, enabling the model to develop strong fundamental language understanding as well as long-text processing capabilities. The final multimodal stage extends capabilities to images and videos by introducing additional parameters including a ViT for image feature extraction, an adapter for feature transformation, and visual experts for multimodal understanding. At this stage, text and visual modalities mutually enhance each other. After pretraining trillions tokens, we extracted the text-related parameters and finally obtained ERNIE-4.5-300B-A47B-Base。
+
+
+## Model Overview
+ERNIE-4.5-300B-A47B-Base is a text MoE Base model, with 300B total parameters and 47B activated parameters for each token. The following are the model configuration details: 
+
+| Key | Value |
+| --- | --- |
+| Modality | Text |
+| Training Stage | Pretraining |
+| Params(Total / Activated) | 300B / 47B |
+| Layers | 54 |
+| Heads(Q/KV) | 64 / 8 |
+| Text Experts(Total / Activated) | 64 / 8 |
+| Vision Experts(Total / Activated) | 64 / 8 |
+| Context Length | 131072 |
+
+
+## Quickstart
+### Model Finetuning with ERNIEKit
+
+[ERNIEKit](https://github.com/PaddlePaddle/ERNIE) is a training toolkit based on PaddlePaddle, specifically designed for the ERNIE series of open-source large models. It provides comprehensive support for scenarios such as instruction fine-tuning (SFT, LoRA) and alignment training (DPO), ensuring optimal performance.
+
+Usage Examples:
+```bash
+# SFT
+erniekit train --stage SFT --model_name_or_path baidu/ERNIE-4.5-300B-A47B-Base-Paddle --train_dataset_path your_dataset_path
+# DPO
+erniekit train --stage DPO --model_name_or_path baidu/ERNIE-4.5-300B-A47B-Base-Paddle --train_dataset_path your_dataset_path
+```
+For more detailed examples, including SFT with LoRA, multi-GPU configurations, and advanced scripts, please refer to the examples folder within the [ERNIEKit](https://github.com/PaddlePaddle/ERNIE) repository.
+
+### Using FastDeploy
+
+Service deployment can be quickly completed using FastDeploy in the following command. For more detailed usage instructions, please refer to the [FastDeploy Repository](https://github.com/PaddlePaddle/FastDeploy).
+
+**Note**: To deploy on a configuration with 4 GPUs each having at least 80G of memory, specify ```--quantization wint4```. If you specify ```--quantization wint8```, then resources for 8 GPUs are required.
+
+```bash
+python -m fastdeploy.entrypoints.openai.api_server \
+       --model baidu/ERNIE-4.5-300B-A47B-Base-Paddle \
+       --port 8180 \
+       --metrics-port 8181 \
+       --engine-worker-queue-port 8182 \
+       --quantization wint4 \
+       --tensor-parallel-size 8 \
+       --max-model-len 32768 \ # Maximum supported number of tokens
+       --max-num-seqs 32 # Maximum concurrent processing capacity
+```
+
+### Using `transformers` library
+
+The following contains a code snippet illustrating how to use the model generate content based on given inputs.
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_name = "baidu/ERNIE-4.5-300B-A47B-Base-PT"
+tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto", trust_remote_code=True)
+
+prompt = "Large language model is"
+model_inputs = tokenizer([prompt], add_special_tokens=False, return_tensors="pt").to(model.device)
+
+generated_ids = model.generate(
+    model_inputs.input_ids,
+    max_new_tokens=1024
+)
+result = tokenizer.decode(generated_ids[0].tolist(), skip_special_tokens=True)
+print("result:", result)
+```
+
+
+
+### Using vLLM
+vLLM is currently being adapted, priority can be given to using our fork repository [vllm](https://github.com/CSWYF3634076/vllm/tree/ernie)
+```bash
+# 80G * 16 GPU
+vllm serve baidu/ERNIE-4.5-300B-A47B-Base-PT --trust-remote-code
+```
+```bash
+# FP8 online quantification 80G * 8 GPU
+vllm serve baidu/ERNIE-4.5-300B-A47B-Base-PT --trust-remote-code --quantization fp8
+```
+
+
+
+## License
+
+The ERNIE 4.5 models are provided under the Apache License 2.0. This license permits commercial use, subject to its terms and conditions. Copyright (c) 2025 Baidu, Inc. All Rights Reserved.
+
+## Citation
+
+If you find ERNIE 4.5 useful or wish to use it in your projects, please kindly cite our technical report:
+
+```bibtex
+@misc{ernie2025technicalreport,
+      title={ERNIE 4.5 Technical Report},
+      author={Baidu ERNIE Team},
+      year={2025},
+      eprint={},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL},
+      url={}
+}
+```