Update README.md
Browse files
README.md
CHANGED
@@ -2,16 +2,18 @@
|
|
2 |
|
3 |
This repository contains the implementation of a lightweight, modified version of the GPT architecture **Leap-0** trained from scratch using FineWeb-Edu, an open-source dataset. The project demonstrates the design, training, and optimization of a custom natural language model on local hardware.
|
4 |
|
|
|
|
|
|
|
|
|
|
|
5 |
## Features
|
6 |
- **Custom GPT Architecture**: A miniaturized version of the GPT model tailored for efficient training on limited hardware.
|
7 |
- **Local Training**: Complete model training executed on local resources, enabling cost-effective development.
|
8 |
- **Open-Source Datasets**: Trained using publicly available FineWeb-Edu dataset to ensure accessibility and reproducibility.
|
9 |
- **Scalable Design**: Architecture optimized for experimentation and scalability while maintaining resource efficiency.
|
10 |
|
11 |
-
|
12 |
-
<img src="LLM.drawio.png" alt="Description of the image" width="300">
|
13 |
-
<p><strong>Figure 1: Architecture of Leap</p>
|
14 |
-
</div>
|
15 |
|
16 |
## Implementation Details
|
17 |
1. **Model Architecture**
|
|
|
2 |
|
3 |
This repository contains the implementation of a lightweight, modified version of the GPT architecture **Leap-0** trained from scratch using FineWeb-Edu, an open-source dataset. The project demonstrates the design, training, and optimization of a custom natural language model on local hardware.
|
4 |
|
5 |
+
<div align="center">
|
6 |
+
<img src="LLM.drawio.png" alt="Description of the image" width="300">
|
7 |
+
<p><strong>Figure 1: Architecture of Leap</p>
|
8 |
+
</div>
|
9 |
+
|
10 |
## Features
|
11 |
- **Custom GPT Architecture**: A miniaturized version of the GPT model tailored for efficient training on limited hardware.
|
12 |
- **Local Training**: Complete model training executed on local resources, enabling cost-effective development.
|
13 |
- **Open-Source Datasets**: Trained using publicly available FineWeb-Edu dataset to ensure accessibility and reproducibility.
|
14 |
- **Scalable Design**: Architecture optimized for experimentation and scalability while maintaining resource efficiency.
|
15 |
|
16 |
+
|
|
|
|
|
|
|
17 |
|
18 |
## Implementation Details
|
19 |
1. **Model Architecture**
|