Add the code
Browse files
README.md
CHANGED
@@ -49,4 +49,8 @@ The model was tested with vLLM and should work with any toolkit supporting Mistr
|
|
49 |
|
50 |
Make sure to use the system prompt provided in the `SYSTEM_PROMPT.txt` file (from Mistral's docs) and the sampling params `temp=0.7, top_p=0.95`.
|
51 |
|
|
|
|
|
|
|
|
|
52 |
There still may be configuration errors in this model which might reduce performance. Let me know if you encounter any weird behavior!
|
|
|
49 |
|
50 |
Make sure to use the system prompt provided in the `SYSTEM_PROMPT.txt` file (from Mistral's docs) and the sampling params `temp=0.7, top_p=0.95`.
|
51 |
|
52 |
+
The code used for creating this model can be found here: https://colab.research.google.com/drive/1UuMo4VSgVoD4GfLrFgHUJvCv0cdALR7m?usp=sharing.
|
53 |
+
It requires ~150 GB of RAM (VRAM is not needed for this) since it loads three 24B models in BF16.
|
54 |
+
4-bit bits and bytes quantization could be used to reduce the memory requirements to 1/4.
|
55 |
+
|
56 |
There still may be configuration errors in this model which might reduce performance. Let me know if you encounter any weird behavior!
|