OptimusePrime commited on
Commit
3392d4c
·
verified ·
1 Parent(s): 27e14d1

Add the code

Browse files
Files changed (1) hide show
  1. README.md +4 -0
README.md CHANGED
@@ -49,4 +49,8 @@ The model was tested with vLLM and should work with any toolkit supporting Mistr
49
 
50
  Make sure to use the system prompt provided in the `SYSTEM_PROMPT.txt` file (from Mistral's docs) and the sampling params `temp=0.7, top_p=0.95`.
51
 
 
 
 
 
52
  There still may be configuration errors in this model which might reduce performance. Let me know if you encounter any weird behavior!
 
49
 
50
  Make sure to use the system prompt provided in the `SYSTEM_PROMPT.txt` file (from Mistral's docs) and the sampling params `temp=0.7, top_p=0.95`.
51
 
52
+ The code used for creating this model can be found here: https://colab.research.google.com/drive/1UuMo4VSgVoD4GfLrFgHUJvCv0cdALR7m?usp=sharing.
53
+ It requires ~150 GB of RAM (VRAM is not needed for this) since it loads three 24B models in BF16.
54
+ 4-bit bits and bytes quantization could be used to reduce the memory requirements to 1/4.
55
+
56
  There still may be configuration errors in this model which might reduce performance. Let me know if you encounter any weird behavior!