Update README.md
Browse files
README.md
CHANGED
@@ -198,4 +198,4 @@ We conduct supervised fine-tuning (SFT) on our base long-context model. In our p
|
|
198 |
| Scheduling | 5% warmup, cosine decay till 10% peak learning rate |
|
199 |
| Total #tokens | 1B |
|
200 |
|
201 |
-
- Synthetic data: we also experiment with several strategies to generate long, synthetic chat data, but they have not yet helped to improve upon our UltraChat-fine-tuned chat models. The synthetic data strategies we tried include (1) using a paragraph of a long book/repo to generate question-answer pairs
|
|
|
198 |
| Scheduling | 5% warmup, cosine decay till 10% peak learning rate |
|
199 |
| Total #tokens | 1B |
|
200 |
|
201 |
+
- Synthetic data: we also experiment with several strategies to generate long, synthetic chat data, but they have not yet helped to improve upon our UltraChat-fine-tuned chat models. The synthetic data strategies we tried include (1) using a paragraph of a long book/repo to generate question-answer pairs; (2) using hierarchical methods to summarize a long book; (3) turning the previous synthetic long QA data into a RAG format.
|