princeton-nlp commited on
Commit
04aa3aa
·
verified ·
1 Parent(s): 9f20d8d

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +1 -1
README.md CHANGED
@@ -198,4 +198,4 @@ We conduct supervised fine-tuning (SFT) on our base long-context model. In our p
198
  | Scheduling | 5% warmup, cosine decay till 10% peak learning rate |
199
  | Total #tokens | 1B |
200
 
201
- - Synthetic data: we also experiment with several strategies to generate long, synthetic chat data, but they have not yet helped to improve upon our UltraChat-fine-tuned chat models. The synthetic data strategies we tried include (1) using a paragraph of a long book/repo to generate question-answer pairs (); (2) using hierarchical methods to summarize a long book; (3) turning the previous synthetic long QA data into a RAG format.
 
198
  | Scheduling | 5% warmup, cosine decay till 10% peak learning rate |
199
  | Total #tokens | 1B |
200
 
201
+ - Synthetic data: we also experiment with several strategies to generate long, synthetic chat data, but they have not yet helped to improve upon our UltraChat-fine-tuned chat models. The synthetic data strategies we tried include (1) using a paragraph of a long book/repo to generate question-answer pairs; (2) using hierarchical methods to summarize a long book; (3) turning the previous synthetic long QA data into a RAG format.