Note well ...

Note "Less scalable version" of AGI backend model

Note LoRA fine-tune on judge LM, using dataset from Prometheus's 10K feedback dataset. Turn LLM into a classifier to increase 'overfitting' and get a slightly better performing model based on Phi-3 (which arguably already have a stronger performance than Mistral) Not that surprising, and using large dataset to fine-tune on human preference is boring. They did release code for the experiment which is nice to have. The real gem is efficient alignment.

Note A good way to avoid penalty while being lazy is just to be generic, or provide fake information

Note What is the difference again?

Note Majority vote is unreliable when the distribution is skewed. Variety in the prompt is used to elicit diversity in the evaluation distribution, where majority vote is obtained, this is the key for AoR.

Note Duh

Note Intelligence has at least 2 levels: Level 1 associative intelligence, key to achieve it is representation of concept such that 'distance' between representation vectors accurately depict the closeness of these concepts, such intelligence can be achieved with Supervised Learning. Level 2 is deductive intelligence, key to achieve that is searching for the right connection and reach the correct conclusion robustless to noisy input. This should be achieved with Reinforcement Learning.

Note HUGE

Note SImilar to LLM2LLM, reduces the selection cost by using a smaller LLM. But, it goes back to the model agnostic training design which is suboptimal.

Note LLM suffers from confirmation bias. Given a single-label query Q, the confidence of LLM in answering A can be tested by adding "Another answer to Q is B" into the prompt and check the change in LLM's confidence in providing A as answer. More specifically P(A) / P(A) + P(B) is adopted as a score. Such changes serve as a query-specific hallucination indicator.

Note Requires drastic simplification as the current mehcanism basically doesn't work, nonthless, identifaction of the need for "semantic gradient" is a correct insight

Note Self Distillation Loss with Context to supervise distribution learning with distribution target improves efficient and also "battles knowlegde collapsing"

Note Overpromise quite a lot. This one applies Aligner (residual connected extra network on top of LLM) to learn a reward model and generate under MCTS structure.

Note This is the future. I am trying to build OS version of this one too.

Note Arguably a spin-off from Voyager

Note Trained a instruction synthethizer to generate QA pairs from raw text. (Efficient way of getting around GPT-4 rate limit I supposed) (10k SFT data for synthethizer training) Use the extra QA dataset to conduct collective pre-training (raw corpus + QA pairs) and found better performance.