It feels awkward having my first post sharing my stuff, but this is a weekend project that I really enjoyed working on. I'd love to meet more people interested in random ideas like this.
A hard part of building AI applications is choosing which model to use. What if we donโt have to? What if we can predict the best model for any prompt?
Predictive human preference aims to predict which model users might prefer for a specific query.
One use case is model routing. If we know in advance that for a prompt, users will prefer Claude Instantโs response over GPT-4, and Claude Instant is cheaper/faster than GPT-4, we can route this prompt to Claude Instant. Model routing has the potential to increase response quality while reducing costs and latency.
One pattern is that for simple prompts, weak models can do (nearly) as well as strong models. For more challenging prompts, however, users are more likely to prefer stronger models. Hereโs a visualization of predicted human preference for an easy prompt (โhello, how are you?โ) and a challenging prompt (โExplain why Planc length โฆโ).
Preference predictors make it possible to create leaderboards unique to any prompt and domain.
3 replies
ยท
reacted to alielfilali01's
post with โค๏ธover 1 year ago
In late 2021, our team of five engineers, scattered around the globe, signed the papers to shut down our startup, Gradio. For many founders, this would have been a moment of sadness or even bitter reflection.
But we were celebrating. We were getting acquired by Hugging Face!
We had been working very hard towards this acquisition, but for weeks, the acquisition had been blocked by a single investor. The more we pressed him, the more he buckled down, refusing to sign off on the acquisition. Until, unexpectedly, the investor conceded, allowing us to join Hugging Face.
For the first time since our acquisition, Iโm writing down the story in detail, hoping that it may shed some light into the obscure world of startup acquisitions and what decisions founders can make to improve their odds for a successful acquisition.
To understand how we got acquired by Hugging Face, you need to know why we started Gradio.
Two years before the acquisition, in early 2019, I was working on a research project at Stanford. It was the third year of my PhD, and my labmates and I had trained a machine learning model that could predict patient biomarkers (such as whether patients had certain diseases or an implanted pacemaker) from an ultrasound image of their heart โ as well as a cardiologist.