The metric used is per-word surprisal: -logprob of each word you type. This is just the same thing as per-word cross entropy or KL-divergence where the user distribution is one-hot. Calibrating it so text generated by frontier models scored poorly was a challenge at first. Originally ChatGPT was scoring around 54%. I'm still having trouble assigning high scores to the personalized Gemini and ChatGPT responses when I'm logged in because all my personal context gives surprising responses.
Ribbon is looking for machine learning engineers who want to be part of a mission driven team at a fast growing, series A startup.
We're helping consumers use a mortgage to buy the home of their dreams. Institutional investors are making the real estate market inaccessible to the average consumer. Our mission is to level the playing field. We're doing research and building production systems to predict home valuations, underwrite consumers for mortgages & simplify the home buying process.
