LLMs and RAG have made it so much easier to prototype and incorporate AI in production systems, but evaluation has really become a whole lot harder.
Comparing different outputs from multiple prompt and pipeline variations to a "gold standard" is mostly out of the question.
Can we ask a powerful LLM to judge between pairs of answers and a set of questions?
We just open sourced a simple tool for tournament-style ELO ranking of LLM outputs. By comparing answers from different RAG pipelines and prompts over multiple questions, RAGElo computes a ranking of the different settings, providing a good overview of what works (and what doesn't).
What are your thoughts? Do you think this is a good direction to go in for LLM evaluation?
Allows you to organize and share your paper collections, get personalized reading recommendations based on that, read papers and annotate the PDFs, and ask questions to papers using GPT.
Citations are of course only a distant proxy for real world impact. What would you recommend as a better early indicator? We also looked at Tweet count, but that seems more biased towards marketing and hype, and in fact does not even correlate very well with citation count.
We used the Google Scholar counts - and in particular a snapshot of end of february for this ranking. There is no perfect number, but at least this one is generally accepted as reasonable, and public, so easy to verify for everyone.
There was great work last year on distributed training and inference:
Petals: Collaborative Inference and Fine-tuning of Large Models
Alexander Borzunov, Dmitry Baranchuk, Tim Dettmers, Max Ryabinin, Younes Belkada, Artem Chumachenko, Pavel Samygin, Colin Raffel
https://arxiv.org/abs/2209.01188
Would be wonderful to use this host this model LLaMA: Open and Efficient Foundation Language Models https://arxiv.org/abs/2302.13971
Comparing different outputs from multiple prompt and pipeline variations to a "gold standard" is mostly out of the question.
Can we ask a powerful LLM to judge between pairs of answers and a set of questions?
We just open sourced a simple tool for tournament-style ELO ranking of LLM outputs. By comparing answers from different RAG pipelines and prompts over multiple questions, RAGElo computes a ranking of the different settings, providing a good overview of what works (and what doesn't).
What are your thoughts? Do you think this is a good direction to go in for LLM evaluation?