Haystack Agents are designed in a way so that you can easily use them with different LLM providers. You just need to implement one standardized wrapper class for your modelprovider of choice (https://github.com/deepset-ai/haystack/blob/7c5f9313ff5eedf2...)
So back to your question: We will enable both ways in Haystack: 1) Loading a local model directly via Haystack AND 2) quering self-hosted models via REST (e.g. Huggingface running on AWS SageMaker). Our philosophy here: The model provider should be independent from your application logic and easy to switch.
In the current version, we support for local models only option 1. This works for many of the provided models provided by HuggingFace, e.g. flan-t5. We are already working on adding support for more open-source models (e.g. alpaca) as models like Flan-T5 don't perform great when used in Agents. The support for sagemaker endpoints is also on our list. Any options you'd like to see here?
That's right. If you want to ask questions about "general public knowledge", a plain LLM will know anyway and would not need tools. However, for many use cases you need data from your private CRM / a SQL db / a private wiki / or your obversability platform. Agents allow to "query" those tools/APIs to get the needed information so that they can answer the question. It's a matter of composition in your architecture. The "monolith LLM" works well for a certain spectrum of tasks, but at some point you want to decouple responsibilities into individual services / tools. Simplifies debugging, explainability, maintenance ...
The demo corpus there just contains documents about countries and capital cities. So you could try asking questions like "What's the climate of Beijing?" or "How many people live in the capital of the US?".
Haystack looks great, but the demo maybe highlights some difficulties with this kind of task.
"What is the population of Italy?" ...gives the population of Rome as first answer at 78.32 relevance :)
I get similar result for some other countries.
"What is the population of Cambridge?" ...to be fair, this is an ambiguous place name as there are several around the world. However the answer it gives is quite far removed from any of them:
"In 1788, Kingston had a population of 25,000", Relevance: 93.14
(Disclaimer: I'm a Haystack maintainer and I helped creating this demo)
I had to try it out the questions you asked, because your first seems totally answerable to me. And indeed I do get the right answer in the first position (60 million). Did you ask exactly the same question you posted?
For the second, unfortunately we included only country pages and capital city pages, so it's likely that the information about the population of Cambridge simply wasn't there.
In general though I agree this task is not perfect for a demo. It's hard to tell whether the model is wrong because it doesn't have enough info, or whether it does have the data but couldn't find it. The best way to evaluate it will always be to try it out on your own data :)
Yep, that's definitely this challenge with commonly available models. In a real-life product development there's most often an important step of evaluating the model(s) and fine-tuning if necessary.
Semantic document search is one of the core use cases we see in the community (besides Question Answering) and Haystack was pretty much started because we saw that you need much more than just models. It's so much pain to integrate models properly with document storage (e.g. elasticsearch), route requests effectively in larger pipelines or track user feedback in production. Have you tried using DPR or sentence transformers for your case?
Disclaimer: I am one of the maintainers of Haystack:)
