> It runs but any audio input (you will need to provide wav not mp3's) I tried (tried 20s/40s/300s) I get just one short sentence returned in target language that seems not related at all to my audio input (i.e. Tous les humains sont créés égaux).
You might want to open an issue on github for that one.
The model is made to work on short utterances, if you have a long speech, you'll want to segment it first.
I've tried "tous les humains sont créés égaux" on the demo: https://seamless.metademolab.com/expressive (which runs the same code as in the repo) and the output was correct. Maybe there is something wrong going on in the conversion of the input audio?
> Oh and why is Whisper a dependency? Seems not need if FB has their own model?
Whisper is a dependency as it's used as a baseline for evaluation. You can check out the paper for explanations.
I tried as short as 10s and it still provides just something random. How short does the audio need to be? Text works fine but can’t get audio-to-audio to work.
Meta employee here. The system is not perfect, or it would not "hallucinate", while it's pretty good, it does sometime make errors (not just hallucination, maybe just some mistranslation due to noise in the training data). What we want is to avoid these errors to introduce toxicity (think swear words) that weren't in the input as this could be very bad for the user. There is a separate system that double checks the output (compared to the input) and tells the translation model to try again if it's too bad.
Hi, I work on seamless. What this refers to is added toxicity mitigation. We try to detect the level of toxicity in the input and make sure that the output toxicity level is not higher. This protects the model from doing egregious errors in the translation.
There are more details in the paper if you want and the mitigation code is all open source if you want to check what it actually does.
That's an awesome feature. I think one of the worst possible outcomes of machine translation is something that ends up being accidentally offensive, and this is a smart way to mitigate that.
> one of the worst possible outcomes of machine translation is something that ends up being accidentally offensive
The Hitchhiker's Guide To The Galaxy claims the opposite:
"Meanwhile, the poor Babel fish, by effectively removing all barriers to communication between different races and cultures, has caused more and bloodier wars than anything else in the history of creation."
How do you account for colloquial (non-English) language which could be naively misconstrued as toxic?
e.g. "geil" (either cool or horny depending on usage) in German
It's not fundamentally different than e.g. "wicked" in English, but the biggest bias that potentially all these ML models exhibit is predisposition towards Anglophoneism
Our goal is to have a good recall, sometimes to the detriment of precision, so for words with multiple meanings, it might consider them toxic when in the actual context they are used in, they are not. The toxicity mitigation algorithm will search for alternative translations that have the correct meaning but not the potentially toxic word so that there is no added toxicity in the output. This means that sometimes the model might prefer a less coloquial phrasing than what a human would.
You can find details on how the multi-language creation of the toxicity lists was done in section 7.3 of the NLLB paper: https://arxiv.org/pdf/2207.04672.pdf. TLDR: it's not just a translation of a base English list, even if we started from that, each language has a curated list that was built by professional translators.
Is there an ability to turn it off? If you're translating an R rated movie with criminals who swear a lot, is it possible to get non-toxic filtered output to make sure it's being translated properly?
it only kicks-in if the output is more "toxic" than the input. If the input has a lot of swear words and the output has the same amount, then it will be left alone.
> What this refers to is added toxicity mitigation.
Oh, well that clears it up! </snark>
I don't see any definition of 'toxicity' on the landing page - it seems to be one of those 'I know it when I (hear) it' kind of words... unless there's some widely-accepted definition in this area of study?
Sorry if I wasn't clear, internally we've been talking about it a lot, but I forgot that it doesn't have such a solid definition outside of our work. Thankfully, we try to define it in section 7.3 of the NLLB paper: https://arxiv.org/pdf/2207.04672.pdf
The tldr is that if you say:
"Thank you for this job offer." you wouldn't want it to be (mis)translated as "Go F*k yourself.". But if you do say "Go F yourself", you still want it to be translated as that.
• 800 dynamic scenarios across ten realistic universes
• Tests adaptability, robustness to failure, and time sensitivity
• Moves beyond static benchmarks to evaluate real-world agent capabilities
- Agents Research Environments (ARE): A simulation platform for agents research
• Dynamic, evolving environments that mirror real-world complexity
• Built-in reward signals and comprehensive evaluation tools
• Realistic apps (email, calendar, file system, messaging) with realistic data
• Event-driven architecture that creates dynamic scenarios for multi-turn tasks