Thanks for sharing this! I'm going to put this on my list to play around with. I'm not really an expert in this tech, I come from the audio background, but recently was playing around with streaming Speech-to-Text (using Whisper) / Text-to-Speech (using Kokoro at the time) on a local machine.
The most challenging part in my build was tuning the inference batch sizing here. I was able to get it working well for Speech-to-Text down to batch sizes of 200ms. I even implement a basic local agreement algorithm and it was still very fast (inferencing time, I think, was around 10-20ms?). You're basically limited by the minimum batch size, NOT inference time. Maybe that's a missing "secret sauce" suggested in the original post?
In the use case listed above, the TTS probably isn't a bottleneck as long as OP can generate tokens quickly.
All this being said a wrapped model like this that is able to handle hand-offs between these parts of the process sounds really useful and I'll definitely be interested in seeing how it performs.
Let me know if you guys play with this and find success.
Apparently, following a recent serious data breach[0], HealthEquity is moving forward with an authentication change to deprecate username/password entirely in favor of Passkeys (webauthn resident key implementation).
This struck me as a bit of "news" given the security posture being taken here. I haven't seen any other institution force adoption of Passkeys as the primary authentication mechanism to access their services; I've mostly seen 2FA/MFA implementations with Passkeys as an optional/additional authentication mechanism. Like many folks here[1], I've been hesitant to fully adopt passkeys as a primary authentication mechanism until the credential exchange specification[2] (which allows users to export/import keys from one credential storage platform to another) is finalized and better supported across the different credential stores.
From this page:
Can I opt out of using a passkey?
No. Once passkey login is rolled out to your account, you’ll need to use it to access your benefits—either through the mobile app or on the web. Traditional login options won’t be available.
The 2021 IETF (Internet Engineering Task Force) working group sessions are ongoing this week, discussing many of the new proposals coming out of the working groups for internet protocol changes.
Even without being registered for the conference, you can stream the audio in real-time by connecting to the audio stream directly:
The most challenging part in my build was tuning the inference batch sizing here. I was able to get it working well for Speech-to-Text down to batch sizes of 200ms. I even implement a basic local agreement algorithm and it was still very fast (inferencing time, I think, was around 10-20ms?). You're basically limited by the minimum batch size, NOT inference time. Maybe that's a missing "secret sauce" suggested in the original post?
In the use case listed above, the TTS probably isn't a bottleneck as long as OP can generate tokens quickly.
All this being said a wrapped model like this that is able to handle hand-offs between these parts of the process sounds really useful and I'll definitely be interested in seeing how it performs.
Let me know if you guys play with this and find success.