What I’m missing from this announcement is the capability to use connectors and tools. I don’t really get it - NONE of the frontier assistants can use tools / connectors while in voice mode - Claude, ChatGPT, Gemini, Grok. It seems so obvious: I want to be able to research stuff, pull up documents, jot down notes and do productive work while I’m talking to it, and not end voice mode whenever I need to connect to an app or service.
It’s weird. The old Claude voice mode WAS able to use tools but when they revamped it, it lost that capability and is now pinned to Haiku :(
So, yay for finally a voice mode that’s powered by a frontier model and hopefully as good as Grok voice, but sad to still not see tool use while in voice mode.
(I haven’t tried it yet, only read the announcement)
One of the videos on the announcement page shows someone making coffee and it _appears_ that the agent counts 30 seconds in real-time. Curious to know how they made it do that without tool support.
A timer tool with a callback feature would take like a couple of hours to implement, a model which has a native internal ability to know the time would take ages to make
I don't think it needs an 'internal clock'. It needs two things. (1) the timer tool, (2) to know not to lie about having a tool or using a tool. I think (2) is the hard thing.
> I don’t really get it - NONE of the frontier assistants can use tools / connectors while in voice mode - Claude, ChatGPT, Gemini, Grok.
Is that true? I have a friend who often brainstorms with Gemini (I think just on an Android phone), and he has it actually do stuff related to the conversation (including adding content to notes).
In any case, you can always vibe code one with pi!
Using gpt-realtime-2 you can achieve very similar results with low latency streaming which gives you this feeling of an assistant that is there in the room with you. I made a tweak to so you can try out the gpt-realtime-2 model with tooling that gives you this: https://sippet.ai/?enabler=hn
Kept trying to use the demo, but despite giving it microphone permissions and seeing that the microphone was in use, couldn't get the demo to 'hear' me at all.
I’ve been using gpt-realtime-1 for my personal assistant that runs my company and plans my day. And it works pretty well, even makes tool calls and all that.
But the multi modal stuff has resulted in a lot of debugging with weird events and message and audio sequences having race conditions, but overall it is pretty awesome.
Looking forward to moving to this model later today and will chime back in with results.
I could see it relating to tools having unpredictable latency but if they already do background hand off to 5.5 then it seems like they could just enable it within that context.
A big part of this announcement does seem to be _delegation_ in the background; they give the example of web search but that could be any tool. I haven't tried it yet either but sounds like they've found a reasonable UX that mixes that sort of high-and-variable latency tool calling (potentially with agentic loops) with a continuously speaking live voice.
Funny enough I have only encountered two voice modes that were okay to use tools and they are so far out there that you wouldn't even think to try. Bixby and Perplexity (in android phone digital assistant mode) both seem happy to use whatever the account is connected to. I mainly use it for managing my local phone's calendar. Claude chat can interact with it but voice can't which is frustrating.
This is something I built for myself, and to experiment with inference stacks. You can obviously just transcribe audio and hand it off to frontier models, so all you really need is a good voice stack and a “driver” for the interaction (like a phone call, place to see their work).
There are two big problems with this space IMO. One isn’t that you can’t get this to work but that people generally aren’t willing to pay for it for themselves, rather as a way to screen or automate stuff to be used by other people. Did you know Claude Code has a voice mode and that openai launched whisper a year ago, both of which have positive sentiment and adoption in heavy ai tool users? Yet it’s a blip in their marketing or why people use their products, meanwhile outside of coding, most of the biggest and highest earning AI product companies so far are voice agents targeting customer service, sales, business processes, etc.
The second is related: voice is genuinely a low-bandwidth medium, so as a primary interface for interacting with AI there is not a lot you can get out of it compared to eg complex technical work or visualizations or interactive applications. It is physically and mentally demanding to speak-aloud a highly detailed prompt fast enough that VAD won’t cut you off and you have something with comparable information density or specificity vs text. But to keep up a shorter and more natural cadence you’ll not be able to wait on a lot of thinking/tool unless you play UI tricks (ums and fillers, two models in a trench coat), break the illusion of a single coherent conversation, or take a lot of long pauses.
That’s why for the supplementary coding use case it’s mostly used for remote steering, and for general use marketed towards the large and very not-online group of people for whom typing is not a natural or common thing for them to spend their time on. Now that so much spend goes through heavily used token subscriptions and they’ve proven that kind of product, they’re not marketing “tool to get the most tokens per $ running your subscription 24/7” anymore lol.
What I’m most interested in is true “ambient” tool use against my own data or work, and for-later (or pushed live via your phone) visualizations or “five models in a trench coat but still coherent” UX, which you probably are too. But I think unless you work a lot with AI tools already it’s hard to understand how that’s any different from asking Alexa to set a timer, and either way something you’re not so desperate to have that you go looking for it, or pay smaller vendors/set up yourself.
Exactly. I told the assistant to create world cup google calendar entries and it said it can't connect to my calendar, whereas I can have it do that with no issues on the normal gpt client.
Because they take too long to run, and have an unpredictable latency and success rate. Seeing loading spinners and error messages in a visual interface is fine, but it would firmly put a natural language conversation in uncanny valley territory.
Regular chat already supports voice input, so might as well use that.
Huh? You can expose tool functions to Gemini's live API, and it will call them as part of thinking about responses. I've used this to expose e.g. current GPS location for a smartwatch voice assistant.
It’s weird. The old Claude voice mode WAS able to use tools but when they revamped it, it lost that capability and is now pinned to Haiku :(
So, yay for finally a voice mode that’s powered by a frontier model and hopefully as good as Grok voice, but sad to still not see tool use while in voice mode.
(I haven’t tried it yet, only read the announcement)