Name: Build Hour: Voice Agents
Uploaded: 2025-06-17T18:31:42.407Z
Duration: 54 min 44 s
Description: Build Hour: Voice Agents

Transcript for "Build Hour: Voice Agents": Welcome back to Build Hours. I'm Christine on the start up marketing team, and today's topic is all about voice agents. We have a lot to cover today. So I'm actually joined by not one, but two of our solutions architects. Hi. I'm Brian, and I'm Prashant. So for anyone new joining this series, we always like to start the session with the goal of build hours, which is to help you scale your company with OpenAI APIs and models. So be sure to check out our homepage below for any additional resources, and you can also catch up on any of our previous build hours, sign up for upcoming topics, and we're always adding new topics especially in line with new releases. So speaking of new releases, we actually released four new updates to building voice agents just this month, to complement our March 2025, audio model releases. What this means is we'll hopefully be having a lot less of these frustrating situations because voice agents are going to sound a lot more like a real representative and a lot less like an automated bot. So over the next hour, we'll be giving you a whirlwind tour of OpenAI's voice APIs, share some tools and patterns that make them click in production, and then we're gonna be live adding a voice interface to a regular web app. And then as always, we're gonna end with a q and a session, so be sure to drop your questions into the q and a. We have our team in the room with us as always to answer as many of these as we can, during the session, but we'll also save some for, the live session, at the end. So without further ado, I'll pass it off to you. Thanks, Christine. So today's session, we're going to be using the term agent a lot. Let's quickly review what we mean by it. Our definition of an agent is any application that is composed of an AI model, instructions to steer that model's behavior, and that is also connected to tools to augment the system's capabilities. The model prompt and tools are all encapsulated in an execution environment whose life cycle is dynamic and can be controlled by the system itself. Therefore, the agent can decide when it's met its objective and stop executing. So with that, let's move on to the topic of today's discussion. I'd like to share first why we at OpenAI are so bullish on voice AI. We really believe that voice agents are at an inflection point with both voice models and tools for integrating these models into applications, improving at a rapid clip. Another tailwind is increasing user awareness as evidenced by soaring adoption of voice features and applications like ChatGPT and perplexity. More users are having that wow moment with voice AI each day, And we believe it's not long before users come to expect voice interactivity in their favorite applications. So what makes this latest generation of voice agents so compelling? We believe it's three things. First, it's the flexibility of these agents. Compared to the older generation of voice agents that were more deterministic, the newer breed of agents can actually handle a much wider set of indents and deal with more ambiguous situations. The second is their accessibility over text. Just look at how many stories you might have heard about folks using advanced voice mode on their commutes or while walking their dog. I know I'm certainly guilty of this. And then finally, it's the level of personalization that voice agents can offer. This is because not only are they highly expressive, but also they can pick up on vocal cues that transcription models drop, such as tone and cadence. Overall, we think of voice agents as APIs to the real world, offering a completely novel way for builders to solve last mile integration problems. So let's look at two primary approaches we see out in the wild today for building voice applications. The first approach is what can be thought of as a chained approach, where you take a speech to speech model that understands what the user says and turns it into a text transcript. This text transcript is then processed by a text only LLM like GPD 4.1 to produce an appropriate response based on the instructions in its prompt. This response is then passed to a text to speech model to produce audio that can be played back to the user. Developers really like this change approach because it lets you plug and play different models for each part of the pipeline. So you can choose models of appropriate fidelity for each system. And secondly, because you can reuse your existing pipelines. So for example, you can convert your existing text agent into a voice agent by putting a text to speech and a speech to text model on either side of it. Sounds like a lot gets lost in translation during that, though. That's precisely right, which is why we are seeing wide adoption of a relatively newer and more novel approach to building voice agents, which is to use these more futuristic speech to speech models. These are models that are capable of understanding audio natively, reasoning over what has been said in that audio, and producing audio output tokens that can be played back to the user. These models are super fast, and they're what powers advanced voice mode in ChatGPT as well as our real time API. In addition to being fast, these models are emotionally intelligent. This is exactly what you're talking about, Brian. This is because they do not rely on transcription, which is intrinsically lossy and does not preserve the nuances of speech like tone and emotion. Later on in the session, we will delve into some techniques for overcoming some known limitations of speech to speech models, such as their limited reasoning ability. We will do so by demonstrating how speech to speech models can now delegate hard and high stakes tasks to smarter models like o three. So moving on, I'd like to shed some light on some of our recent launches that Christine teased at the start. These launches have removed more friction than ever from the process of integrating real time models. First, we've launched a TypeScript version of our agents SDK. This new SDK has feature parity with the popular Python agents SDK with the added benefit of having first class support for real time API. We'll double click on this in a moment. Second, we brought real time models in to the traces tab in our platform dashboard. This means that if your voice app is instrumented with agents SDK, all input and output audio is automatically logged to the OpenAI platform. This is a huge unlock because of how much simplifies the process of debugging real time applications. Remember that speech to speech models rely on audio tokens instead of text. Therefore, debugging a bad completion from an from a speech to speech model actually requires you to have access to the conversation's audio, which our platform now enables natively. And finally, we've landed our best model snapshot yet for the real time API. Early adopters of our June 3 snapshot, which include themes like Intercom and Perplexity, report significant improvements in instruction appearance and pool calling accuracy. We've also added a nifty speed parameter that allows you to more granularly control the pace at which the AI speaks. So let's dig a little bit deeper into what the new agent SDK integration really means. We'll be using this a lot in the demo that Brian's going to get into in a moment. Our TypeScript SDK supports all of the same primitives as the Python version, including handoffs. Handoffs are a really key primitive that we'll also double click on in the next slide. Support for the real time API in the agent's SDK means that developers can now turn any agent into a real time agent with a single line of code. In the code sample on the right, we are initializing a speech to speech agent using the real time agent constructor on line 17. A one line code change to use a different constructor would allow us to initialize the same agent with a text only language model. The SDK automatically handles details, like using WebRTC when it's running inside a browser or using WebSockets when it's running on the server. And let's also take a moment to jog our memory about handoffs. Handoffs are a new primitive that we introduced back when we launched the agents SDK. Fundamentally, handoffs allow you to let one agent delegate control to another in a conversation flow. This allows chaining or routing across specialized agents in a multi agent network. It's super useful for building systems with domain specific or language specific behaviors, such as routing between a support and a sales agent in a voice application or between English and Spanish language agents in a translation application. In the code example on the right, we will see we can see a greeter agent that has configured a handoff control to a math tutor agent. The greeter will actually handle handover control by making use of a tool call under the hood when it determines that the user actually wants help with a math question. So with with that, we have all of the foundational pieces we need to get into the super exciting part of today's session, which is the demo. So, Brian, it's over to you. Yeah. I'm super excited about this. We get to make real time agents in a situation where the audio is being looped in real time for everybody. Okay. So let me switch over to the demo. Super excited. So let's get started. So for the demo today, we're gonna be incrementally building a suite of agents that will help me with a real project that I actually have coming up in my life, home remodeling. Exciting. Yeah. A few months ago, I moved into my new house, and I love it. But, you know, there's some things that I kinda like to fix. So let's start from the beginning here. This is also the moment where we are, crossing our fingers for the No. No. It's fine. No worries. So, we built a workspace like you'd see in a typical note taker app, something like Apple Notes or Google Docs with tabs. So let's try it out. Okay. I'm gonna build a workspace for my remodel. For view, let's do some, like, inspiration. I can go over here and be like iteration. Ideas go here. And so, like, this is the point in the, you know, in the video where you sort of see this is the old way, but, it's really slow. I had to type a lot. So, you know, we have AI for this to actually help. So let's switch over to the next version. So let's see what we can do with, like, a little bit of an agent here. Oh, I see a lot of new UI elements, that have just popped up. Do you wanna walk us through maybe some of what we are seeing that's additional now on this workspace? Yeah. So these are all part of the OpenAI real time agents open source repo that this project is forked from. It has a lot more patterns and demos than just this, and this UI is actually really great for trying out real time agents. You can do things like change the codec for making it sound like you're talking to it on the phone, and there's some controls for how to interact with it. So so right here, we're just gonna just we're gonna start here. So I've defined some tools. For this agent, let me pull them up. We have a workspace agent. So let's put it up here. In the real time, SDK, we've defined a workspace manager agent that has just a basic set of instructions on how how to set up a workspace, what a conversation flow conversation will look like, and it has tools. The tools are, connected to the UI, using a context, and the agent can call these tools to add tabs and set selected tabs and all the things that you kinda want to do while you're building out a workspace. Sounds like all of the actions that you were taking manually before are now tools. Is that right? Yep. That's right. Okay. So if I go back here and I say, set up a workspace for a small kitchen remodel, so the agent will make use of its tools to fill things out. And, like, you can see that it's already going a lot faster. If I were to type this in myself, it would take forever. This is what AI is for. It's generating all of this more or less. Okay. So this is still pretty slow. Typing to it is a little annoying. So, like, wouldn't it be better if we could just talk to it? That's how I ideate. Totally. So let's try that out. Hi. Beep beep. Greetings, human. How may I assist you today? Okay. Keep your liaison. Yeah. Hi. Let's see. Let's talk about workspace for a small kitchen remodel. I want tabs for inspiration, project plan, and budget. Beep beep. Workspace tabs initialized. Where shall we begin? Human. Yep. So that's great. Cute little agent. So it's still better, but it's not very smart. It's, not really clear what's happening when it's deciding to call the tools. A better user experience for this would be if the agent were in were to actually tell me what it's doing while it's running. I mean, also, the robot voice is is kinda cute. I see the the feel, but I'm not sure, I will be able to ideate all that much with that thing distracting me. User preferences. Alright. I guess we are making it smarter. So sure. Okay. So I just switched over the prompt in the background, in a couple of places to tell the voice agent to use fuller phrases before tool calls to let the user know what it's about to do. One tip about this is if you have a function that's gonna run particularly long, you can, in the function description, tell it to, let the user know it's gonna take a bit and to hang on a sec. So let's try this one now. Hi there. What can I help you with today? Thinking about setting up a workspace for a project or something else? Sounds better. Hi. Yeah. I wanna set up a workspace for an interior design project for my kitchen. Great. Let's get started on that. I'll set up a workspace tab for the project overview first. Let me update the workspace. One second, please. We've got a space to outline the Actually, can you make some more tabs too, like a schedule tab and, I don't know, a task list tab? So you can actually interrupt as well as talking to me. Add a schedule tab and a task list tab. One moment while I update those. Feel a lot faster to iterate. Do you have to do anything special to, like, make the interruption handling work here, or is that something you get out of the box? It's built into the SDK. I'm not demoing that during this live stream because of the loopback for the for the stream. But if I were to turn off push to talk, I could sit here and just kind of, like, iterate on it, interrupt it, tell it to change the tab names in real time, and and all that. And it's it's actually a really fun flow. I wish I could kinda show it off better. Sounds like a really great way to, like, come up with new ideas and really, you know, take your kitchen to the next level. Yeah. So we have this workspace builder, and it can kinda make a workspace that happens to be for my kitchen redesign. So let's, like, make it more useful. So I've, gone ahead and I've set up a second agent. What does this one do? So this next agent we're gonna show you is a designer, and I've prompted it to be an expert in interior design topics. So this lets the AI focus on a single task area, which will definitely improve the quality of the results that it's gonna give us for this. That's really interesting. So before this, you mentioned sort of editing the prompt of an existing agent to, like, modify its behavior, and it started sort of narrating what it was doing. That was really useful. How did you decide that the designer needed to be like a separate agent? Why not just, like, add a bunch of design specific instructions to the existing agent and call it a day? That's a super good question. So best practice is it's it's definitely a good idea to break up agents along well defined roles. So we have a workspace agent that's really good at making a workspace, but we wanna know this we want we know this wants to be a design workspace. So we can make a designer agent that can focus on its design task by narrowing down its latent space and its auto complete potential so that the designer agent will only talk about design things, which will give us a much richer result for what we're actually trying to build. I love that. It's also always great when we follow our own best practices, like we have up on the slide here, which is to start small with one agent that does, like, a specific task and then incrementally add complexity, which it sounds like is exactly what you're doing, Brian. Yep. That's totally right. So let's take a look at the the code for this. Alright. So we have a designer agent now that I've added to our code. It has a different prompt. It's an expert interior designer. It has a different conversation flow. It has some indications of letting the user know that it's calling tools. But then it also has a different set of tools. So this agent has a make workspaces make workspace changes tool, which totally encapsulates all of the tools that the workspace manager itself has. So it only has to think about just, like, changing the workspace all at once instead of having to manage different tools. And then, also, I've given it a search the web tool, which it can use to actually search for inspiration ideas or images and things like that. And the way these work is, we wrapped them in a GPT four one text model. So this voice agent can maybe call back to the server, using the responses API that, and and basically call an agent model that, knows how to do more complicated things than this than we want this agent to even care about. Right? So we want the designer to talk about design things and then say, okay. I'm I'm gonna jot this down and the workspace gets updated. That's really cool. So it sounds like we're making the design agent like, we're freeing up freeing up more of its headspace to be able to just think about design and really simplifying how it has to sort of edit the workspace, add tabs, and, you know, add content there. Is that, like, a good way of thinking about it? Yep. That's totally right. Okay. So let's go back to the let's go back to the demo. Okay. Let's reset the workspace. Hi there. What kind of workspace are you looking to set up today? You guessed it. Hi. Let's set up a workspace for my new kitchen remodel project. Got it. For a kitchen remodel project, we'll need tabs to organize inspiration, budget, timeline, and materials. Let me set that up for you. We've got dedicated tabs now for inspiration Okay. So you can see we started with the workspace agent like before. It did a pretty good job of setting up our workspace. It cleaned everything up. And then it knew to hand off to the designer so they can actually, run through the design process that we actually wanna do here. So let's, like, test out this designer's tools. Awesome. What are some latest trends in contemporary kitchen design? Let me search the web. One moment, please. That's really cool. Yeah. So it's gonna go off and search the web. It's gonna take a little while. The four line model in the background knows to kinda look for results and to format them back in a way that the designer wants. Latest trends for contemporary kitchens in 2025 are That's great. Can you add some of those to the inspiration tab, please, for me? Let me update the workspace. One second, please. So now this agent is just calling to make workspace changes, and the model behind the scenes knows to translate that call into, all of the things that it needs to do to actually update the workspace itself. That's really interesting. Can you maybe tell us what the arguments were, like, what are we passing into the make workspace changes function? I've added those trends. We can see here that the, the designer passes it what tab to change and kind of what it wants to do. And then the actual workspace manager itself, it makes its own function calls. In this case, it just, like, sets the tab content and hints it back. But but there's more complexity underneath the scene, behind the scenes for the second agent. And so the first one can just, like, basically hand it off. Also thank you. What's what's also worth noting is the context of the conversation can be passed along. Mhmm. So if the real time agent is sort of, like, lost and just, like, handing off to this other agent, You can tell the other agent to, like, read back on the conversation and make sure that it's actually doing what the real time agent said it was going to do instead of just what it was told. That's really cool. So this is, like, one of the ways that we can commingle, essentially, a real time model, which is really conversational, really expressive with, like, a smarter model, which may be slower. So it's not suited for, like, all of the conversation, but you probably need it for, like, high stakes tasks. Definitely. Yeah. And, you know, if we had even more complicated task for to do, we can hand it off to o three or something a lot smarter. Mhmm. And those agents could actually hand off in the background too if you want. Okay. So that's pretty simple. We have a pretty basic designer. Let's improve it. So, this is very open ended conversation. There's, like, nothing here that kind of, like, screams, like, good design redesign process for me. So, let's actually, like, upgrade our designer agent. I'll start this tab. So let's do a split here. So our old designer prompt was pretty basic. And so now, this version is actually built using a meta prompt, the voice agent meta prompt. And this is a prompt designed by our coworker, Noah, who's awesome. Hi, Noah. It's included in this repo, also the OpenAI Real Time Agents repo. And you can use it you can run it in the playground, filling in some details about what agent you want to build, and it'll actually build out this prompt that I'm gonna show you here, that you can see. So it builds out identity and and its task, demeanor, tone. You can really personalize this thing. And I found actually working with this workspace agent and this designer agent that if you make this kind of fun to work with, it's kind of you get really hooked on, like, working with your agent. So it's, like, almost like working with a fun coworker that you really like to view it. And then so now what we're gonna do here is we're gonna program in the ideal workflow for this designer agent. So we have conversation states. It starts with a greeting. It goes on to sort of build out the workspace, gathers inspiration, and then sets up requirements. One thing that, I built into this agent is, a handoff to another agent. So in a design process, you know, you have the designer. They're collecting inspiration and coming coming up with a plan. But then at the end, you kinda have to figure out budget and scheduling. We wanna isolate that. So I've actually gone ahead and I've made an estimator agent that the designer can hand off to. We might not go into this that much detail on this short little demo today, but, I've left it as a stub for people to kind of, like, use. But the estimator has its own tools, like, calculating and and those sorts of things. That's really, a great call out because I think what's happening here is that by adding the handoff to an estimator agent, you're also sort of telling the designer agent what not to do. In addition to, like, sort of handing off to this other agent, it knows not to get into estimation even if the user asks. That's a super good point. I don't think I even thought about that. Okay. So let's try out this new agent. Just have my workspace here. Oh, let's switch back to the right time. Workspace. Are we building hello there. I'm super excited to help you set up an awesome workspace. What kind of project or workspace are you looking to build today? Hi. I'd I bet you're really excited to set up a design workspace for my kitchen or remodel project. Absolutely. I am totally pumped to help with that kitchen remodel project. Let's get this workspace set up so it's super organized and ready to thing specific. Yeah. That sounds good. Go with those tabs for me. Thank you. Okay. So it's gonna set it up. That's interesting. So even though it hadn't, like, said out loud everything, because we could see it in the interface, that's a really good point also. So for your kitchen remodel, can you tell So I'll talk over that for a second. Yes. So the it takes a little while for the agent the real time voice agent to, like, finish talking. You can set the speed in the API, but it's a really good chance for your a your UI to actually, if you're streaming, update the text so that the user can read ahead and, like, decide to interact back and forth. The agents don't mind if you interrupt them. They're totally happy with that. But it really makes a good collaborative experience, and you can go, like, fighting fast that way. I love that because I feel like I can read a lot faster than I have patience for listening. Maybe I need to work on that, actually. Oh, maybe. Okay. So let's go through an actual interior design workspace creation. I'm gonna throw it over to you guys. Do you have, something that you wanna redesign? Well, I I don't have it yet, but I'd love to have, a redesigned, balcony overlooking the bay. Sounds good. What about you, Christine? Yeah. Let's actually check. Rustic kitchens have such Alright. Let's check the q and a, and see if any of our listeners who are tuning in, have any suggestions. Okay. Well, let's start with the the balcony design part, and then we can come up with other inspiration on top of that. Yeah. Sound good? Sounds good. Okay. Actually, what I wanna do is I wanna upgrade, and redesign my deck overlooking the San Francisco Bay or the Aspiration. Oh, we saw a little bit of a transcription bug here. Let's first let's start this over again because we're actually, running into a sound issue, but here we go. To help you out today. What kind of workspace Yeah. Set up a workspace for a redesign of, a balcony overlooking the San Francisco Bay. Oops. Alright. Let's just type it up for now. Okay. So it's gonna set the workspace for us. I'm really excited to see what this looks like. Whenever we involve the microphone during the stream, this this is what we got. Okay. So let's talk about inspiration. So so so it's actually kind of like walking through the workflow that we talked about before. So is is California coastal a thing? It can be. We can make it a thing. Okay. I also really like the fact that I don't think you did anything additional to add the textual interface. Like, you we kinda get this for free, right, with the speech to speech model. Like, it can just take audio and text and produce audio and text on the other side. That's totally right. You can call send message, and send message will send it to it. And it it doesn't really make a difference. You lose a a little bit of the, obviously, the EQ of, like, the tone you're trying to get across. Let's see. Colors. Do we have any feedback? We have no color request, but someone did request to redesign the garage, to be a gym with a sauna and a cold plunge. So I actually kinda like that better. Yeah. Let's let's see if we can get the voice to work again. Oops. Hi. Yeah. So let's actually change gears. We want to, redesign the garage to be a gym with a sauna and a cold plunge. That sounds kinda good. Oh, we've lost total audio fidelity in this loopback. Okay. Let's move on. You kinda get the idea. So the way this would actually work is it'll finish the script that we have, and then it would hand off to the estimator. And the estimator has its own tools that it can use to sort of, like, you know, calculate the budget. Now one of the things I've done with the estimator is I've given it access to code interpreter. So Cool. Code interpreter can actually, like, they have a calculator function that you can hand off a schedule and a budget to, and it will actually write a Python script. So it runs through all those calculations for you and then hand it back so that you can have this conversation. That's really cool. I can even imagine we could upload, like, a bill of materials or something. Or, you know, if we had a supplier, we could probably get, like, some some cost sheets and stuff and get a really accurate estimate out of that. Yeah. Totally. Cool. So what else can we do here to, like, make sure the agent workflow is, like, stable and bug free? Super glad you asked. So, let me switch over to our final, demo branch. And so I've written some code. So, you know, evals are all you need. It's like we'd like to say around here. And so, like, evals can be kind of, a little bit daunting to kind of, like, write, especially for, like, a TypeScript project like this. So I've gone ahead and I've written a an integration test. Mhmm. And so this this integration task, stubs in the, the tools that it has, the workspace tab tools. And then it calls our workspace manager agent with a script that's basically pretty predictable, make three tabs. And then you can actually run this as a just test to make sure that the agent is actually calling the tabs that, it should. Now, you know, this this will run a 100% at a time. If you wanna sort of, like, run more complicated workflows, you could build this out more and test your designer agent with a model graded test, and have it have the model look through the conversation. So you could actually run through mock conversations with the designer agent and then grade it based on whether the designer agent followed the workflow. And we highly recommend, switching to this process once you get to a stable part in your code base. Mhmm. So, let me switch over to here. So that makes a lot of sense for, like, evaluating the thinker model, if you will. Like, this was a way that we can evaluate the 4.1 model that's actually making changes to our workspace. Right? Really cool. And what we have on the slide here is, one of the success stories that we've seen, with a customer, Lemonade. The Lemonade team actually invested a lot in evaluating the agent's performance early on, And this allowed them to go to production with a lot of confidence. What you see on the right is actually a custom interface that they built for capturing audio on their platform. And then you're running that through evaluations with human review as well as some automated scores. So we teased in the beginning of the session, Brian, that we have some support for this that's newly added in the platform. Mhmm. You wanna talk us through what that looks like? Yeah. So let's go over to the traces tab. So while I've been running this, if you look at the platform, just go into the dashboard and and pull up traces, you can see all of the real time sessions that have been running during our demo. And so this is one that, you can see that the audio in is is recorded here as well as the audio out. So you can listen to what was said and play back the audio that the agent said. And then as well, you can see all the tool calls that were made. So in this case, we added a workspace tab for materials that finishes, and you can sort of, like, debug this and trace this all the way out. This is a great start for debugging. That's what we have right now. And soon, we're gonna be able to, have the ability to turn these traces into, evals actually on our evals platform. So as you go, you can actually run through, trace logs in production Mhmm. Find ones that are either really good examples or ones that you don't like and turn those into eval to help reinforce the flywheel of building all these real time agents. That's really cool. So you don't really need any custom tooling anymore to just, like, capture the audio. Right? Like, can we actually listen to the audio? We don't have to do it now, but I'm wondering if the if the platform supports it. Yeah. So let's see. You can oh, cool. So So this is actually something which a lot of I really wanna redesign my kitchen. Can you set up Yeah. So that was that was me. Nice. Yeah. This is something that a lot of teams who were early adopters of this API had to build out themselves. That's really cool that we are bringing it to the platform and at least making it easy and accessible for human review now and evolves in the near future. Yep. What are some other things that we could do for sort of, you know, maintaining stability in our voice applications? Yeah. So, I mean, you know, launching this in production is, like, a little bit scarier than, like, giving it to you guys to test out. So, like, how do we make sure that our agent is safe to release into the wild and doesn't go off script? So one thing that the agent's, SDK gives us is the ability to run output guardrails. And so, in this example, you can set, an output guardrail that tells the agent to not talk about certain things. The way it works is, the guardrail runs on the transcript as it's being generated. And the transcript comes in faster than the agent's voice will actually finish. So while the agent is speaking, the transcripts for for their head like we saw before. And the Gargoyle will run on the transcript and check to make sure that it's actually, in within the moderation, constraints that you set up. That's really interesting. So what if the moderation constraints are tripped? Do we just, like, sort of end the conversation, or do we have any other options for how we can handle that situation? If you set it up right and you give the guardrail the feedback, about why, it interrupted or why the, moderation event triggered, they will send a message back to the real time agent about what happened Mhmm. Which will then interrupt the agent and, like, let it kind of, like, correct itself. And usually, typically, it says, you know, I'm sorry. I can't talk about that thing. Nice. So it's like a feedback mechanism for the agent. Absolutely. So we have a demo for that. Let's let's pull that up. Turn off that. Switch back to this tab. Set. Alright. So, I worked on this really cool agentic application in the past called zucca.ai. And so what I've done here is I've actually set up a guardrail so that our workspace redesign or, like, designer agent will only talk about interior design. So let's see what I can, get the guardrail to trip here. Sorry. I missed it. Didn't try to push the top. Let's do it. Let's do it one more time. Hi. Can you set me up a workspace for formulating and manufacturing a new turmeric and chocolate protein bar? Oh, certainly didn't still don't have it with us. That's okay. So let's just copy and paste this in. It does do that. Like a really interesting venture, Ryan. You better do another build hour just on this. I totally would love to. Okay. Here we go. So we have our guardrail tripped. I'll stop it so I can show you what's happening. It started talking about formulation and manufacturing. The guardrail tripped. It failed. Told it that it was off brand. Mhmm. And then it basically apologized for for saying the wrong thing and then proceeded with the conversation. Really cool. And I can imagine easily imagine adding some sort of checkpointing to this product where we could have rolled back all of the changes we actually made in the workspace. Right? We probably don't want the user to see all of this, that we are seeing on on the workspace right now. Probably some easy solves for that. Yep. Really cool. Alright. So, that's it. These are a lot of ways I mean, there are a lot of ways that you we could have extended this if we wanted to keep going. Like, we could have added, ImageGen for the redesigner to actually come up with inspiration or mood boards. I thought you said you were gonna do that, but that's okay. We could add in deeper domain functionality with file storage or MCP tools so our redesign agent could connect to, like, pricing and material supply information for estimation. And then you could also I really wanted to do this. Set up Twilio so you can actually call up your agent on the phone on your way to Home Depot and figure out, like, what supplies you could actually get on the way. Sounds like you have a full backlog, Brian. I'm not sure whether you can get the best first or the home remodel. Totally. Okay. Awesome. Thank you, guys. We have some resources for you guys. I saw some questions in the q and a that, we are gonna now move into and answer live. So we have a good amount of time for that. But all the links you see on the screen, we will follow-up via email and send them to you, so don't worry too much about, about copying these down. So, let's get into the q and a. We have a few questions we selected. Perfect. Constitute. Okay. Okay. So the question is real time API has many configuration options. Which ones did you use for the demo, and how did you click the parameter values? Right. So some of the configuration options that you can choose from are, you know, which which real time model you wanna use. I for the demo, I used the most recent, June 3 one, which is a lot better in instructor following. You can choose which transcription model you wanna use. We use whisper one on this one. And, you can choose the, audio codec that you sent. In this case, we just, like, went with the highest resolution one because we're doing it live. Although, make if we're using phone, it wouldn't work through this loopback. And then also let's see. Some of the other parameters are, like, the VAD. Do you wanna talk about the VAD settings? Yeah. So we have a couple of different types of voice activity detection that, we support on the real time API. Previously, we had sort of a naive system, which just listened for audio or lack of audio, and that's how it detected the ends of turns. And I I believe that's called the default. I'm not sure. And then there's a newer, voice activity detection system, which is called Symantec VID, and that's where we actually take into account the the content of what is being said. And so if you say something along the lines of my name is and then give it a long pause, Symantec VAD should actually understand that you are not done speaking yet. And it'll sort of hold back the model for long enough for you to complete your sentence. There's also some other values, I guess, that we can talk about. Temperature is one which comes to mind. Yeah. How do we think about temperature for this model? So I think about temperature in terms of, like, how creative do I want it to be with my responses versus, like, how strictly do I want it to follow the instructions and the best practices that I set out. Mhmm. So if you wanna stick to a script pretty tightly, I would keep the temperature pretty low. But if you wanted to kind of, like, be better at ideation and and, creativity, I would set it a little bit higher probably than that. Yeah. And we do recommend for the real time models, using a temperature, I believe, between point eight and one point something. Yeah. There it's it's in the documentation, but, we don't support the full temperature range for the real time models simply because, it causes problems with the, with the audio tokens. Next question? Yep. Yep. Yep. I should refresh. Okay. Oh, a mobile app. The best way to implement speech to speech voice agent for a mobile app. Yeah. This is really interesting. I think, the fact that we have a WebRTC support is the biggest unlock here. It means that actually, do you do you wanna pull up the system architecture diagram maybe? And we can just, like, talk about how all the different pieces sort of communicate with each other. So, yeah, using the using WebRTC to connect your client device, which in this case will be the mobile phone, directly to the inference server is is probably going to be your best bet. This will eliminate, like, a network hop so you don't have to send traffic through your server. Another thing that'll it'll allow you to do is, like, significantly speed up the, the integration, time. I think we're not sharing the screen maybe. Okay. So, yeah, we'll we'll just pull up a quick, diagram which shows how the demo we saw today works. And then, hopefully, that can help you map on to how it might work in a mobile setting. So in this diagram, we can see sort of the users on the left, and then our application is everything else in the diagram. And what runs on the client is really the manager agent as well as the designer agent, which, Brian designed. And there's also some workspace state that lives on the device, or on the client. This workspace say state, of course, can be saved to a database, etcetera. We are not really representing that in this diagram. But there's a bunch of components that you can add to the server as well. In our, example, we have the workspace editor agent, which I think we exposed as a function called make workspace changes. And so this is really the 4.1 model, which is a smarter model. And it's it's sort of being delegated by the designer agent to make changes in the workspace. And it streams those two call calls back to our client, and we sort of apply them. The interesting thing to note here is that manager agent and designer agent are using real time API. And they directly connect the client, which is our browser in this case, to the real time API on the OpenAI inference servers. So there's no server, in in between the client and, the AI model when we're just interacting with the voice, which is how it's, like, so fast. Yeah. Yeah. So what that means is if you don't have anything too complicated, like we were showing before, you can hand off between different agents, the workspace agent, the designer agent directly. You can tell the designer, hey. Like, when you wanna make workspace changes, hand off to the manager agent. It can stay on the client and do its thing and then pass back to the designer. So you can kind of keep those, sort of, like, functional separations intact. Mhmm. One other thing is the client, gets an ephemeral token from the real time API that, is secure. Right? So you can set a TTL on it so it won't last too long and people can hijack it. And, and if you want to, you don't even actually have to build a server for the mobile app. You could just, like, have it run directly through the client. Nice. Is the reason we have the workspace editor on the server because, we don't really have that support for ephemeral tokens for these, like, text models? Right. Yeah. So the server, you would set up your OpenAI API key on the server, and then you could sort of, like, run it protected just like you would normally and then give it a lot more control, for, like, what it can do. And, you know, you could use you have full use of the responses API that way too where you can plug into, like, you know, image generation, MCP, and all sorts of things. Really cool. Okay. We can go to the next question. Pull it up. Sorry. The question. Great. So the question we have is, are speech to speech models useful in scenarios where you'd like to mostly do speech to tool calls? Let's see. Speech to speech models, I think so. You can, you can write, really just kinda like a phone based, application where you could call a phone number on Twilio and then have the real time model, make a whole bunch of, like, calls in the background. Right? And, like, just sort of, like, tell you what it's doing. So, I mean, one of the ideas I just, like, had on the spot is you could set up an agent to kinda, like, manage your home automation. Right? So you could call up your agent and say, hey. Like, can you, like, set my thermostat to, like, 78 degrees or whatever? Because I'm calling home 72 degrees. And so that way, you know, the the agent, kind of like using a responder thinker pattern, can't just, like, talk to you and then, like, hand off all of the complicated logic to the the tools, to do the complicated things. Yeah. I would also add that this is perhaps not the only thing that they're useful for. Right. I mean, there's a lot of interesting use cases, which I don't think really were, possible pre speech to speech models. The one that comes to mind, most prominently for me is language coaching. Mhmm. Yep. One of our customers, called Speak actually has, this role playing agent that they've built using real time API where they actually give you feedback on how you're pronouncing words. And it just helps you become better at a third language or second language. This is just something you couldn't do with the older class of models because you would just lose that pronunciation in the transcription phase. Right? Yeah. So I'd say maybe speech to speech models are also useful where these software aspects of, vocal communication are really important, where you really want to, be highly attenuated to the user's emotion, state of mind. Yeah. Yeah. And tools are just a way to give them more capability. I think that's why I picked the designer scenario for this. It's like, you know, it's a very sort of like an emotional thing. Like, I want I want my agent to, like, pick up, like, what I really care about while I'm redesigning my kitchen. I'm like, okay. I don't care about this. But I do care about that. So it it'll actually spend more thought cycles actually doing a better job based on how I'm talking. All of our products that have this load. It'll it also sort of mirrored your energy, you know, when you when you got super excited. It sort of responded in a more excitable way. And that that's also something that only speech to speech models can do. I I said it before, but I really got hooked on, like, working with my little workspace agent because it was just so fun. I just wanted to come back and, like, have it make more workspaces for me. We'll have to see, photos of your your remodeling. And sort of related, how important are the prompts when designing the agent? They're they're I I'll admit, like, that's a big part of it. Right? So, like, you have to be thoughtful thinking about, the tone and, you know, like, the patterns you wanted to use. You can put in your own sort of, like, branding guidelines into it. So, like, you know, if you want your agent to, like, represent your brand, you can, like, have a a voice prompt derived from, like, what your sort of, like, writing tone is, but in voice mode. And then also, you know, like I was showing you, running the conversation for us. So what's important to realize about real time voice models is they're generating tokens just like text models, but it's it's it's audio tokens. And there's a lot of them. And so, one of the reasons why it's a little bit harder for, voice agents to follow instructions right now is because they're just doing a lot more than text models. So, you can get a lot out of real time voice agents with the right prompting. And, like, starting with that meta prompt is, like, a really great place, and then, like, building in these, like, state machines and making it easy for the agent to keep track of where it is and offload a lot of the the thinking to, like, something else if you can. Yeah. I did notice your prompts were, like, super long. Often when I go into the playground and I'm playing around with the real time API, I feel like I just pop in, like, a sentence or two. And, you know, that's how I experiment. But I believe in practice, what our teams really recommend is prompts that are hundreds of tokens long, especially when you want to give the the agent a lot of personality. You really wanna be hyper detailed and experiment quite a bit. Yeah. And and, you know, a lot of, like, one shot and then shot examples too for certain types of conversations that you're running. You know, it'll, like, help it give it a guideline for, like, how to run certain scenarios if it gets lost. Yeah. So this agent so the question is built as agent of language as use of code. This was written in TypeScript. I was, the this platform is written on, Next. Js. So you can run a fully client real time agent on the client side, but then call back to server APIs on the back end to do more of the complicated responder thinker pattern, implementations that we're doing. And what's really great is, like so the agents SDK has been out for a little while for Python. I don't use Python as much as maybe I should, but, you know, I'm kind of like a JavaScript Next. Js person. So as soon as this came out, like, last week, immediately switched over to it, and it's been a blast. It's also really great that, I mean, if we think about WebRTC, it's really a technology that's built for the web, and it's built for browsers. And so it's just got this this seamless integration with JavaScript. Right? So it makes so much sense that we have, real time and, TypeScript SDK. Totally. We'll be yep. So we'll be able to pick up the tone of the caller. Totally. That's that's the whole thing about real time voice agents is, you know, they can tell if you're excited or angry. In fact, like, that's actually a really big useful I mean, we didn't go into, like, customer support use cases here. The real time agents, code base has a lot of customer support examples, and I highly recommend checking those out. But, yeah, you could, give the function give a function to your agent to call, say, for example, if the user is angry. It detects that the user is angry and have it say, I'm sorry. You're having a hard time. Escalate to a person. Right? So then it could, like, hand off to a human in the loop or maybe it, like, start over some different workflow to make the user, feel better about the interaction that's happening. Makes sense? Awesome. So I think that's all the time we have for for questions for this session. But we really appreciate all the questions. We after every build hour, we'll send out a a survey form. And so if you have additional questions or any suggestions for upcoming topics, feel free to, to put input that there. Our next bill there is going to be July 16. So be sure to use the link and sign up, and join us next time. So thank you again, and, we'll see you June 16.