Giving AI The Keys to Your App: Agents Controlling UI State

Kewin Wereszczyński
No items found.

In short

In this post, we explore how to delegate UI state management to AI agents, enabling them to autonomously control an application’s flow and logic outside of React components. Using Jotai for external state management and Flows AI for orchestrating decision-making agents, we build an interactive AI-driven game based on the classic trolley problem, demonstrating how AI can dynamically interact with UI elements and modify application state in real-time.

The crucial part of this project is moving the flow and logic responsibility outside the React component, and letting agents control the UI state autonomously.

Ever since learning about agents and workflows, I have been asking myself how I can seamlessly integrate those AI features into an application. So, after brainstorming, I came up with the idea of making a small game in which AI agents are the participants.

The game is rooted in the trolley problem, a well-known ethical thought experiment, where the train is coming to a junction and has to be directed to the left or right side. There are possible casualties on both sides, each of them tied to moral implications. In this version of the game, AI agents are responsible for presenting the case to the judge agent to spare their side of the tracks, so you can be sure that nobody was harmed.

Creating UI

The UI is very simple, so instead of going through the steps of writing it, I’ll show you how it ended up looking and explain the idea behind connecting it to its AI brain. The UI is written in React using shadcn/ui as a components library to speed up the process.

Simplified UI and animation flow

There are three main sections: victims, debate, and players.

Preparing state

The most important part was to get away from any internal state and move that responsibility outside of React so all the components are only communicating with the external state. My favorite tool is Jotai Atom, but any other state management library will work fine as well. After that, I set up a custom “Store” for the atoms so they could be modified outside the React render tree.

With that setup, let’s shift the focus to the AI part.

Creating agents

The main AI libraries this project will use are flows-ai and vercel/ai. Flows-ai provides a set of utilities and helpers to speed up agents' creation and compose them into reusable and serializable flows. This means that once we figure out the game flow and create agents, we can quickly try out different configurations without writing any complex loops or conditions. Let’s start with the main judge agent, as their logic will be singular, independent of the side.

In the simplest form, that is an agent able to run in the flow. But we also need to give it the ability to modify the state of the UI.

Here is the tool that allows the agent to announce the verdict, and show it on the screen. It even allows for a short dramatic pause after the final speech (very suspenseful). Now that we have that function, let’s put all those parts together into the agent.

In this step, it’s important to make the judge use the tool instead of just returning a response without taking any action. The judge will get information about the arguments and possible casualties in the prompt when the flow is running, so for now, it’s ready. Time for the players!

Each player needs to be able to plead their case, so the argument will be visible. It also needs to be aware of the previously provided arguments, to not use the same ones over and over. To achieve that, let’s take advantage of the flexibility of flows-ai.

After making sure that the function returns an agent that conforms to the interface that flows-ai expects, we can now pass the “outside” state into the agent context and modify the state after we get the response. Now that the main agent can access and modify the state, let’s abstract away the creation of its personality.

It looks like all the AI participants are ready. Let’s finish up the project by creating an easily modifiable game flow!

Creating flows

Let’s start simple, by establishing the initial flow of one argument for each player and a verdict.

Now we have a flow that will run a sequence. The first step is running player agents in parallel (because time is of the essence here). Then, the results are taken and given as context to the judge agent, which is the second step of the sequence. With that flow out of the way, let’s create the agents and run the flow.

That’s it! Now our players will display their arguments on the screen, add their responses as a context for the judge, and the judge will give the final verdict.

Now that I have shown that flows-ai is easy to set up, let me also show you that it’s flexible. Let’s start by making it more exciting by providing three arguments each instead of just one.

Since flows-ai is fully serializable, it is very easy to create helper functions for modifying the flow. In that scenario, our players have context passed through atoms, but if they didn’t have atoms, they would still receive their previous responses as “context”.

So, with that one easy change, we now implement three steps for arguments, and after them, the judge makes the final call. But what about showing buttons to control the game or the overall game state? We can also modify the execution of the flow to be even more flexible.

Thanks to the fact that each flow has its own separate name, we can make a couple of simple “ifs” to control and react to each step of the flow. The “ifs” can be as granular or as complex as needed.

In the end, I introduced a couple of small name changes and improvements to the UI, but here is the final result:

Final app UI

You can fetch the repo and play with it by yourself. There are plenty of amazing flows that you can easily implement with flows-ai. It integrates seamlessly with Vercel’s AI library, so push your imagination to the extreme, and have fun with it!

Latest update:
February 4, 2025

FAQ

No items found.
React Galaxy City
Get our newsletter

By subscribing to the newsletter, you give us consent to use your email address to deliver curated content. We will process your email address until you unsubscribe or otherwise object to the processing of your personal data for marketing purposes. You can unsubscribe or exercise other privacy rights at any time. For details, visit our Privacy Policy.

Callstack astronaut
Download our ebook

I agree to receive electronic communications By checking any of the boxes, you give us consent to use your email address for our direct marketing purposes, including the latest tech & biz updates. We will process your email address and names (if you have entered them into the above form) until you withdraw your consent to the processing of your names, or unsubscribe, or otherwise object to the processing of your personal data for marketing purposes. You can unsubscribe or exercise other privacy rights at any time. For details, visit our Privacy Policy.

By pressing the “Download” button, you give us consent to use your email address to send you a copy of the Ultimate Guide to React Native Optimization.