KubeCon + CloudNativeCon North America 2026: Taking Agents to Production

That’s a wrap for my KubeCon + CloudNativeCon North America 2026 in Salt Lake City. I’m heading home a little early this time to make it back for my 40th birthday tomorrow, Friday the 13th.

It’s been an intense few days: co-chairing Agentics Day: MCP + Agents on Monday (my recap is here), moderating a panel on agents on the platform, and giving a talk with William Rizzo on platform guardrails for AI agents.

If one topic kept coming up all week, it was this: how do we actually take agents to production? Not just getting them to work in a demo, but giving them an identity, limiting what they’re allowed to do, understanding their decisions, and keeping the systems they touch safe.

A lot of these questions sound familiar. We’ve spent years solving similar problems for cloud native workloads. Now we’re figuring out what changes when the workload starts making decisions on its own.

Taking agents to production

Tuesday was pretty much all about agents. In the afternoon, I moderated the panel “Agents on the Platform: Four Perspectives on the Conversation We’re Already Having in Private” with Joseph Sandoval, Alexa Griffith, Juraci Paixão Kröhling and Jed Salazar. The idea was to bring four perspectives together: platform engineering, AI infrastructure, observability and security.

Interestingly, there wasn’t much disagreement. We came at it from different angles, but kept coming back to the same questions. Who owns the agent? What identity does it have? Where do you enforce the boundaries? And how do you understand what it’s actually doing?

Both Juraci and Joseph argued that tracing is more important than ever for understanding what’s going on. So if you’re considering bringing agents into the mix but aren’t doing tracing yet, maybe start with the platform and the fundamentals first.

I think that’s quite telling. We’re starting to agree on the problems, even if we’re still figuring out how to solve them. And we all agreed that agents probably aren’t going to kill us anytime soon.

Agents are changing how we think about compute

Jonathan Bryce opened Wednesday’s keynotes with a framing I keep coming back to: we live in a world built for human speed, but more and more of it now runs at agent speed. One slide showed how Claude Code sessions changed over six months. Autonomous actions per session more than doubled, from 9.8 to 21.2, while human turns went down.

The effects reach well beyond Kubernetes. Wikimedia’s caching was built around how people behave, like everyone looking up the same candidates before an election. Agents don’t follow those patterns, and according to the keynote, 35% of page views now generate most of Wikimedia’s resource-intensive traffic.

Agents don’t only generate different traffic. They also run differently. They spend a surprising amount of their time doing nothing, waiting for a model, a tool, another agent or a human. Yet we still allocate compute as if they were running all the time.

On Monday, Alex Van Boxel showed how Agent Substrate suspends idle agents and resumes them when there’s work to do. It kept coming up all week. On Thursday, Google announced Managed Agents on the Gemini API, building on Agent Substrate 1.0.

ScaleOps came at it from the cost and reliability side. An agent waits, then suddenly needs a lot of CPU to run what the model told it to do, so you size it for the spike and waste the CPU the rest of the time. When it crashes mid-step, the step runs again, and you pay for the tokens twice. And the services it calls weren’t necessarily built for that kind of traffic.

It made me think that scaling agents isn’t really about adding replicas. It’s about scheduling, isolation, suspend and resume, and protecting the services agents talk to. Familiar distributed systems problems, with a different execution pattern.

Six months later, what did the agent do?

What happens when an agent makes a decision, the pod disappears, and six months later someone asks why it did what it did? You can’t just run it again. The same prompt might give a different answer, and whatever the agent knew is gone with the pod.

Jorge Palma from Microsoft took on that problem in Thursday’s keynotes. Microsoft open-sourced Agent Sessions, which writes each model completion to a hash-chained, append-only journal. You can replay a session without calling the model, fork it, and check that nothing has changed. The session also outlives its pod, so the compute can be released while the agent waits for a human. Alongside it, AgentSuite packages an agent’s definition, including instructions, tools and evals, as a versioned OCI artifact.

I think it’s worth separating recording an agent’s execution from understanding it. A journal of model completions helps you reconstruct what the agent saw and said. An OpenTelemetry trace shows how it interacted with tools, platform components and other services along the way. You probably want both.

The platform should carry the complexity

On Wednesday, William Rizzo and I gave our talk, “Verticalizing Platform Intelligence for Regulated Domains”.

The idea behind it was simple: why reinvent security, governance and observability for agents when we’ve spent years building those capabilities into our platforms?

We showed it with an insurance claims workflow where agents process claims and execute payments. Instead of trusting the agents to follow instructions in a prompt, we made the platform enforce the boundaries. A prompt can say “never pay out more than €6,000 without a human”, but if the agent holds a tool that can pay any amount, the prompt is only a suggestion.

Using SPIFFE/SPIRE, Linkerd, agentgateway, OpenFeature and Kyverno, we gave the agents identities, restricted which tools and models they could reach, and controlled how much autonomy they had. Small payouts went through automatically, larger ones needed a person, and the agent couldn’t change its own limits.

The other part was understanding what actually happened. With OpenTelemetry, we traced the whole workflow across HTTP, MCP and A2A, including every policy decision along the way. That gives you strong evidence of what the agent did, as long as you record, keep and protect it properly. AgentEvals then read the traces and checked whether the agents behaved as expected.

That’s the point I wanted to make: we don’t need to start from scratch. Most of what we need already exists in the CNCF and Agentic AI Foundation ecosystems. There’s still plenty of integration work to do, but we have a good foundation. The slides and demo code are online, and I’m hoping to do some deeper dives into the individual technologies and how they fit together.

Some of the end-user stories showed the same idea from the other side. CVS Health measures its platform against a simple question: can a developer hired today ship an application to production on their first morning? Their answer is governance built into the platform, so developers, and agents, are compliant just by using it.

The Access Group told a story I suspect many teams will recognise. They tried letting AI write Terraform and OpenTofu across repositories, and it didn’t hold up at scale. So they put APIs in front of safe, trusted capabilities, and their app builder, now running 5,000 apps, calls those instead. Salesforce made a similar point about Heroku, which has an OpenTelemetry Collector built into the platform, so every builder sees the full journey of a request out of the box.

The platform carries the guardrails, so the person or agent shipping doesn’t have to.

Lin Sun’s drone demo on Thursday brought all of this together, and it was a lot of fun. In Amsterdam in March, her drone flew, but the camera never streamed to the big screen. This time, with pilot Cory Jett, the camera worked, and a team of agents on Agent Substrate and kagent turned footage of the audience into a short film of us all flying.

Halfway through, the demo hit a rate limit in agentgateway. Lin asked her platform agent to fix it. It opened a pull request, the room reviewed and approved it, and Argo CD synced the change. Afterwards, she showed the whole sequence in Jaeger, including the 429.

My favourite part was that the agent didn’t get to make all the decisions. It could ask for a change, but the platform still controlled what was allowed, and with OpenTelemetry we could see everything afterwards. That’s much closer to what running agents in production should look like.

If you can’t understand it, can you govern it?

One of the most practical talks I attended had nothing to do with agents, at least not directly. Regina Voloshin (Octopus Deploy, Argo CD maintainer) and Dag Bjerre Andersen (Egmont) talked about “Health Check Hell”: how do you actually know whether a Kubernetes resource is healthy? It sounds simple, but anyone who has worked with custom resources knows it isn’t.

The spec of a custom resource is a contract the API server validates. The status is whatever the author chose to return, and it’s rarely documented. So every tool that needs to know about health, like Argo CD, Flux, kro and Crossplane, has its own way of finding out. Argo CD carries hundreds of user-contributed Lua health checks its maintainers can’t verify, and kstatus, the attempt to standardize this, has seen little adoption.

Dag’s prototype, kubehealth, returns three dimensions instead: availability, reconciliation and lifecycle. “Available, failed to reconcile” tells you far more than “degraded”. Regina’s favourite answer to who should own the logic: each controller writes health into its own resource’s status.

I think there’s an interesting connection to agents here. A platform can only act on health, safety or compliance signals if someone has agreed what they mean. That’s true for a CRD’s status, and it’s just as true for the spans and policy decisions an agent leaves behind. It’s also why, in our talk, we defined our own telemetry attributes as a contract and checked them with Weaver.

Can we prove it?

Something I didn’t expect to discuss as much as I did was formal verification. I didn’t have booth duty this time, so I spent more time in the hallway track, and it came up in several conversations.

I remember learning about formal methods at university, but I never really found a practical use for them in the world of microservices and Kubernetes. Now, with agents making more and more autonomous decisions, I’m starting to wonder whether that’s changing.

We might not be able to predict what a model decides to do next, but perhaps we can establish stronger guarantees about what the system around it will allow. Traces tell us what happened. Formal verification could tell us what can and can’t happen, under stated assumptions. The two complement each other, and it’s something I’d like to explore further.

Built by humans

My favourite keynote of the week was Jeffrey Sica’s on Thursday morning. Jeffrey is Head of Projects at the CNCF, and he opened with the story of Final Fantasy, as he told it: a small studio in Tokyo, running out of money in 1987, decides to make one final game, the one it really wants to make. Forty years later, there are 16 mainline games and around 100 spin-offs and remakes.

He used it to push back on the idea that this is the last chapter of engineering. Every maintainer in the room recognised the pull request he described: 1,500 lines of confidently described code, written by something that didn’t understand the codebase. But maintainers didn’t close the door. They wrote down what a good contribution looks like, and what they expect from the human on the other side.

He also argued that cloud native was AI native before anyone had a word for it. A small blast radius, fast rollback, sandboxing untrusted code and an API for everything were rules we wrote for unreliable machines on unreliable networks. They turn out to be the right rules for an unreliable caller that fires off 21 actions at three in the morning.

At the end, he came back to Final Fantasy. The games have changed enormously over the years, from combat systems to settings and art. But the series is still recognisable, because the things people love about it stayed.

I think that’s a nice way of looking at cloud native. We’ve gone from containers and Kubernetes to platforms, AI infrastructure and agents. The technology keeps changing, but the principles, and the people behind them, remain. I also liked his take on the landscape: not a huge map of projects, but groups of passionate people. A lot of people put a lot of hours into this ecosystem, and the rest of us benefit from it.

His closing message was simple: be patient, and get involved. I liked that. There’s a lot of uncertainty around AI right now, including what it means for us as engineers, and it can feel overwhelming. But we’re still the people building these systems, and we get to decide what good engineering looks like.

Giving back ran through the other keynotes too: GPU capacity from CoreWeave and Nvidia for project CI, open source credits from Vultr, and 3,000 Claude codes from Anthropic for CNCF maintainers. Jorge Castro handed out the community awards, with Lifetime Achievement awards for Julius Volz, co-creator of Prometheus, and Rita Zhang.

The TOC used its keynote slot for a small play about a software update that goes wrong twice, to pitch rollback readiness: projects testing and documenting upgrade, roll back and upgrade again. With all the talk about agent speed, it was good to see the basics of safe change on the main stage.

Elsewhere at KubeCon

Seven projects graduated this year: Buildpacks, Karmada, Kubeflow, KubeVirt, Kyverno, OpenTelemetry and NATS. Kubeflow celebrated on stage and launched an agents working group. llm-d reached 1.0, and KubeRay joined the CNCF as a sandbox project. And Google and Meta showed GKE Frontier, which cut the time from an upstream pull request to GKE from about 11 weeks to around a week.

The people and the hallway track

The best part of KubeCon is always the people. On Tuesday, at the Dash0 booth, I finally met an old friend in person for the first time: Ray Tsang. Ray was one of the early people doing cloud demos with Raspberry Pis and Kubernetes, and he helped out immensely in 2016, when Martin Jensen and I built a Raspberry Pi Kubernetes cluster for our master’s thesis. We’ve chatted on and off for years, so it was great to finally meet him.

I also managed to escape the conference center for a bit on Wednesday afternoon and walk up Ensign Peak. Fresh air, sunshine and a great view of Salt Lake City.

In the evening, Dash0 co-sponsored the escape room party with Buoyant and a few other companies, and I ended the night at Hive Minds by Isovalent, where I got this cartoon drawn of me.

Wrapping up

Looking back at the week, I think we’re finally moving beyond asking what agents can do, and starting to have more serious conversations about how we actually run them in production. I like that.

Identity, security, orchestration, observability and governance aren’t new problems. We’ve been working on them in the cloud native ecosystem for years. Agents add new challenges: they make decisions at run time, interact with systems in ways we can’t always predict, and make us think differently about autonomy and trust. But I don’t think we need to reinvent everything. We have a good foundation already. Now we need to connect the pieces and make it easier for platform teams to support this new kind of workload.

And perhaps that’s what I enjoyed most about this KubeCon. We’re still figuring things out, but a lot of talented people are working on these problems together.

A big thank you to everyone who joined our panel, came to our talk, or stopped for a conversation in the hallway. Now it’s time to catch my flight and head home to celebrate turning 40 with my family.