Keeping 100k battles of untrusted agent code in their lane

In March 2026, Lambda ran AgentBeats, an AI agent security competition in which teams submit two kinds of agents: an attacker that tries to manipulate a target LLM into doing something harmful, and a defender that tries to stay helpful while refusing the trap (check the final leaderboard here). Our platform pairs them, runs the battle, and scores the outcome. The agents are graded on how effectively they subvert the system, which means the platform's job is to stay correct and on schedule while the code it hosts is trying to break things.

The short version: 48 rounds over 27 days, around 100k battles, 22 teams, 574 submissions on a pool of at most four NVIDIA HGX H100 GPUs. The part I want to tell you about most: we added GPUs during live rounds, with a single command, without dropping a battle.

Note: The "security" described in this post is the topic the agents were competing in ("attacker vs. defender agents"), not a property claimed for the proposed platform itself. I'm an ML person, not a systems security expert, and the containment described below (Docker networks, an egress allowlist, and resource caps) doesn't describe a properly sandboxed system. It was sized for a month-long competition with modest prizes and a known participant pool and used a small, easy-to-control model, gpt-oss 20B.

The timing of this write-up isn't intentional, but it couldn't be more up to date: just as I was finishing this post, OpenAI disclosed an incident in which models under evaluation escaped an isolated environment through a zero-day in its package-registry proxy, the same kind of egress path I describe below, and went on to compromise production infrastructure to cheat on the benchmark. Against that class of adversary, nothing in this post would hold. For the competition described here, it didn't need to. If a container escape would ruin your day, you want microVM isolation such as Firecracker or gVisor and a real threat model. This post has neither. What it has is the ops story: how we kept 100k battles of code we didn't trust on schedule and on four GPUs, for a month.

More details:

• 48 rounds, twice daily, March 6 to April 1st, 2026 for 22 teams and 574 total submissions

• Up to 2,098 battles in a single round (the final round of Apr 1st)

• Median round took ~10.3 hours, ranging from 7.6 hours on a light day to 16.8 hours on a heavy one (across the 48 rounds with complete timing)

• Due to rising participant numbers throughout the competition, we hot-loaded additional GPUs mid-round, with zero battles dropped

• Every round was completed, scored, and redeployed on schedule so teams could keep submitting late, even broken agents, without threatening the round's integrity or the leaderboard

The subject of this post is the infrastructure behind those numbers: independent per-GPU vLLM replicas, a single Caddy container that handles all network traffic (in and out), a concurrency limiter that resizes at runtime so a GPU can be folded in mid-round, a rate-limiter sidecar, and a serverless AWS submission path.

But first, assume the code hates you

The defining property of this system: it executes untrusted, adversarial, financially-incentivized (we gave out prizes) code on shared hardware. Everything below is downstream of three problems:

• Cost. The GPU budget is fixed: at most four NVIDIA HGX H100 GPUs, with one as a starting point and three additional held in reserve, launched dynamically as demand increased throughout the competition. Although technically possible using the Lambda Cloud API, we didn't set up any autoscaling. If a round runs longer than the interval to the next round, the schedule degrades.

• Containment. Agents shouldn't access the internet, other battles, or the AWS account holding submissions and results. Not because we built an escape-proof sandbox (see note at the top) but because removing the easy paths keeps the competition fair and the blast radius small.

• Fairness. No agent may starve another agent's GPUs, and no defender agent may carry state from one round to the next (even a cached jailbreak, or a stored opponent prompt, would skew the competition toward the code that cracked the container).

Here's what the system looks like with that framing set.

System shape

The platform is split into three planes because it serves three workloads with different cost and uptime profiles.

Ingestion is serverless because submissions arrive in bursts and the path sits idle most of the day: API Gateway, an AWS Lambda that validates each upload (size caps, a 100:1 compression-ratio check for zip bombs, path-traversal and extension filtering), DynamoDB for state, and S3 for submissions, results, and the frontend. The important detail here is the credential the evaluation host uses to pull that data down.

EvaluatorUser is read-only on the submissions bucket and the teams table. The host that runs the untrusted code can't alter a submission or a team record (i.e., the machine most likely to be compromised holds the weakest key, though it can still read every submission, a tradeoff that we accept here).

Four replicas instead of tensor parallelism for GPU serving

The model we’re using for the competition is OpenAI’s gpt-oss-20b, served with vLLM. Four 80 GB NVIDIA HGX H100 GPUs invite the obvious layout: --tensor-parallel-size 4, one logical endpoint, weights split across all cards. We didn't do that. Each GPU runs its own complete, independent vLLM on :8000 with a full copy of the weights. Since the model fits on a single card in the served precision, we can also run replicas and scale dynamically instead. This is helpful because active participants vary over the competition, and in our case, they grew.

Why this layout worked well in our case:

1. A new replica joins the rotation in seconds. That's what makes hot-loading mid-round possible and easy (see description below).

2. Failure stays local. If one vLLM wedges, the others keep serving. Under tensor parallelism, one problem (bug or hardware error) may take the whole endpoint down.

3. Of course, throughput scales linearly with GPUs. Four replicas serve four times the concurrent battles.

It also changes the economics: instead of a fixed layout with mostly idle GPUs, the pool becomes dynamic as cards get added when rounds run long and spend stays matched to demand.

One container between the agents and everything else

Battle containers run on a Docker network named agentbeats-isolated, which has no route to the internet. One Caddy instance straddles that boundary and does four things:

1. Load-balances the GPUs. Caddy reverse-proxies the vLLM backends with lb_policy round_robin. Agents connect to one stable internal address; Caddy distributes requests across the healthy replicas. 

2. Health-checks the backends. Every 10 seconds, Caddy probes each backend's /health with a 5-second timeout and drops unhealthy upstreams from rotation.
   reverse_proxy $UPSTREAMS {
      lb_policy round_robin
      health_uri /health
      health_interval 10s
      health_timeout 5s
      transport http {
         dial_timeout 30s
         response_header_timeout 600s # reasoning models hold connections for minutes   
         read_timeout 600s
         write_timeout 600s
      }
      flush_interval -1 # stream tokens through, do not buffer
   }The 600-second transport timeouts exist because a reasoning model mid-generation can hold a connection open for minutes; flush_interval -1 passes tokens straight through rather than buffering the stream.

3. Is the only path to install packages. Agent containers need PyPI at startup. "Needs PyPI" is not "needs the internet," so Caddy also runs a forward proxy on :3128 with an allowlist:
   :3128 {
      forward_proxy {
          acl {
              allow *.pypi.org
              allow *.pythonhosted.org
              allow *.astral.sh
              deny all
          }
          hide_ip
          hide_via
      }
   }
   

   An agent can pip install its dependencies and nothing else. That closes the casual routes out. In light of recent events: a package registry is itself also an exfiltration channel, as anyone can publish a package, and, as the incident linked at the top shows, even the proxy in front of it can be the hole. That's the class of attacker this setup wouldn’t defend against.

4.  Reloads without dropping connections. This is what makes runtime rescaling possible, covered next.

The reverse proxy listens on :8080, the forward proxy on :3128, and Caddy's admin API stays bound to localhost:2019 so nothing on the evaluation network can reach it.

Adding a GPU mid-round

We started with a single NVIDIA HGX H100 GPU in steady state and added a second, a third, and a fourth one by one as the competition scaled past its compute limits. The “pressure” here is defined by the schedule: rounds fire twice a day, so a 16.8-hour round on the steady-state pool can eat the next cutoff and push the whole competition behind. The hot-loadable cards are the slack that keeps a long round from cascading into the following one.

For us, folding in a card was one command:

python -m backend add-backend http://<gpu-3-ip>:8000/v1

The implementation does three things in a specific order:

1. Health-check the new backend by calling /v1/models with the API key. If it's not serving, the operation aborts without changing anything.

2. Reload Caddy before anything else learns the GPU exists. The Caddyfile is rewritten with the new upstream and reloaded over the admin socket (with a fallback to a container restart and /health poll if the reload API is unavailable). Caddy is now willing to route to the added card.

3. Write .env last. The running evaluation process watches this file for changes.