Toss the Keys & Don't Watch the Catch: How We Let AI Merge Dependabot PRs Overnight

Another flaw in the human character is that everybody wants to build and nobody wants to do maintenance.

– Kurt Vonnegut

Do you love Dependabot?

Strange question, maybe. Let me narrow it: do you love Dependabot pull requests?

You know the pile I mean: it's Monday morning, and twenty of them are waiting. Each one a single patch bump, each one green, each one needing a human to open it, squint at it, and click the same three buttons as last Monday. Nobody reviews the seventeenth one as carefully as the first. Some teams just stop looking, and the pile quietly becomes a long vertical scroll that you hope your Security team doesn't notice.

We built something that deals with the pile. It's called the Dependabot Janitor, and every night it looks at open Dependabot Pull Requests across our GitHub organization, merges the boring ones, and hands the interesting ones back to humans with receipts. It has merge rights, it runs unattended, and it's powered by an LLM.

I'd like to tell you why that last sentence doesn't scare us - because making it not-scary was most of the work.

The Scary Version and the Useful Version

The scary version of this project is: an AI agent can merge PRs overnight, by itself.

The useful version is smaller. The Janitor only merges

1. routine patch and minor bumps,
2. when the required checks are green,
3. and the repository has a real, regression-catching test suite

A dependency bot that merges everything is a liability. A dependency bot that only merges what your repository can prove is boring is a useful coworker. The narrow promise is the product, and the engineering is mostly about making the promise structurally impossible to break - not about making a prompted model better, safer, or smarter.

Major version bumps illustrate the spirit: the Janitor will investigate a major - read the changelog, identify the breaking changes, check whether this specific codebase touches any of the affected surface - and it will leave a reasoned recommendation, and it may even approve the PR. It still will not merge it. Majors are where "probably fine" most often hides an incident-to-be, so the merge button stays with the team.

Standardize Before You Automate

A confession before the architecture: the Janitor was not step one.

Step one was boring. Dependabot's configuration lives in every single repository, which means every repository teaches Dependabot a different dialect: different grouping, different scheduling, different title formats, security majors bundled with dev-dependency patches... or split out! You'll struggle to automate the handling of PRs that have no predictable shape.

So first we built centrally-managed Dependabot configuration: one canonical config per package ecosystem, owned by the platform team, distributed automatically to any repo that opts in. Grouped patch bumps, isolated majors, predictable titles. Cooldown.

Proof-positive (or confirmation bias?) that we were on the right track: GitHub themselves dropped a blog post about Dependabot hygiene recently - exactly a month to the day after our internal Dependabot configuration management went live!

Repos opt in with a repository topic and never touch the file again.

That project was pitched internally as "you can stop hand-tuning Dependabot config." True! But it was also a setup. Once the PRs have a predictable shape, a machine can triage them. And that machine(ry) was coming!

The Hard Line

The Janitor is two components with a hard line between them.

The first half is a small Google Cloud Run Function, and it is aggressively deterministic. Every night it runs one org-wide search for open Dependabot PRs, then applies the rules:

• Is this repo opted in?
• Is this PR opted out?
• Is this bump within the repo's configured semver ceiling?
• Is it mechanically mergeable (no merge conflicts, etc.)?

Plain field comparisons - cheap, fast, and covered by unit tests. Survivors get bundled per-repository, enriched with what the function can resolve about the repo's CI test plan, and handed off.

The second half is the judge: a Claude Managed Agent that receives one batch per repository per night and works through a fixed sequence of Agent Skills:

1. assess-test-suite: is the CI actually protective here?
2. check-if-can-merge: can this PR be merged mechanically?
3. check-if-should-merge-major: is this a major bump that should be merged? Recommendation only.
4. diagnose-failing-ci: (optional) - if CI is red, is the bump to blame, or something else?
5. execute-batch-actions: Label PRs, post comments, and merge what's safe to merge.

You absolutely could have an agent do both halves of it. In fact, that's how it started, as a /merge-dep-prs skill just on my machine. We didn't ship that out at scale, and the obvious reason is cost and speed: a field comparison in TypeScript is free and fast, and the same comparison performed by a frontier model is neither.

The less obvious reason is more-important: every¹ deterministic decision is made before the agent session exists. The agent cannot decide a repo is in scope, because scope was resolved before it woke up. It cannot widen its own semver ceiling, because the PRs above the ceiling were filtered out of its input. An agent can be sweet-talked, confused, or just wrong - but it cannot reason its way around a rule it never saw. Determinism here is a containment mechanism, and only second a cost optimization.

What's left for the model is judgment, the stuff that would be really hard to build in normal software:

1. Does this repo's CI actually protect it? Not "is there a test directory" or "what's the line coverage %?" - would the checks that gate PRs catch a regression in the code this bump touches? Codebases can express that a dozen or more ways, and you've got to understand what the codebase is supposed to do, to know that.
2. Is this major bump safe for this repo? That's a changelog, a breaking-change list, and a codebase, walked together.
3. Why is CI red, and is the dependency to blame? That's reading job logs and attributing failure honestly, including "this was already broken before the bump." (I'm looking at you, npm audit!)
4. What should the comment say? The output is prose a human will use to make a decision.

Each one is a research task over unstructured input, through an unknown path discovered through exploration, with a synthesized conclusion at the end. That's what the model is for. Everything else stays in the function.

No Tool, No Power

Now the part security folks have been waiting for: what can this thing actually do?

The agent never holds a credential as a string. Its only routes to GitHub are:

1. a deny-by-default tool allow-list - a curated MCP toolset where the authenticating token is injected server-side, after the request leaves the sandbox, and
2. a read-only clone of the repository, mounted into its container.

The allow-list contains core capabilities like:

1. approve a review
2. squash-merge a PR
3. post a comment
4. apply labels

And a few similar PR-shaped verbs for reading CI statuses, other comments, etc.

What it doesn't contain matters more:

1. There is no shell access to a gh CLI (it's not installed)
2. There is no push-a-file tool of any kind

The agent cannot push code, even though the token that backs its tools carries scopes that theoretically could. The boundary is enforced by the tool surface, not by the credential, and not by an instruction.

The sandbox means it, too: inside the container, only MCP tool calls and git operations against the mounted clone can authenticate at all. Anything else - a shell script, a stray curl - is an arbitrary process, and arbitrary processes get no¹ credential. We didn't build any of that auth framework; it's how Claude Managed Agents' Vaults work. Restrictive, but nifty!

Filter, Declare, Validate

The Janitor's most expensive possible failure is easy to name: silently merging breaking changes across the org overnight. So the rule - never auto-merge a major version bump - is enforced three times:

1. Filter, deterministically. Each repo sets a semver ceiling, and the function drops PRs above it before the agent's input is ever assembled. This is the strongest guard of the three, and the reason is specific to LLMs: absence of information beats any instruction. A model can be talked past a rule, or pay it half attention on the wrong night - but it's much harder to hallucinate a decision about a PR that was never in its input.
2. Declare the rule in prose. For the majors that do arrive (investigation is a feature - we want the recommendation), the system prompt and the investigation skill state the invariant outright, and the investigation skill's only possible outputs are "recommend" and "hold." There is no path from it to a merge.
3. Validate the rule in prose, at the write boundary. The batch executor independently refuses to pass any PR marked as a major to the merge tool - whatever category it arrived with, however the upstream reasoning described it.

Why say the same thing twice in prose? Because prose is the weak material here. Models follow instructions well and keep getting better at it, but you cannot trust any single sentence to be paid full attention every time, on every run, forever. So the invariant is declared where the decision happens and validated again where the write happens. The two prose guards rhyme on purpose - repetition at the boundaries is how you emphasize a rule to a reader who might be skimming.

Deliberately Rude About Tests

Did you think that just because the Janitor wasn't sending a major version bump through, you were safe? Why would you think that?

Auto-merging any dependency bump is only safe if you have some check that would catch an unsafe bump. So before the Janitor merges anything in your repo, of any bump level, it assesses your test suite - and it is deliberately harsh about it.

Not "does a test directory exist." The question is whether the checks that gate pull requests would catch a regression in the code this bump is part of. A test suite that exists but runs on main instead of PRs fails that bar. A test command that's decorative fails it. And - the one that stings - tests that gate PRs and pass honestly but don't honestly exercise the codebase, fail too.

That last one is where the false confidence lives. Here is (lightly sanitized) what the Janitor said about one of our own services - a repo with a real, PR-gating pytest suite, green checks, the works:

Sit with what the humans saw on that PR: green checks, a familiar test suite with hundreds of passing tests, a two-line version bump. Approve.

The tests that passed were communicating a true message about the code they covered... it's just that coverage clustered where tests were easy to write, and the modules that actually touch the database, the ones this bump lands on, had none. Nobody (almost nobody?) audits that on a routine version bump; a human reviewer's confidence too-often comes from the color of the checks. The Janitor reads everything on every PR it considers, because its permission to act depends on the answer.

This makes the Janitor stricter than some of the humans it replaces, and that's the point. The goal is to automate only the cases where the evidence makes automation boring - imitating the fastest human reviewer was never on the roadmap. When a repo wants the Janitor to merge more, there's exactly one way to get it: make the repo easier to trust. Fix the trigger, add the missing tests, and the bumps flow.

The assessment is also scoped, because monorepos are a common case and one well-tested package sharing a repo with one untested package shouldn't resolve to a single verdict in either direction. Verdicts are issued per dependency-resolution root (matching how Dependabot submits bumps), and a PR proceeds only if every file it touches lands in a root that passed. Improve one root's tests and that root's bumps unblock immediately - a ratchet teams can climb one notch at a time, instead of a bar the whole repository has to clear at once. And when a PR's blast radius can't be fully mapped, the gate fails closed.

The Label Is a State Machine

When the Janitor can't safely merge a PR, it parks it: an explanatory comment plus a label, depjan-needs-review. That humble little label turned out to do a lot of semantic and mechanical work - it's actually three mechanisms in a trenchcoat:

It's the off switch. The nightly filter drops labelled PRs before any agent session is created, so the Janitor never spends two sessions rediscovering the same blocker. One investigation, one comment, then silence - an agent that re-diagnoses the same red CI every night is burning tokens to repeat itself and annoying the humans.

It's the worklist. Search your org for open PRs carrying the label and you have the definitive list of dependency PRs that actually need a human - each one already investigated, already explained, with the Janitor's reasoning attached.

And it's the resume button. The comment says so explicitly: fix the issue, remove the label, and the PR re-enters the queue on the next nightly run. No dashboard, no config change, no ticket.

Receipts

Every action the Janitor takes is designed to be re-adjudicated later, by someone who wasn't there.

Every approval it issues leads with a fixed attribution string, so a human scanning a PR can tell at a glance that a bot approved it, and on what basis. Every merge and every parking gets a comment: readable prose on top, and underneath, collapsed, the verbatim machine-readable verdict the prose was synthesized from - so if the summary ever drifts from the evidence, the drift is visible right there in the comment rather than lost. And every judgement comment ends with a link to the actual agent session that produced it, so "why did it do that?" has a one-click answer for as long as we retain sessions (about a month).

That session link earns its keep in a way I didn't anticipate: while the session exists, a maintainer can open it and ask the Janitor about its decision. Our WordPress repo got parked - on pull requests, the only gate the Janitor could find was a security scanner - and that verdict was missing something. The team had wired Tugboat to deploy every PR to a live preview environment, and their real review process was a human clicking around the preview. The Janitor never saw it: the preview check comes from an external app, and the Janitor's investigation scope was the repository's own GitHub Actions Workflows defined in .github/workflows/.

So I opened the still-idle session and asked it - did you see this, and what do you make of it? Pointed at the integration, it recognized Tugboat, acknowledged the miss, and produced an updated verdict.

Still no-go, and for a sharper reason: a preview deployment proves the change deploys; it emits no report about whether the deployed thing is correct. The outcome didn't change. My confidence in the reasoning did - and so did the team's, because the second verdict engaged with their actual process instead of ignoring it.

Interrogating a judgement after the fact, in the very context that produced it, is now a first-class capability of the Janitor. It does require platform access, so it's a maintainer's tool rather than every developer's - the evidence comments are the general-audience artifact, and the session is the appeal process.

Claude leaves nice summaries for us at the end of each session - richer than what gets posted to the already-busy PR threads - which can sometimes preclude even needing to ask:

The Job Was Never the Merging

The Janitor is still in early access, onboarding team by team since mid-June - thirty-odd repositories so far. The scoreboard after seven weeks: about 250 boring bumps merged themselves, and more than 800 PRs came back to humans carrying an investigation, an explanation, and a label. A merge bot that parks three PRs for every one it merges might sound broken. But look at what the parked pile is: dependency PRs that would have needed a human anyway, now pre-investigated and queued in a searchable worklist - and about 245 of those have already been cleared by teams acting on the Janitor's reasoning.

The first win is the one we advertised: fewer patch bumps waiting for a human to click the same buttons again.

The second win is the one I care about more: the flashlight. The Janitor's bar has teams thinking about quality assurance in a way no metric ever got them to. A coverage threshold is a fixed-metric game - set it at 80% and every repo drifts to 81%, with no guarantee the uncovered 19% isn't the mission-critical part. The Janitor's bar is not a number, so there's no percentage to farm. Its park comments are specific enough to show you where the real gaps are - which modules, which missing gate - but the bar itself is a judgement, and the straightest path through it is to build a check suite you would personally trust to catch a bad bump. Do that work and the Janitor recognizes it and takes you at your word. (Check suite, not test suite: what counts is everything bound to your PR CI, not just what's in the unit-test directory.) Can you still game it? Probably. But a policy memo saying "your checks must actually protect your code" would have changed nothing, and a janitor that politely refuses to merge until it judges the check suite meaningful has teams doing the real work, one repo at a time.

That's the deal we advertise to our development teams: the boring ones can merge themselves. The interesting ones still come back to you - already investigated, already labelled, and carrying receipts.

The rest of what this project taught me aren't wins so much as lessons - patterns I'd carry into the next agentic system, whatever its job:

Determinism is a containment mechanism first and a cost optimization second.

Every decision you make before the agent wakes up is a decision the agent cannot be talked out of. Draw the deterministic/judgement line, then push it as far toward deterministic as it will go.

Filter, declare, validate.

When an invariant really matters, enforce it at least three ways, in this order:

1. deterministically withhold what the model must not act on
2. declare the rule in prose where the decision happens
3. validate it in prose again at the write boundary

The two prose guards repeat each other on purpose - prose is the material you can't trust to be paid full attention every single time.

A bias-to-safe design can substitute for an eval harness - but only when one failure direction is cheap.

We have no eval measuring whether the Janitor's judgement is good.

We can afford that honesty (for now) because the verdict is binary and asymmetric: a false "no-go" means a human reviews a PR manually, like they would have anyway; a false "go" means merging into code that can't catch the break. So every ambiguity, every fetch failure, every unmappable file resolves in the same direction - park it. If your agent's outputs are graded rather than binary, or both failure directions cost real money, you don't get this shortcut. Budget for the eval.

You'll notice none of those lessons mention dependencies. Those lessons are about the shape of the trust, not the task at hand, and they'll follow us to the *next* agent we toss the keys to, too.

Postscript: Design Choices for the Curious

A few mechanical decisions that didn't fit the narrative but might be interesting:

¹The mechanical check runs on the model anyway

"Can this PR merge?" is a handful of field comparisons - merge state, check conclusions, review status - with no judgement anywhere, and we fully intended to keep the model's hands off it.

Plan A was a deterministic script bundled with the agent; the sandbox's design vetoed it, because inside the container only MCP tool calls and git-against-the-clone can authenticate, and a bundled script is an arbitrary process that gets no credential. It literally could not fetch the fields it needed to compare. In the time since we made this design, Claude Managed Agents now do support authentication through Environment Variables so we would be able to do things this way.

We won't, though, because Plan B is the architecturally-correct answer when you can no longer trust the model to compose simple tool calls: a custom Tool the agent calls, backed by a deterministic service we run. That trades a one-if predicate for a persistent, authenticated callback endpoint with uptime and state to babysit.

So for now we're still on the lazy road: the comparison lives in a skill, executed by the model, promotable to a real Tool the day it misbehaves. It hasn't yet.

One agent session per repository, per night

Per-PR sessions would re-derive the same repo knowledge - the same test-suite assessment, the same layout - once per bump, on repos that often have five open at once, and the batch executor couldn't act coherently on a repo it only sees a slice of. One org-wide session fails the other way: thirty repositories in one context window degrades attention on all of them, and one repo's failure threatens the rest. Per-repo is where the work is genuinely independent, so it's the unit of isolation, attention, and billing.

Conflicted PRs heal themselves

A Dependabot PR that's fallen into conflict isn't worth an agent's time, but it's also not worth a human's. The function comments @dependabot recreate on such PRs (with a two-week cooldown per PR), Dependabot rebuilds it, and a future nightly run sees a mergeable PR instead.

Write the idempotency key at the boundary whose failure direction is benign

Every multi-write GitHub sequence in the system orders its writes so a mid-sequence crash strands you in the harmless state:

• merge then label
  • merged-but-unlabelled is fine
  • labelled-but-unmerged lies
• label then comment when parking
  • labelled-but-uncommented just goes quiet and the human has to do the investigation they would have done anyway
  • commented-but-unlabelled re-enters the queue and spams the PR nightly

Behavior ships as versioned skills, not one mega-prompt

Each of the five reasoning phases is a separately versioned skill bundle with an input and output contract, loaded when its step is reached rather than all competing for attention at once. Changing how CI diagnosis works edits one artifact, and the deploy pipeline can tell exactly what changed. Any changes to part of the agent's behavior will show up isolated to a file with a clear purpose - easy to review!