How to Auto-Triage 300+ PRs a Day With AI Agents
For Open-source maintainers · Based on Solmaz On-Demand Disposable Agent Orchestration Framework
// TL;DR
If your open-source project is drowning in AI-generated pull requests with poor descriptions and failing CI, the Solmaz disposable agent orchestration framework lets you encode PR intake as a Standard Operating Procedure and hand it to agents. Each PR gets its own disposable Kubernetes pod that independently determines intent, judges implementation quality, checks conflicts, verifies CI, and runs a shallow refactor loop — escalating to you only when a fundamental design decision is required. You review pre-processed, CI-passing, conflict-free PRs instead of a raw fire hose, dramatically reducing mechanical review load while keeping human judgment where it matters.
Why are AI-generated PRs overwhelming your project?
Popular open-source projects can receive 300–500 pull requests per day. Many are AI-generated with poor descriptions and failing CI. This is the fire hose — a stream of inbound work that exceeds any maintainer's capacity to process manually. The instinct is to close low-quality PRs on sight, but that discards signal: even slop PRs are crucial user feedback data points indicating where something in your codebase is broken or confusing. The real problem isn't the PRs — it's that mechanical review scales linearly with your time, and your time doesn't scale.
How do you encode PR review as an SOP?
Instead of ad-hoc prompting, build a Standard Operating Procedure for agents — a structured, JSON-output-driven loop your agent replays for every PR:
1. Find intent. Never trust the PR description — most are AI-generated and low-signal. Have the agent independently determine what the PR actually does.
2. Judge implementation quality. Is this the best possible fix?
3. Check for conflicts against the current branch.
4. Verify CI status.
5. Run a shallow refactor loop to fix superficial CI failures.
6. Relate to a human only when a fundamental design decision is required.
Each step emits structured JSON so the whole decision trail is auditable and pluggable into your workflow engine. This is the difference between a black-box bot and an inspectable, improvable pipeline.
How do you run this at scale on Kubernetes?
Stand up ACPX as your control-plane CLI and bind it to your harness — Codex, Claude Code, or OpenClaw. ACPX functions as an Argo-like workflow engine, but instead of orchestrating raw containers it drives a harness session through your SOP nodes. For each incoming PR, deploy an on-demand disposable agent pod: a full Kubernetes compute environment spun up on demand and torn down when done. Use a goal operator like the Spritz pattern (textcortex/spritz) to handle provisioning, lifecycle, and teardown, and helm charts for repeatable deployment.
Because multiple PRs process in parallel, add a state synchronization layer — grant read/write GitHub access and layer an rsync-style or Dropbox-algorithm mechanism so concurrent agents don't silently produce conflicting artefacts.
What should you never automate here?
Use the agent's refactor loop only for shallow bugs — superficial fixes it can uncover and resolve without human input. Never let it make fundamental refactors or architectural design decisions inside the loop; that produces slop. The distinction is your quality gate. When the SOP hits a fundamental design question, it must break out and escalate to you.
The payoff: you stop seeing the raw fire hose. Maintainers review only pre-processed, CI-passing, conflict-free PRs with independently-determined intent attached. The mechanical scaffolding is gone; your judgment goes only where it's irreplaceable.
Next step: Audit one week of your inbound PRs, classify them into fully automatable, agent-assisted-with-sign-off, and human-design-required, then encode your first PR-intake SOP in ACPX and deploy a single disposable pod to process one class end to end.
// FREQUENTLY ASKED QUESTIONS
Won't auto-processing PRs let bad code slip through?
No, because the SOP escalates any fundamental design decision back to a human and only auto-handles shallow, CI-verifiable fixes. The agent independently determines intent, judges implementation quality, and runs shallow refactor loops, but architectural judgment stays with you. You review only pre-filtered, CI-passing, conflict-free PRs — a strictly higher-signal set than the raw fire hose.
Should I just auto-close low-quality AI PRs?
No — even slop PRs are crucial user feedback data points showing where something in your codebase is broken or confusing. Instead of discarding them, categorize and bin them. The SOP can bucket low-quality PRs while extracting the signal about what problem the contributor was trying to solve, which is often more valuable than the PR itself.
Do I need Kubernetes to start?
You need a Kubernetes cluster or equivalent to deploy on-demand disposable agent pods with a goal operator and helm charts. However, you can prototype the SOP logic in ACPX against a single harness session first, validate the review-judge-refactor loop on real PRs, and then move to per-task disposable pods once the workflow proves out.