Dev Blog

Turning Human Review into Improvements: How Stuut Agent Learns from Its Coworkers

Emma Teng

Head of AI

Oct 14, 2026

min read

Abstract

Many teams adopting AI agents start with a review step: the agent prepares the work and a person approves, adjusts, or sets it aside before anything goes out. Those reviews are a rich and continuous signal, but a single click says only that something could be better, not what or why. We describe a production learning loop that turns these everyday review decisions into specific, prioritized improvements. The loop separates describing what happened in each case from explaining why a pattern of cases happens. A language model writes a one-sentence description of every reviewed case; descriptions are grouped into named, stable patterns and counted weekly; an investigation agent with read-only access to data and code explains the largest patterns and ranks candidate fixes from several angles; and the best fixes are combined into a small number of draft changes that engineers review. We describe the design, what we have seen in the first weeks of running it on three of Stuut Agent's jobs in accounts receivable, and how we plan to measure it.

1. Introduction

Stuut Agent handles collections work for finance teams. Three of its most frequent jobs are replying to emails from the companies a customer bills, capturing payment promises from those emails, such as a commitment to pay an invoice on a given date, and answering questions about a team's receivables in Ask Stuut.

Each team decides how much to automate. Many let Stuut Agent act on its own; others start with review turned on, where a collector sends, adjusts, or sets aside what Stuut Agent prepared. That is a sensible way to start with any new teammate, and it produces thousands of review decisions every week. Our goal is to make each of those decisions count: to learn from them until the review step adds little, and teams are comfortable turning on full automation.

The difficulty is that a review decision is an outcome, not an explanation. An adjusted reply might reflect tone, a missing detail, information that arrived after the agent ran, or knowledge only the collector had. Reading decisions by hand works and is how we started, but it does not scale, and it showed us something important: the most valuable improvements were rarely a better prompt. More often Stuut Agent did well with what it had, and the real opportunity was a missing capability, the timing of upstream data, or a setting that suited one team but not another. Finding those takes investigation, not just labeling.

This post describes three ideas:

A separation of symptom and cause. A quick, consistent description of every case, and a deeper investigation of only the patterns that matter, with access to data and code.
Stable, measurable patterns. Named groups that stay fixed week to week, so the effect of a change shows up as a group shrinking.
Fix ranking and consolidation. Candidate fixes from several angles, ranked by impact against effort and risk, then combined across patterns into a few reviewable changes.

2. Setting

The loop currently covers three of Stuut Agent's jobs, which together account for most of the tasks that reach a reviewer. Each job has its own instructions, tools, and review step, so the loop measures and improves each one separately.

JobWhat it preparesWhat the reviewer can doReplying to emailsA reply to an inbound emailSend as drafted, edit and send, or dismissCapturing payment promisesA payment promise captured from an emailAcknowledge, complete by hand (for example, supply a date), or clear

We measure progress with the same task KPIs our product team uses: for each task type, the share of resolved tasks that were dismissed, and, for tasks that were completed, the share that needed changes. The loop's purpose is to move these numbers in the right direction, so every pattern it reports is expressed as the share of a KPI it accounts for.

The data comes from three places: the task records (what the agent prepared and what the reviewer did), Stuut Agent's execution traces (which tools it called and what they returned), and the business data in our warehouse (invoice state, payments, documents, and each customer's setup at the time the agent ran).

3. Method

The loop runs weekly in five steps (Figure 1).

Figure 1: the five-step learning loop. Gather the facts, describe each case, group the patterns, find the root cause, and fix with people in charge.

The first three steps are quick and run on every case. The deep investigation runs only where it matters, and people decide what ships.

3.1 Which cases

Every task where the reviewer changed or set aside Stuut Agent's output is analyzed, plus a 10% random sample of tasks accepted as-is, which serves as a baseline. Only decisions made by people are included; tasks closed automatically are excluded.

3.2 Gathering the facts

The most useful facts are often not in Stuut Agent's trace. A reply that described an invoice as open may have been accurate when written and outdated minutes later, because a payment was reported after the agent ran. We therefore reconstruct the state of the business data at the time of the agent's run, and keep it separate from what arrived afterward. This is what lets the loop tell a missing capability from a timing issue from knowledge only the reviewer had.

Traces are matched to tasks without relying on personal identifiers. For email replies, the match uses an identifier embedded in Stuut Agent's input; for payment promises, it uses the customer and partner identifiers together with the time of the run, and a case is left unmatched rather than guessed when two runs are equally likely.

3.3 Describing each case

A single language-model call (Claude Opus 5.5, through a batch interface) reads the facts for one case and answers two questions only:

Was there a real gap? Either Stuut Agent recognized the limitation itself, it did not, or there was no gap (for example, a stylistic edit, a process step, or information only the reviewer had).
What happened, in one sentence specific enough to group: what was asked, what the relevant tool returned, what state the data was in, and what the reviewer did. The sentence excludes names, amounts, and quotes.

This step deliberately does not say why something happened or who should fix it. With one case and no access to the system, any explanation would be a guess. Every result carries a version code derived from the model, the prompt, and the output format; when any of these change, recent weeks are described again, so trends reflect Stuut Agent rather than our own edits.

3.4 Grouping the patterns

A one-time step proposes named groups from a sample of descriptions and merges overlapping proposals. Every later week, each new description is assigned to the closest existing group or to none. Keeping the group list stable is what makes weekly counts comparable; descriptions that fit no group feed the next revision of the list.

Each group is counted across all customers and per customer. A pattern concentrated at one or two customers usually points to their setup or playbook; a pattern spread across many points to the product. A group is flagged for investigation when it is mostly real gaps, large enough to matter, and either among the largest for its job, growing quickly, or new.

3.5 Finding the root cause

For each flagged group, an investigation agent works the way an engineer would. It reads cases in full, forms hypotheses, tests them with counts across the whole group, and reads the relevant code. Its tools are read-only: queries against the warehouse, full case views including traces, and search over the current code.

Two instructions proved important. First, the agent must not take the current design as given: it proposes three to five fixes from different angles before choosing, such as giving Stuut Agent a new capability, adding a deterministic check, letting Stuut Agent decline when it cannot do the job well, changing the product or task design, fixing upstream data, or changing a customer's setup. Second, it follows established practice for production agents: prefer tools and deterministic checks over longer prompt instructions, give Stuut Agent ground truth instead of asking it to infer, and make "no action needed" a first-class answer.

Each fix is estimated for the cases it would remove, its effort, and its risk, and the fixes are ranked. The agent marks at most two as ready for a draft change, only when a focused change can deliver them without a product decision first. A short follow-up step rewrites the report in plain language for readers outside engineering. Figure 2 shows a simplified example.

Figure 2: an example report for one pattern, where the agent cannot attach an invoice copy because the document has not reached the system yet, with five suggested fixes ranked by angle, coverage, effort and owner.

A simplified report for one pattern, with fixes from several angles.

3.6 Fixing, with people in charge

Reports are written independently, so several may propose the same change. Before anything is drafted, the marked fixes from all reports are combined: fixes that change the same code for the same reason become one change, and only the most valuable few are drafted each week. A drafting agent implements each combined change in an isolated copy of the code, runs focused tests, and opens a draft for engineering review. When part of a change needs a product decision, it implements the rest and lists the open decision explicitly.

There is no automatic merging. Every change is reviewed by an engineer, and changes whose effect cannot be checked by replaying past cases are labeled as such.

4. Implementation

The loop runs on our existing data warehouse and transformation layer, a weekly job, and a small internal dashboard. The dashboard shows each week's KPIs with week-over-week change, the flagged patterns ranked by the share of each KPI they account for, plain-language reports with ranked fixes, and a per-customer view.

The split between a quick description of every case and selective investigation is what keeps the loop practical: investigating every case in depth would take far longer, and most cases would not need it. Figure 3 shows how one week narrows from thousands of review decisions to a handful of changes.

Figure 3: one week of the loop, narrowing from thousands of review decisions to about 60 named patterns, 16 investigated patterns, and 6 changes for engineers to review.

One week of the loop on three of Stuut Agent's jobs.

5. Early observations

The observations below come from the first weeks of operation and are qualitative by design; Section 7 describes how we plan to measure them.

Most opportunities we investigated are fixable within Stuut Agent's own design, and few come down to prompt wording. In the first weeks, the investigations pointed mostly to structural changes: giving Stuut Agent a way to say a payment was already made rather than recording a future promise, checking an invoice's status before creating a task, not padding a stated date, and recognizing routine automatic replies before acting on them. These are tools, checks, and output options rather than longer instructions, consistent with established practice for production agents.

Some of the most valuable opportunities sit upstream of Stuut Agent. In one pattern, Stuut Agent correctly deferred a request because a document had not yet reached the system. The ranked fixes ranged from bringing documents in sooner, to letting Stuut Agent find documents already stored elsewhere, to generating a copy on demand where customers allow it. Seeing these side by side, with their estimated coverage, makes the trade-offs clear to the teams that own each option, which a single recommendation would hide.

The patterns are stable and concentrated. About 60 named groups described most cases, and a small number of them accounted for a large share of each job's opportunity. That concentration is what makes a weekly list of a handful of changes meaningful.

Lessons from building the loop

Challenge the setup, not only the symptom. An early version of the investigation prompt took the current design as given and recommended restoring missing inputs. Requiring several fix angles, including new capabilities, surfaced options the first version never considered.
Consolidate before drafting. Independent reports naturally propose overlapping changes. Drafting one change per report produced many overlapping drafts; combining them first produced a few coherent ones that are far easier to review.
Ship the decision-free part. When a combined change included a part that needed a product decision, an all-or-nothing drafter set aside the easy parts too. Implementing what is ready and listing open decisions explicitly kept progress moving and put the decisions in front of the right people.
Write for the reader who acts. Reports are useful only if a product manager can understand them on first read. A plain-language summary, a breakdown of where cases come from, and a short ranked list of fixes, with technical evidence one click away, made the reports far easier to act on.

6. Related work

Learning from reviewers' decisions in enterprise workflows has been studied directly. Learning Selective LLM Autonomy from Copilot Feedback uses operators' accept and reject decisions in customer support as a training signal, sorts rejections by cause, and gradually automates low-risk steps. Agent-in-the-Loop builds feedback into live support work, including agents flagging missing knowledge. Evidence-in-the-Loop uses traces to decide which component a change belongs in, and Self-Improving Coding Agents Through Behavioral Rules turns accepted review comments into rules. Commercial tooling such as LangSmith Engine groups production issues from traces, reads the code, and drafts changes.

Work on attributing outcomes to steps in an agent run (Who&When, AgenTracer) and on multi-agent systems (MAST) finds that many issues stem from specification and coordination rather than the model, consistent with our observations. Who Validates the Validators addresses checking model-based judgments against human labels, and Bias in the Loop shows reviewers can accept suggestions too readily, a reason to anchor evaluation in outcomes. The selective labels problem frames the difficulty of judging models when people make the final decision.

Our work combines existing ideas rather than introducing a new technique. What we believe is distinctive is the combination of: reconstructing business data as of Stuut Agent's run to separate capability, timing, and reviewer knowledge; explicit ranking and consolidation of fixes across patterns; a planned outcome signal that does not depend on the reviewer, such as whether an invoice was paid; and a domain, accounts receivable, that has received little study.

7. How we plan to measure it

We plan to evaluate each part of the loop separately, then together.

QuestionHow we plan to measure it**Are case descriptions accurate?Agreement between the description step and human labels on a stratified sample of casesAre the groups coherent?Share of cases that fit their group strongly, and stability of group membership across weeksDoes the investigation find real causes?A blind test on patterns whose causes we already understand, and how often owning teams accept its conclusionsDo fixes work?Change in a fixed group's share of the KPI after release, compared with similar groups that were not changed in the same periodDoes it move the KPIs?**Week-over-week dismiss and edit rates per task type, outcomes such as whether invoices were paid, and the share of customers who turn on full automation

We are also exploring replay testing: running past cases through a proposed change before review, so that changes whose effect can be checked automatically arrive with that evidence.

8. Limitations

Early stage. The observations above come from the first few weeks of operation and three of Stuut Agent's jobs.
Model-based description. The description step can misread a case. Group statistics are robust to occasional errors, but systematic misreadings would bias them; the human-labeled sample in Section 7 is designed to catch this.
Estimates, not measurements. The case counts attached to fixes are the investigation agent's estimates. Only the post-release comparison measures actual effect.
Reviewer behavior varies. Some decisions reflect team habits rather than the agent. Per-customer counts and outcome signals help separate the two, but not completely.

9. Privacy and safety

Email text is read only while the weekly job runs and is not stored in analysis tables. Case descriptions exclude names, contact details, amounts, invoice numbers, and quotes. The investigation agent's access is read-only. Every proposed change is reviewed by an engineer before it can ship, and nothing merges automatically.

10. Conclusion

A review step does not have to be a cost of adopting an AI agent; it can be how the agent learns from its coworkers, the way feedback helps any new teammate get better. By describing every case quickly and consistently, grouping the descriptions into stable patterns, investigating only the patterns that matter, and turning the best fixes into a few reviewable changes, the loop converts everyday review decisions into a steady, measurable stream of improvements. Next, we will measure it: the accuracy of each stage, and whether the patterns it targets shrink once its changes ship.

Emma Teng

Head of AI

Emma is Head of AI at Stuut, where she leads the Data & AI team building next-generation AR automation and analytics. With 10+ years of AI experience, she's passionate about scalable AI infrastructure and creating product experiences that are as enjoyable as they are powerful.

We’re hiring

If you’re looking for a new exciting challenge, Stuut might be the place for you.

View job openings

October 14, 2026

Emma Teng

Turning Human Review into Improvements: How Stuut Agent Learns from Its Coworkers

How Stuut Agent turns everyday human review decisions into prioritized fixes: one-sentence case descriptions, stable weekly patterns, root-cause investigation, and a few engineer-reviewed changes.

Read More

July 21, 2026

Emma Teng

Building Ask Stuut: Your AI AR Analyst

Learn how Stuut built Ask Stuut, an AI analyst for accounts receivable, using governed data pipelines, semantic views, tenant isolation, and evals to deliver trustworthy AR insights.

Read More

See all articles