Skip to content
BlogEngineering

Designing a code reviewer engineers don’t turn off

Octavo Team · 5 min read

An automated reviewer earns its place one comment at a time. When a bot comments on every pull request, engineers learn to scroll past it, and sooner or later they scroll past the comment that mattered. Then someone turns it off.

We built Octavo Review around that failure. It reviews every pull request on GitHub and GitLab, and it is tuned to post few comments, each one worth stopping for. This post covers how we decide what earns a comment, how a comment is written and placed, and how teams make the reviewer stricter where they care.

A comment has a cost

Every comment asks the author to stop, read and decide. A useful comment repays that time with a bug that never reaches production. A wrong one costs the same time and some trust as well, and the next comment gets read with less attention.

So we treat each comment as a cost the reviewer has to justify. A reviewer can find many things to say about any diff. Most of them are true, and most of them are still not worth an interruption.

What earns a comment

Octavo Review comments on three kinds of problem:

  • Logic bugs: code that gives the wrong result for an input it will receive.
  • Security issues: untrusted input that reaches a query or a shell command, or a route that skips the permission check its neighbors use.
  • Regressions: a change that breaks behavior that callers, tests or other services depend on.

The bar is the same for all three. The reviewer has to name a concrete input or sequence of events that goes wrong and say what should happen instead. If it cannot name one, it does not post.

Here is a comment that clears the bar. The change adds usage-based tiers to a billing service:

type Tier = {  name: "Starter" | "Growth" | "Scale";  limit: number;}; const tiers: readonly Tier[] = [  { name: "Starter", limit: 1_000 },  { name: "Growth", limit: 10_000 },  { name: "Scale", limit: Infinity },]; export function tierFor(requests: number): Tier {  return tiers.find(    (tier) => requests < tier.limit,  ) ?? tiers[tiers.length - 1];}

Octavo Review leaves one inline comment on the comparison line:

tierFor() puts usage equal to a tier’s limit in the next tier up, so an account with exactly 10,000 requests bills as Scale. Compare with <= to keep each limit inside its own tier.

The suggested change comes with it:

   return tiers.find(-    (tier) => requests < tier.limit,+    (tier) => requests <= tier.limit,   ) ?? tiers[tiers.length - 1];

The comment names the function, the input that fails, the effect on a customer and the fix. It leaves the fallback, the tier names and the numeric separators alone. Those are choices, and the author made them.

What stays out

Some comments are easy to generate and rarely change any code. We leave them out by design:

  • Style and formatting. Formatters and linters already check them on every save.
  • Comments that restate the diff. The author knows what they changed.
  • Hedged suggestions such as “consider” or “you might want to” with no failing case behind them.
  • Opinions on naming or structure that the team has not written down as a rule.

The last one matters most. A reviewer that pushes its own taste turns every pull request into a negotiation. Taste belongs to the team, so Octavo Review applies it only where the team has stated it.

Reading past the diff

Many regressions are invisible inside the changed lines. A function changes what it returns for an empty list, and the problem shows up in a caller three directories away. So Octavo Review reads the change against the rest of the repository: the callers of a changed function, the tests that cover it, and the types and schemas it depends on. A finding about a caller goes on the line in the diff that caused it, and it names the file and function that break.

Where a comment goes

Every finding is an inline comment on the line that causes it, with a suggested fix the author can apply from the code host. One finding gets one comment.

Some fixes reach past the commented line. For those, the author can press Fix in Octavo, which hands the fix to an agent. The agent makes the change and runs the tests, and the author reviews the result like any other commit. Every comment ends with the author’s decision: apply the suggestion, hand it to an agent, or reply and resolve it.

Rules make it stricter where you care

Teams know things a reviewer cannot infer from code alone. Amounts are stored in cents. Invoice writes go through one class. Billing routes check the account’s role first. Octavo Review learns the team’s rules, so conventions like these become findings. Project rules live as Markdown files in .octavo/rules:

# Billing- Amounts are integers in minor units.  Flag float arithmetic on any amount.- Writes to the invoices table go through InvoiceRepository.- Billing routes check the account's role before they read invoices.

Rules work in both directions. A written rule tells the reviewer what to flag, and a missing rule tells it to stay out. Teams that want more comments write more rules, and the default stays quiet.

Quiet is a valid result

Most pull requests are fine. When Octavo Review finds nothing that meets the bar, it adds no comments and its check passes.

That restraint is also what makes it reasonable to require. Teams can make Octavo Review a required check on every pull request. A required check that blocks for the wrong reason is soon bypassed, so the same constraint applies there: speak only when there is something to fix.

The author decides

Octavo Review finds the problem, explains it and proposes a fix. The author chooses what to do with it, and the merge stays a human decision.

Read more about Octavo Review, or set it up from the docs⁠.