Skip to content
Little Lab Rats home
Cluster monitoring

Twenty-eight generated pull requests, one running product

High-availability clusters are watched one at a time, through a console built for one at a time. This is the view somebody running a hundred of them actually wants: every cluster, one screen, one verdict each. It ships as a single executable with its own frontend inside it, and the pipeline wrote it in twenty-eight separate pieces.

Deliverable
One Go binary
Fleet on screen
100 clusters
Generated PRs
28, consolidated
Status
Production-ready demo
Context

A console built for one cluster at a time

High-availability clustering keeps a service alive by failing it over between nodes. The tooling that ships with it is built around a single cluster: you open its console, you look at its groups and resources, you close it. That is the right shape when you run two clusters.

At a hundred it is the wrong shape entirely. The question stops being “what is the state of this cluster” and becomes “which of these hundred needs me right now”, and answering it by opening a hundred consoles is not a workflow. What was missing was an aggregate: one verdict per cluster, rolled up from whatever is broken underneath it, on one screen, refreshing on its own.

The interesting constraint was deployment. This lands in operations environments where installing a runtime, a package manager and a build toolchain is a procurement conversation rather than an afternoon. So the whole product had to be one file you copy and run.

The surface

What an operator actually gets

One screen for the whole fleet

A hundred clusters with a rolled-up verdict each, in a grid or a sortable table. Status resolves upward with a fixed precedence, resource then node then group then cluster, so a single failed resource is visible from the top without hunting for it.

Drill down to the resource

Opening a cluster gives the groups, nodes and resources beneath it with the status at every level, and a deep link out to that cluster's own management console for anyone who needs to act rather than look.

Polling you can turn down

The refresh interval defaults to two minutes and can be set anywhere from thirty seconds to five, or switched off. The server seeds the client's starting interval over a response header rather than the two disagreeing about a default.

It degrades instead of blanking

When the backend stops answering, the last known state stays on screen behind a banner saying so. An operations screen that goes empty during an outage is worse than useless, because an outage is exactly when somebody is looking at it.

28

Generated pull requests consolidated into one application

1

Executable: JSON API and user interface in the same file

0

Build steps in the frontend, and no bundler to keep current

100

Clusters rolled up on a single screen

How it was built

The pipeline wrote the parts. Assembly was the job.

This repository exists because the pipeline made it. It was created as the output of one run, and the twenty-eight pull requests numbered twenty-four through fifty-one are the fragments that run produced: each one a coherent piece of work, none of them aware of the others.

That is worth being precise about, because it is the honest version of what agent-assisted delivery currently gives you. Twenty-eight correct fragments are not an application. They overlap, they disagree about names, and they each assume a slightly different shape for the thing they are part of. Turning them into one binary that builds, serves, authenticates and passes its tests was engineering, and it is the part the pipeline is designed to set up rather than replace.

What the generated work did do is remove the typing. The API contract, the sequence diagrams, the security checklist and the design system all came out of the same run, which is why a single-purpose internal tool has a documented interface and a colour system with recorded contrast ratios instead of whatever the first developer liked.

The shape of it

One binary, and the one trap in getting there

The server is Go: routes, the status rollup, bearer authentication and the JSON handlers, with the frontend compiled in through go:embed and served out of memory. The frontend is plain HTML, CSS and JavaScript, with no framework and no build step, which is a deliberate match to a product whose whole promise is that it is one file.

Authentication is an HMAC-signed bearer token with a twenty-four-hour life, verified in constant time, with the signing secret supplied by environment variable and never committed. The cluster endpoints require it; the health and version probes do not, because a liveness check that needs a credential is a liveness check that will eventually be disabled.

The trap: go:embed can only read files underneath the package it is declared in, so the canonical frontend cannot live where you would naturally put it. A sync step copies it into the server package before every build. This is the kind of detail that fails quietly rather than loudly: the build succeeds and the binary serves yesterday’s interface. That is why it is a scripted step in the build rather than a line in a README nobody reads.

On the numbers

What is proven, and what comes next

The fleet is generated. A hundred clusters with a background loop moving their statuses around is a store standing in for the real inventory, and the README calls the whole thing a production-ready demo rather than a production deployment. What has been proven is the product, not the integration.

That distinction matters, so: the rollup precedence, the API contract, the authentication, the polling behaviour, the degradation path and the interface are all real and tested. Swapping the generated store for live cluster data is a known, bounded piece of work behind the same interface. Presenting it as already done would be the exact failure this whole approach exists to prevent.

The test suite covers the API surface rather than the browser: liveness, version, the login rejection path, listing with and without a credential, fetching one cluster, fetching one that does not exist, and the status rollup itself. That last one is the only piece of real business logic in the server, which is why it has a test of its own.

Want one of these for something you run?

If you have a fleet of anything being watched one at a time, the useful version of this conversation starts with what your operators currently keep in a browser tab.

hello@littlelabrats.com