Skip to main content
    All posts

    Watchrr: what a green status page doesn't tell you

    We poll 228 services every few minutes to answer one question: is it down? Here's what we learned about why official status pages are the least reliable part of that.

    Heyitskenton
    6 min read
    The Watchrr dashboard showing live status across service categories

    There is a specific kind of frustration in staring at a service that is clearly broken while its official status page sits there, green, insisting everything is operational.

    Watchrr exists because of that gap. It tracks 228 services across 15 categories — gaming, streaming, cloud, AI, payments, app stores, developer tools — and refreshes every 10 to 15 minutes. The pitch is simple: one page that tells you whether the thing you are trying to use is having a bad day.

    Building it taught me that the hard part is not polling. It is deciding what to believe.

    A 200 is not a status page

    Most large services publish a JSON feed through Atlassian Statuspage. You fetch it, you read status.indicator, you are done. That works until it doesn't.

    Lyft deleted its Statuspage account. The URL we had pointed at — lyft.statuspage.io — still resolves and still returns HTTP 200, because Atlassian serves its own marketing page there. Nothing about the request looks like a failure. It has a status code in the 200s, it has a body, it has headers. It only falls over at JSON.parse.

    The obvious fix is to point at lyft.com instead and check whether it loads. That fix is worse than the bug. Lyft's marketing site is served by different infrastructure than Lyft's ride dispatch. A 200 from the homepage would let us paint Lyft green while telling you exactly nothing about whether anyone can get a car. We would have replaced "no data" with "confidently wrong", which is the worse of the two.

    So Watchrr leaves it broken on purpose. The staleness guard notices that nothing has parsed in a while and renders Lyft as not reporting. That is an unsatisfying thing to show a user. It is also the only honest thing we can say.

    Some status pages can't be read at all

    A surprising number of "official status pages" are not machine-readable. Paramount+, EA's help site, Activision support, Rockstar support — these are JavaScript-rendered single-page apps, or they sit behind a Cloudflare challenge. From an edge function with no browser, there is nothing to parse.

    Those services get a deliberately weaker check: a HEAD request to the primary domain. If it answers 2xx or 3xx we call it operational, otherwise partial outage.

    That is genuinely worse monitoring, and Watchrr labels it as limited rather than pretending it is equivalent. The alternative — running a headless browser against a few dozen status pages every ten minutes — is a real option, and one day it may be worth the cost. It is not worth it yet.

    Polling politely, and the bug that created

    Two hundred and twenty-eight services on a ten-minute cycle is a lot of requests pointed at other people's infrastructure. The fix is old and boring: conditional GETs. Store each source's ETag and Last-Modified, send them back as If-None-Match and If-Modified-Since, and let the server answer 304 Not Modified when nothing has changed.

    A 304 has no body. Nothing to download, nothing to parse, nothing to write. On a normal day, when almost nothing is broken, the overwhelming majority of polls cost almost nothing — for us and for them.

    Then this happened.

    A 304 means nothing changed, so the early code returned immediately and skipped the write path entirely. Which is correct for status — and completely wrong for last_checked_at. A service whose upstream was perfectly stable never had its timestamp advanced, so its "last checked" froze at whenever its status last changed.

    DigitalOcean was the one that gave it away: zero consecutive failures, a successful 304 a few minutes earlier, and a database row claiming it had not been checked in two days. The monitoring was working. The monitoring's own report of itself was broken.

    The fix is one line in concept — advance the timestamp on a 304, because a 304 is a successful check — but it comes with a trap.

    The trap in the fix

    Watchrr keeps a set of service IDs it polled this run. That set gates a cleanup step: any incident that no longer appears in a service's upstream feed gets deleted, because it has been resolved.

    The instinct when fixing the timestamp bug is to add the 304'd service to that set. It was polled, after all.

    Do that and you delete live incidents. A 304 carries no body, which means it carries no incident list. An empty incident list is not the same as "no incidents" — it is "no information". The cleanup would read that absence as resolution and wipe out every ongoing incident for every service whose feed happened to be unchanged that minute.

    So the 304 branch refreshes exactly one field and touches nothing else, on a throttled interval so it stays cheap — at worst one row per service per 30 minutes rather than one per service per poll.

    Not just status pages

    Official feeds only tell you what a company is willing to admit, usually after someone has confirmed it internally. So Watchrr also pulls from sources that do not need anyone's permission:

    • Cloudflare Radar — traffic patterns, protocol mix, attack vectors
    • IODA — internet outage detection at the network level
    • NetBlocks — connectivity disruption reporting

    These feed the Internet Pulse score on the dashboard, and they are how you notice that something is wrong before a status page catches up.

    Where it goes next

    Watchlists, saved favourites, team monitoring, and a public API are the next things on the list.


    If you want to see it working, Watchrr is live — no account needed. And if you catch it reporting something wrong, there is a bug reporter on every page of this site. Reports that turn into fixes get credited by name.

    Heyitskenton

    Builds and maintains everything Socialocca ships — the apps, the sites, and the systems behind them.

    Share