Color theme

NotesJason Chance

Nothing crashed. That was the problem.

When people picture an outage, they picture something loud. The site’s down, the error page is up, and everybody’s phone is buzzing.

Those are bad days, but they’re not the ones that bite hardest. The worst problems I’ve dealt with in 20 years of running software were silent. Nothing crashed. Nothing threw an error. Something just didn’t happen, and nobody noticed until the damage was done.

I could tell a hundred of these. Here are three.

The cleanup job

A server had a cron job that cleaned up old log files every night. One day it stopped running. There was no error, because there was nothing running to produce one. The logs kept growing, night after night, until the disk filled up and the server fell over.

The server crashing was the loud part. The actual failure had happened much earlier, and it didn’t make a sound.

The emails nobody got

A database job was supposed to send out transactional emails, the kind customers are sitting there waiting for. The job didn’t run. Customers didn’t get their emails. We found out the way you never want to find out: customers wrote in to ask where their emails were.

The domain that expired

The registrar sent every renewal notice to one person’s email address. That’s the default with a lot of registrars. Nobody else on the team saw those notices, and when that one inbox missed them, the domain expired. A website, email, everything on that domain went with it.

What these have in common

Every one of these failures was the absence of something. A job that didn’t run. An email that didn’t send. A renewal nobody saw.

A typical uptime monitor can’t catch that. It checks whether a site answers, and a site can answer perfectly while the job behind it has been dead for a week.

For years we patched this together the way most teams do: one service for uptime, another for cron jobs, a dead man’s switch for the scripts nobody remembered, and calendar reminders for certificates and domains. Each had its own alerts, its own bill, and its own blind spots.

So we built Pingstack to watch for the things that don’t happen.

How Pingstack catches them

Cron jobs check in. Every job gets its own ping URL, and the job calls it when it finishes. If that call doesn’t arrive within the schedule plus a grace period, Pingstack opens an incident. The cleanup job would have been caught the first night it didn’t run, not the night the disk filled up.

Jobs can report a number. A job can send a value with its ping, like how many rows it processed or how big the backup was, and Pingstack warns you when it crosses a threshold. A job that runs but does nothing is its own kind of failure, and this catches it.

Domains and certificates get a countdown. Every domain and SSL certificate sits in one list, and you hear about it well before anything lapses. One person’s inbox stops being the only safety net.

Alerts go to a team. Alerts go to notification groups and on-call rosters, by email, Slack, Teams, Discord, SMS, or a phone call. When someone’s on vacation, the alert still finds a person.

None of this is fancy. It’s the boring stuff that keeps a silent failure from turning into a loud one.

Pingstack is free for 15 checks with no time limit. If you’ve got a cron job you haven’t thought about in a while, that’s a good first one.