Your nightly import quietly stopped working three days ago? How to monitor scheduled jobs

A scenario that happens more often than anyone admits: every night at 2 am, a job picks up new orders from the webshop and enters them into the ERP. It has run for years and nobody thinks about it.

On Tuesday, the partner who runs the webshop changes the access credentials. At 2 am the job starts, the webshop rejects it, and the job ends. Quietly.

On Friday a customer calls: where are my goods? Only then does someone look and realise that three days of orders never reached the system.

Why scheduled jobs fail silently

• They run in the background. No screen, nobody watching, in the middle of the night.
• The error goes to a log nobody reads. Somewhere on a server there is a file that says exactly what happened. Nobody opens it.
• They run from an old tool on an old server. Windows Task Scheduler on a server nobody remembers setting up.
• The sneakiest case: the job “succeeds” but does nothing. It starts, reports no error and moves zero orders. Everything looks fine, and nothing is.

Which jobs we are talking about

Almost every company has several of these jobs, without anyone having listed them:

• importing orders from the webshop or partner portals
• syncing stock and prices between systems
• exporting data to accounting
• nightly reports emailed to management
• database and document backups
• renewing certificates and cleaning up old data

Each of them is essentially what we covered in the article on connecting systems: automation replacing manual work. But automation that stops quietly is worse than manual work, because at least someone notices manual work.

A log nobody reads, or an alert that arrives by itself

The same error, two completely different outcomes. On the left, three days of lost orders and angry customers. On the right, five minutes after the error someone knows what happened, and it is fixed by morning.

Four questions good monitoring asks

Knowing whether a job reported an error is not enough. Good monitoring checks four things for every job:

1. Did it start at all? If the server went down or someone changed the schedule, there is no error, because nothing happened.
2. Did it finish without an error?
3. Did it do what it should? If the import usually moves 120 to 180 orders and today it moved zero, something is wrong, even without an error.
4. How long did it take? A job that used to take two minutes and now takes two hours will probably fail tomorrow.

All of it ends up on one screen:

Job	Last run	Result	Status
Webshop order import	today 02:00	0 orders (usually 120–180)	Warning
Export to accounting	yesterday 23:00	did not start	Error
Stock sync	today 03:00	2,412 items	OK
Nightly sales report	today 06:00	sent to 4 addresses	OK

How to know a job has stopped: four ways

You don’t need everything at once. Start with the simplest, and each next step catches what the previous one misses.

1. 1The job reports when it fails
2. 2The job checks in
3. 3Checking the result
4. 4Central monitoring

1. The job reports when it fails. When an error occurs, the job sends an email or a Teams message describing what happened. It is a few lines of code in the existing job. But if the job never starts, there is nobody to send the message.
2. The job checks in. At the end of every successful run, the job checks in with an external service for monitoring scheduled jobs. The service knows it expects a check-in every night by 2:30 and, if none arrives, sends an alert itself. That catches a server that went down and a changed schedule too, because the service is waiting for a message that never came. There are several such services, with a small monthly fee or free for a handful of jobs.
3. Checking the result. In the morning, one check in the database looks at how many orders arrived overnight and compares it with the usual. Zero instead of 150 means an alert, even though no job reported an error. It is the only way to catch the sneakiest case.
4. Central monitoring. When there are many jobs, they all send data to one place, for example Application Insights, which we covered in the article on application monitoring. That is where the screen from the table above lives, with rules such as: the job has not finished successfully for 26 hours, it moved zero records, it took three times longer than usual.

Way	Error	Did not start	0 records	Slow job	Effort
The job reports when it fails	✓	–	–	–	small
The job checks in	✓	✓	–	–	small
Checking the result	✓	✓	✓	–	small to medium
Central monitoring	✓	✓	✓	✓	medium

For most companies, the second and third ways together cover almost everything, and they take a few days to set up, without touching the servers.

Do the jobs need to move to Azure?

Not necessarily. Monitoring is added to jobs where they already run: on the old server, in Windows Task Scheduler or inside the application. Moving them to Azure Functions means rewriting and retesting every job, which is no small task.

A move makes sense when something has to change anyway: when the old server is going, for example because of Windows Server 2016 end of support, or when the job is being reworked anyway. Then jobs in Azure Functions get monitoring almost by themselves, and there is no server that goes down and takes the job with it. If you do move a job to Azure Functions, our free Azure Functions cron expression generator shows exactly when it will run.

What you can do this week

List every scheduled job in the company: what it does, when it runs and where. Then answer one question for each: who gets notified when this job fails?

In most companies the most common answer is “nobody”. Which means that for every one of those jobs, you find out when a customer calls.

How we work

This is exactly what we do: we add monitoring to existing scheduled jobs where they run, so you know as soon as one stops or doesn’t do what it should, and we set up alerts that reach the right person. We suggest a move to Azure only when it pays off. We start with a list of the jobs and the ones that cost the most when they stop. The first step is a free 30-minute call.