Lesson 2 of 3 in this module

2/14 in track

Scheduling and cron

python12 min

Overview

cron decides when a job starts. 0 2 * * * is minute 0, hour 2, every day, so 2 AM daily. There is no cron running in this tab.

On this page8 sections

What you will do

From manual triggers to automated schedules

Your pipeline is ready, but somebody still has to start it. In this lesson, you learn to read schedules written in cron, the format that orchestrators parse.

Schedules replace manual developer triggers with precise, unattended execution times.

A schedule defines the rule for when a job begins. In production data platforms, jobs do not run on personal developer schedules. They execute unattended according to business cadences.

When a pipeline depends on an engineer remembering to run a command every morning, it fails the first day that engineer is sick, commuting, or busy with an incident.

The five cron fields

cron is a Unix program that parses time rules and runs commands. The expression contains 5 fields separated by spaces.

The cron expression 0 2 * * * split into five boxes labelled minute, hour, day of month, month and day of week
The five cron fields. The two blue boxes are fixed values, the grey stars mean every value.

Read the picture left to right. The first two boxes pin minute 0 and hour 2. The three stars mean every day, every month, and every weekday: exactly 02:00 daily.

This tab has no cron daemon running in it, so nothing will actually fire at 2 AM. You practice parsing strings in Python so you can configure real schedulers like Airflow with confidence.

Every modern orchestrator translates cron strings into internal datetime triggers. If you know how the five slots behave, you can configure schedules across any tool without guessing.

Words you will see in this lesson

Six words. The rest of the lesson uses only these.

WordWhat it means
CronA schedule written as five fields, such as 0 2 * * *
ScheduleWhen a job should start
UTCOne world clock that does not change with daylight saving
Data intervalThe slice of time the data belongs to, like yesterday
SLAThe deadline you promised for the data to be ready
Batch windowThe quiet hours when the job is allowed to run

Why this skill

The 2 AM batch window

Why run at 2 AM? Yesterday's transactions are finished, transactional databases are quiet, and analytical tables finish building before morning reports open at 8 AM.

Why a nightly job runs at 02:00
23:59 the daycloses02:00 cron startsthe job02:40 gold tablelands08:00 dashboardsopen

The day closes, the source is quiet, and the numbers are ready long before the dashboard opens.

Running overnight avoids competing with customer traffic. A dashboard needing fresh trends during the day uses an hourly cadence instead.

Online transactional databases experience lower read and write traffic during early morning hours. Pulling bulk extracts during that lull protects production applications from slow queries or locks.

Watch outHour 2 is 2 AM, not 2 PM

Cron hours run from 0 to 23 on a 24-hour clock. 0 2 * * * is 2 AM and 0 14 * * * is 2 PM. Swapping them is how a heavy job lands in the middle of the working day.

Standardizing production clocks on UTC

The most common scheduling mistake is the timezone trap. Cron expressions do not declare timezones; they inherit the server clock.

  1. Spring forward01:59 jumps to 03:00, 02:30 job skipped
  2. Autumn fallback02:59 reverts to 02:00, 02:30 job runs twice
  3. UTC standardNo daylight saving drift, monotonic clock
  4. Business conversionConvert local requirements to UTC offset

Local daylight saving time shifts skip jobs in spring or double-count runs in autumn. Standardize production servers and schedulers on UTC to maintain predictable execution.

When team members work across multiple regions, local time zones cause confusion. UTC provides a universal reference point that never shifts forward or backward across the year.

Execution time versus data interval

When a pipeline runs at 02:00 on October 16, it processes data generated during October 15. In orchestrators like Airflow, the data interval covers the completed period, while execution time is when the worker runs.

ConceptExample timestampWhat it represents
Data Interval Start2026-10-15 00:00:00Beginning of the completed business day
Data Interval End2026-10-15 23:59:59Close of the completed business day
Execution Time2026-10-16 02:00:00When the worker wakes up to run the pipeline

Confusing execution time with the data interval causes off-by-one errors. Always parameterize queries with schedule interval boundaries instead of current date functions.

If a job queries current timestamps directly, re-running a failed job the next day would process the wrong day. Passing explicit interval dates makes pipeline tasks repeatable. Later lessons explore interval mechanics in depth.

Wall-clock cron and fixed intervals

Schedulers support both wall-clock cron schedules and delta-based fixed intervals.

Scheduling modelHow it behavesHandling slow runsIdeal workload
Wall-clock cronFires at fixed clock ticks (00:00, 02:00)Can overlap unless concurrency limits are setReports tied to daily business deadlines
Fixed intervalWaits for previous run to finish before timer startsRuns never overlapContinuous incremental ingest and change capture

Wall-clock cron aligns with human business schedules, while fixed intervals ensure stability for high-frequency ingest.

If a fixed 30-minute job takes 45 minutes to run, cron starts a second run immediately while fixed intervals wait. Knowing this difference prevents resource exhaustion on overloaded workers.

Delivery deadlines and SLAs

Setting a schedule is only half the battle. Guaranteeing that runs finish within a business window is the other half.

  1. 02:00 Job startsNightly warehouse run begins
  2. 05:00 SLA limit3-hour maximum expected runtime
  3. 05:01 SLA miss alertOrchestrator notifies on-call engineer
  4. 08:00 Delivery metIssue resolved before stakeholders open reports

Orchestrators trigger automated SLA alerts when tasks exceed duration thresholds, giving engineers time to resolve pipeline stalls before stakeholders see missing data.

An alert that fires early gives the team hours to investigate bottlenecks before business users log in to view dashboards.

How the code works

Splitting cron strings in Python

A cron expression is plain text. Python's split() method separates the string by spaces into five list items.

PythonSplit the cron string
expr = "0 2 * * *"
parts = expr.split()
print(parts)

Predict what list of strings split produces from the 5 fields.

['0', '2', '*', '*', '*']

Position 0 is the minute and position 1 is the hour. Because items remain strings, parse numeric fields using int() when computing time comparisons.

Calling split with no arguments automatically handles single spaces and multiple spaces between columns. That gives clean tokens regardless of spacing differences.

Field ranges and wildcard syntax

The 5 fields follow a fixed sequence from minutes up to weekdays. Each position accepts numbers, ranges, or symbolic operators.

The 5 fields in the order cron expects them.

PositionFieldAllowed valuesIn 0 2 * * *
0minute0 to 590
1hour0 to 232
2day of month1 to 31*
3month1 to 12*
4day of week0 to 6 (0 is Sunday)*

The asterisk means every value. Commas list discrete values (like 9,17), hyphens define inclusive ranges (like 1-5 for weekdays), and slashes define step intervals (like */15 for every 15 minutes).

Combining these symbols creates expressive rules. For example, 0 9 * * 1-5 runs at 9 AM every business day from Monday through Friday without touching weekend records.

Worked examples

Extracting minute and hour

Start with the simplest extraction: isolate the minute and hour values and store them as integers.

PythonMinute and hour parsed
DAILY_2AM = "0 2 * * *"

parts = DAILY_2AM.split()
result = {"minute": int(parts[0]), "hour": int(parts[1])}
print(result)

Predict the dictionary keys and numeric values before opening.

{'minute': 0, 'hour': 2}

Accessing indices 0 and 1 pulls the runtime clock without scanning the remaining wildcard fields.

Converting to integers lets downstream functions run comparisons, such as checking whether a scheduled time falls inside an approved maintenance window.

Mapping all 5 fields with zip

You can map all five positions to named keys using Python's zip() function.

PythonMap all five fields
FIELDS = ["minute", "hour", "day_of_month", "month", "day_of_week"]

def parse_cron(expr):
    return dict(zip(FIELDS, expr.split()))

print(parse_cron("30 6 * * 1"))

Predict which value lands in day_of_week for Monday.

{'minute': '30', 'hour': '6', 'day_of_month': '*', 'month': '*', 'day_of_week': '1'}

Values stay as strings because fields can hold wildcards, step intervals, or ranges that cannot convert to a single integer.

This dictionary structure makes it easy for test suites and deployment linters to inspect every field and enforce scheduling guidelines.

Standard cron patterns

Practicing standard patterns makes reading scheduling strings second nature.

Same 5 fields, different pins.

ExpressionWhen it runs
0 2 * * *02:00 every day
0 * * * *At minute 0 of every hour
*/15 * * * *Every 15 minutes
30 6 * * 106:30 every Monday
0 9 1 * *09:00 on the first day of each month
0 14 * * *14:00 (2 PM) every day

Each expression uses the same five positions, pinning specific minutes or days while leaving others open as wildcards.

Notice how the asterisks act as wildcards. When day of month and day of week are both stars, the expression fires every single day that matches the hour and minute.

The swapped position trap

The most common beginner error is reading fields backwards, mistaking 0 2 for hour 0, minute 2.

PythonSwapped minute and hour
parts = "0 2 * * *".split()

wrong = {"minute": int(parts[1]), "hour": int(parts[0])}
print(wrong)

Predict the hour and minute when the 2 index positions are reversed.

{'minute': 2, 'hour': 0}

That swapped dict fires at 00:02 (2 minutes past midnight), 2 hours earlier than intended. Minute always precedes hour.

Small units come first: minutes come before hours, just like seconds precede minutes in tools that support 6 fields.

Watch outCron has no timezone

A cron expression does not store a timezone. The server or orchestrator decides, and many default to UTC. Check which timezone your scheduler uses before promising an 8 AM delivery to business teams.

Midnight spikes and catchup floods

Two common scheduling traps affect growing data platforms: midnight resource spikes and accidental catchup floods.

TrapRoot causeSystem consequenceRemediation
Midnight spikeMultiple teams schedule jobs at 00:00 or 02:00 exactlyDatabase connection exhaustion, API rate limitsStagger minutes across jobs (such as 14 2 * * *, 38 2 * * *)
Catchup floodEnabling catchup on an old start dateLaunches dozens of past runs simultaneouslyDisable automatic catchup by default and backfill deliberately

Staggering schedule minutes flattens cluster load, while disabling automated catchup prevents runaway historical executions.

By picking non-zero minutes like 17 or 43 instead of 00, worker CPU utilization distributes evenly instead of creating queuing bottlenecks.

Production scheduling rules

Disciplined scheduling balances automated execution, clock synchronization, and resource distribution.

Production scheduling rules and lifecycle
  1. Pin production schedulers and databases to Coordinated Universal Time.

    1. timezone
    2. UTC only
  2. Stagger job start minutes to prevent midnight resource exhaustion spikes.

    1. concurrency
    2. stagger minutes
  3. Separate execution time from data interval parameters in downstream queries.

    1. logic
    2. data interval
  4. Disable automatic catchup by default to avoid accidental historical backfill floods.

    1. backfills
    2. catchup false
  1. Read cron fields strictly in order: minute, hour, day of month, month, day of week.
  2. Standardize server and orchestrator clocks on UTC to prevent daylight saving errors.
  3. Stagger job schedules across non-zero minutes to prevent resource contention.
  4. Pass explicit data interval parameters into SQL queries instead of system clocks.
  5. Attach SLA thresholds to core reporting pipelines to catch processing delays.

Common beginner questions

What does */15 mean?

Every 15 units of that field. In the minute field it fires at minutes 0, 15, 30, and 45 of every hour.

I have seen cron strings with 6 fields. Are they wrong?

No. Some systems, such as Quartz, add a seconds field at the front. Classic Unix cron and Airflow cron strings use 5 fields, so count the fields before reading them.

If my job fails at 2 AM, will cron run it again?

No. cron only starts the command. It does not know whether the job succeeded or failed. Retries, error alerts, and dependencies are why dedicated orchestrators exist.

What comes next

Now your pipeline has an automated clock. The next lesson covers monitoring: how to determine whether last night's 2 AM run succeeded, how to triage errors, and when alerts should notify engineers.

Practice

The exercise gives you the 2 AM expression and asks for a dictionary with minute and hour as integers. Use the split technique from the examples above, and preserve the field order.

Going further: a daily schedule and daylight saving time

A schedule written in local time breaks twice a year in places that change their clocks. In the spring 1 hour does not exist. In the autumn 1 hour happens twice. A job scheduled inside those hours can be skipped or run twice. This is why many teams schedule in UTC, which never changes.

In New York in 2026 the clocks jump from 02:00 to 03:00 on 8 March, and fall back from 02:00 to 01:00 on 1 November. Predict what happens to a job at 02:30 on the first date and at 01:30 on the second.

PythonLocal times inside a clock change
from datetime import datetime
from zoneinfo import ZoneInfo

ny = ZoneInfo("America/New_York")
utc = ZoneInfo("UTC")

for label, day, hour in [("spring forward, 02:30", (2026, 3, 8), 2), ("fall back, 01:30", (2026, 11, 1), 1)]:
    for fold in (0, 1):
        local = datetime(*day, hour, 30, tzinfo=ny, fold=fold)
        print(f"{label:22} fold={fold}  {local:%H:%M %Z}  = UTC {local.astimezone(utc):%H:%M}")
spring forward, 02:30  fold=0  02:30 EST  = UTC 07:30
spring forward, 02:30  fold=1  02:30 EDT  = UTC 06:30
fall back, 01:30       fold=0  01:30 EDT  = UTC 05:30
fall back, 01:30       fold=1  01:30 EST  = UTC 06:30

On the spring day, 02:30 never happens, so Python has to guess: the 2 lines give 07:30 UTC and 06:30 UTC, which are 03:30 and 01:30 on a real clock. A scheduler may run the job at a surprising time, or skip it.

On the fall day, 01:30 happens twice, once in daylight time (05:30 UTC) and once in standard time (06:30 UTC), so a naive scheduler may run the job twice.

If people need the data at a local time, schedule in UTC and pick the UTC hour that works for them.

Schedule inSpring forwardFall back
Local time inside the changed hourThe run may be skipped or shiftedThe run may happen twice
UTCAlways exactly onceAlways exactly once

Try a new scenario

Your daily job is scheduled for 01:30 local time in a country that does not change its clocks. Is there a daylight saving problem? What other schedule problem remains?

Work it out, then open the answer

No clock change means no daylight saving problem for that country. The other problems remain: a run that takes longer than the interval can overlap with the next one, and a missed run (the scheduler was down) needs a rule for whether to catch up. Decide both on purpose.

What does catchup do in a scheduler?

When catchup is on, the scheduler creates a run for every interval since the start date that has not run yet, which can launch hundreds of runs at once on a new pipeline. Turn it off if you do not want history rebuilt automatically, and use an explicit backfill when you do.

Close the lesson and explain it

Without scrolling up, explain why a local-time schedule can skip or double a run, and what UTC changes.

Practicals · load into the editor

After you read the theory, run these in the pane on the right. They execute in this tab, no cluster.

Finished this lesson?

Mark it as read when you could explain it to someone else.

Rate:
Was this useful?