Skip to content
LakeBench
ProblemsCommunityPricing
Sign inStart practicing

Orchestration & Reliable Pipelines

Progress0/14
x

What is a Pipeline

  • What is a data pipeline?12m
  • Scheduling and cron12m
  • Monitoring and alerting12m

DAG Mental Model

  • Why a scheduler12m
  • Tasks, edges, topological order14m
  • Operators, sensors, and XCom12m

Retries, Idempotency, Dedup

  • Retries in the DAG runner14m
  • Idempotent loads14m
  • Deduplicate before load12m

Backfills, Catchup, Intervals

  • Backfills and catchup12m
  • Data interval vs execution clock12m

CDC Concepts

  • CDC: inserts, updates, deletes12m
  • Apply CDC idempotently14m

Capstone

  • Capstone: a reliable night job16m
Back to track
  1. Learn
  2. Orchestration & Reliable Pipelines
  3. What is a Pipeline
  4. Scheduling and cron

Lesson 2 of 14 · Theory first, then run it

Scheduling and cron

orchestrationpythonbeginner12 min

Overview

cron says when a job starts. 0 2 * * * means minute 0, hour 2, every day: 2 AM daily. This tab has no cron daemon.

On this page7 sections›
  1. 1What you will do
  2. 2Why this skill
  3. 3How the code works
  4. 4Worked examples
  5. 5Common beginner questions
  6. 6What comes next
  7. 7Practice

What you will do

A schedule is a clock for a pipeline. You decide when the job should start: every night at 2 AM, every hour, every Monday. The machine, not a person, owns that clock.

cron is the classic way to write a schedule. A cron expression has five fields: minute, hour, day of month, month, day of week. Stars mean 'every.' The expression 0 2 * * * means minute 0, hour 2, every day of month, every month, every weekday: 2 AM daily.

No cron daemon and no Airflow timetable run in this tab. You parse a string with split. On a real machine, cron, systemd timers, or a managed scheduler (Cloud Composer, MWAA, Data Factory) fires the job.

Why this skill

Warehouse night jobs usually wait until source systems finish the business day, then run in the small hours so gold is ready by 08:00. If someone has to wake up and click Run, the dashboard is late whenever they sleep in. A schedule makes 2 AM boring, which is the goal.

Hourly jobs exist when a dashboard cannot wait until morning (operations, fraud, live inventory). Daily versus hourly is a product choice: freshness versus cost and complexity. The expression is how you write the choice down.

How the code works

Think of an alarm clock on the dock door. The alarm rings at 2 AM. It does not load the truck. It only starts the shift. Retries, dependencies, and 'what ran last Tuesday' are later lessons. This lesson is the ring.

A daily 2 AM night job
00:00 day ends02:00 cron firesextract →transform → load08:00 dashboardreads gold

cron fires at 02:00. Extract starts. Transform and load follow. Gold is the output, not the clock.

Read a cron expression left to right. Field 1 is minute (0 to 59). Field 2 is hour (0 to 23, 24-hour time). Field 3 is day of month (1 to 31). Field 4 is month (1 to 12). Field 5 is day of week (0 to 6, often 0 is Sunday). 0 2 * * * fills only minute and hour; the stars mean every calendar day.

Hour 2 is 2 AM. Hour 14 is 2 PM. Mixing them up collides with daytime traffic.

ExpressionMinuteHourMeans
0 2 * * *022 AM every day
0 14 * * *0142 PM every day
0 * * * *0everyTop of every hour
30 2 * * 13022:30 AM every Monday

Airflow and other orchestrators often use the same five fields, or a timetable object that still means 'daily at 02:00.' The product UI may hide the string. The idea stays: a clock, a timezone, and a pipeline that starts when the clock says so.

Worked examples

Split the 2 AM daily expression and pick the first two fields. The Python editor is not cron. It is enough to prove you can read the string.

PythonFive fields, split on spaces
expr = "0 2 * * *"
parts = expr.split()
print("minute", parts[0], "hour", parts[1])
print("fields", len(parts))

Wrap the parse in a function that returns a dict. Daily jobs only need minute and hour for this exercise. Day-of-week expressions can wait until you have a Monday-only report.

PythonSame parser, 2 AM versus 2 PM
def parse_daily_clock(expr):
    parts = expr.split()
    return {"minute": int(parts[0]), "hour": int(parts[1])}

print(parse_daily_clock("0 2 * * *"))
print(parse_daily_clock("0 14 * * *"))

Common beginner questions

Does cron run in my timezone?

On a laptop, cron uses the machine timezone. In the cloud, the scheduler has its own timezone setting (often UTC). 2 AM UTC is not 2 AM in India. Name the timezone when you write the schedule, or gold will be late for the dashboard's morning.

Is cron enough by itself?

For one independent script, yes. For extract-then-load with retries, cron will start a second job while the first is still running, and it will not skip load when extract failed. That is why orchestrators exist. cron still shows up inside them as the timetable.

What about 'every 15 minutes'?

*/15 * * * * is a common spelling. This lesson grades a daily 2 AM job. Fifteen-minute jobs need the same parse of minute and hour, plus a conversation about whether the previous run finished.

Hour 2 is 2 AM, not 2 PM

cron hours are 0-23. 0 14 * * * is 2 PM. 0 2 * * * is 2 AM. A 2 PM warehouse job will fight the same cluster the analysts are using.

A clock is not a dependency

Scheduling starts the pipeline. It does not wait for the vendor file. Sensors and upstream edges (later in this track) are how you wait. Do not use sleep(3600) as a substitute.

What comes next

The next lesson is monitoring: how you know a scheduled run succeeded, where logs live, and when to alert because the DAG is red or an SLA was missed.

Practice

Run Sample to parse 0 2 * * *. Then complete Exercise: assign {"minute": 0, "hour": 2} to result and print it.

Practicals · load into the editor

After you read the theory, run these in the pane on the right. They execute in this tab, no cluster.

Rate:
Was this useful?
What is a data pipeline?Monitoring and alerting