Lesson 2 of 6 in this module

2/13 in track

Git: working tree, index, commit

python12 min

Overview

git status names three areas. git add copies a file into the index. git commit snapshots the index. This tab has no Git remote.

On this page8 sections

Goal

A pipeline that worked yesterday is broken today. The first question is always: what changed?

Without a history you can only guess. Someone edited a script, or a connection setting, or a query. Nobody knows which.

Many people save copies by hand, like ingest_v2_final_FINAL.py. Git does this properly. It records a snapshot of your project each time you ask, and remembers who made it and why.

Six words for this lesson.

WordWhat it means
GitA tool on your computer that records the history of your files.
RepositoryA project folder that Git is watching. Short name: repo.
CommitOne saved snapshot of the project, with a message that says what changed.
Working treeYour files as they are on disk right now, the ones you edit.
IndexA waiting area where you pick which changes go into the next commit. Also called the staging area.
UntrackedA file Git has never been told about.
RememberWhat this tab runs

This tab runs Python, not Git, and it cannot reach GitHub. The practice reads Git's status output as text. To run real Git, install it on your computer and try it in a folder of your own.

Saving a file in your editor does not create a Git snapshot. It only overwrites the file on disk. To record a version, you tell Git.

Three boxes in a row labelled working tree, index and commit, with arrows labelled git add and git commit pointing right
The three areas. Files move left to right only when you run a command.

Read the picture from left to right. You edit in the working tree. git add copies your chosen changes into the index. git commit seals the index into a permanent snapshot.

Why this order

Why two steps? Because your folder often holds changes that should not go together.

Say you fixed a bug in orders_load.py, and also made a scratch file, notes.py, full of test prints. The fix belongs in history. The scratch file does not.

One folder, two different states. A commit right now records only orders_load.py.

FileWhat you didWhere it is now
orders_load.pyFixed the bug, then ran git addIn the index, ready to commit
notes.pyMade a scratch file, never added itOnly in the working tree

The two commands cannot swap. git commit packs only what is already in the index. If you commit first, Git says there is nothing to commit.

Watch outgit commit -a skips the choosing

The -a flag stages every changed file for you and commits straight away. It is quick, but it also commits scratch edits you forgot about. Check git status and add files on purpose.

The checklist

Recording a change is a five-step cycle.

  1. git status. See which files are changed, new or staged.
  2. git add ingest.py. Put that file in the index.
  3. git status. Check it now appears under Changes to be committed.
  4. git commit -m "ingest: add order extraction". Seal the index with a message.
  5. git status. Check that nothing is left over.

The text after -m is for your teammates and your future self. "fix date parsing for Pune orders" is useful. "updates" is not.

Status looks, add stages, commit records, push shares.

CommandWhat it doesWhat it changes
git statusShows the state of every file.Nothing. It only looks.
git add ingest.pyCopies the file into the index.The index. Your file stays as it is.
git commit -m "ingest v1"Seals the index into a commit.Adds a commit to history.
git pushSends commits to a shared copy such as GitHub.Needs a remote. Not available in this tab.

Worked pass

Here is a real session. We made a new file, ingest.py, and ask Git what it sees.

ShellBefore git add (shown for reading, not run here)
$ git status
On branch main
Untracked files:
  (use "git add <file>..." to include in what will be committed)
	ingest.py
nothing added to commit but untracked files present (use "git add" to track)

Untracked means the file is on your disk but Git has never recorded it. Now we add it and look again.

ShellAfter git add (shown for reading, not run here)
$ git add ingest.py
$ git status
On branch main
Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
	new file:   ingest.py

It is now under Changes to be committed. That means it is in the index. Now we commit.

ShellAfter git commit (shown for reading, not run here)
$ git commit -m "ingest v1"
[main 4d09b66] ingest v1
 1 file changed, 1 insertion(+)
 create mode 100644 ingest.py
$ git status
On branch main
nothing to commit, working tree clean

4d09b66 is the commit's short ID. working tree clean means your files, the index and the last commit all match.

Following one file across the three areas.

After thisWorking treeIndexHistory
You save ingest.pyingest.pyemptyno ingest.py
git add ingest.pyingest.pyingest.pyno ingest.py
git commit -m "ingest v1"ingest.pyemptyingest.py in commit 4d09b66

Staging only the fix

Now the realistic case. You fixed orders_load.py and left a scratch file notes.py. You add only the fix.

ShellTwo changed files, one commit (shown for reading, not run here)
$ git add orders_load.py
$ git status
On branch main
Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
	modified:   orders_load.py
Untracked files:
  (use "git add <file>..." to include in what will be committed)
	notes.py
$ git commit -m "skip refunded orders when loading"

Git shows both areas in one report. orders_load.py is staged and goes into the commit. notes.py stays untracked and is left out.

A shorter way to read status

The practice reads the short form, git status --short. Each line has two letters, then the file name.

ShellShort status (shown for reading, not run here)
$ git status --short
M  ingest.py
 M README.md
?? notes.txt
A  config.yml

The first letter is the index. The second is the working tree.

LineFirst letterSecond letterMeaning
M ingest.pyM(space)Changed and staged.
M README.md(space)MChanged but not staged.
?? notes.txt??Untracked. Git has never seen it.
A config.ymlA(space)New file, staged.

So a file is staged when the first letter is not a space and not a question mark. You will use that rule in the practice.

Keeping secrets and clutter out

A repository should hold your code and settings. It should not hold passwords, virtual environments or temporary files. A file called .gitignore lists what Git should skip.

ShellA starter .gitignore for data work
__pycache__/
*.py[cod]
.venv/
.env
*.pem
credentials.json
.scratch/
*.duckdb
Watch outDeleting a password in a later commit does not remove it

If you commit a password and delete it next time, it stays in the first commit, and anyone with the repo can read it with git log -p. Change the password first. That is the only step that truly protects you. Then clean the history.

Common beginner questions

Is Git the same as GitHub?

No. Git is the tool on your computer. GitHub and GitLab are websites that store a shared copy of a Git repo so a team can work together.

Can I undo a commit?

Yes. The safest way is git revert, which adds a new commit that cancels the old one. History stays honest.

Why does a data engineer need Git?

Pipelines, SQL models, scheduler files and cloud setup all live in Git. That is how teams review changes, test them and roll back.

From a changed file to a saved snapshot
  1. Git records snapshots of your project, so you can see what changed and why.

    1. snapshots
    2. history
  2. Three areas: the working tree is your files, the index is the next snapshot, and commits are history.

    1. working tree
    2. index
    3. commit
  3. git add copies a chosen file into the index. Files you do not add stay out.

    1. git add
    2. choose files
  4. git commit seals the index into a snapshot with a message.

    1. git commit
    2. message
  5. git status shows where every file is, at every step.

    1. git status
    2. clean tree
  6. A committed password stays in history. Change it right away.

    1. .gitignore
    2. change the password

What comes next

So far you worked alone on one line of history. Next you will work with a team: branches, pull requests and updating main.

Practice

Run the sample once to see the two commands as a list. Then you receive the lines of git status --short as text.

Write staged(lines) to return the names of the files that are staged in the index.

Going further: undoing mistakes, and what git remembers

Most beginner anxiety with Git comes from not knowing how to undo mistakes. Three mechanisms handle most recovery needs: commit --amend fixes the latest commit, git stash shelves unfinished work, and the reflog tracks every movement of HEAD so even hard resets can be recovered. Predict what the log shows at the end.

ShellAmend, stash, and recover from a reset (illustration, not run here)
git init -q
echo v1 > a.txt && git add a.txt && git commit -qm "add a"
echo v2 > a.txt && git commit -qam "updatee a"
git commit -q --amend -m "update a"
git log --format=%s
echo "half done" >> a.txt
git stash -q
git status --short
git stash pop -q
git status --short
git commit -qam "more work"
git reset -q --hard HEAD~1
git log --format=%s
git reflog --format='%gd %gs' | head -3
git reset -q --hard 'HEAD@{1}'
git log --format=%s
update a
add a
 M a.txt
update a
add a
HEAD@{0} reset: moving to HEAD~1
HEAD@{1} commit: more work
HEAD@{2} reset: moving to HEAD
more work
update a
add a

The typo in the commit message was corrected with git commit --amend, replacing the bad commit with 'update a'. git stash cleared the working tree, and git stash pop restored the pending change ('M a.txt').

Then git reset --hard discarded the 'more work' commit from the branch. The reflog recorded the move, and resetting to HEAD@{1} recovered the discarded commit.

Git recovery commands and their operational precautions.

SituationTool or commandPrecaution
Typo in the latest commit message (unpushed)git commit --amendNever amend a commit that teammates have already pulled
Need a clean working tree temporarilygit stash, then git stash popStashes are local and easily forgotten if left unpopped
Reset too far or lost a commit pointergit reflog, then reset to the old refThe reflog is local to your machine and expires after retention limits
Discard uncommitted edits in a tracked filegit restore <file>Discards uncommitted changes permanently; cannot be undone

Try a new scenario

You committed a file with a database password locally and have not pushed. How do you resolve it, and what changes if the commit was already pushed to origin/main?

Work it out, then open the answer

If unpushed: remove the secret, add the file to .gitignore, and amend or reset so the commit holding the secret disappears from branch history. If already pushed: treat the credential as compromised and rotate it in the database immediately, because rewriting remote history does not remove copies colleagues already fetched. After rotating, clean the Git history as a secondary step.

What is the fundamental difference between git add and git commit?

git add copies selected file changes into the staging area (index), preparing the exact content for the next snapshot. git commit permanently seals whatever is staged in the index into a snapshot with an author, timestamp, parent pointer, and cryptographic hash.

Close the lesson and explain it

Without scrolling up, explain what the Git reflog records and why a hard reset does not immediately destroy committed data.

Practicals · load into the editor

After you read the theory, run these in the pane on the right. They execute in this tab, no cluster.

Finished this lesson?

Mark it as read when you could explain it to someone else.

Rate:
Was this useful?