🛠️ Essential Git Commands for Data Engineers: A Practical Guide

Posted on Mar 28, 2025

Git Logo

Git is an indispensable tool for data engineers managing code, configurations, and data pipelines. This guide covers essential Git commands with practical examples tailored for data workflows.

🔍 git diff - Inspecting Changes

📌 Use case: Review modifications before staging or committing

# Show unstaged changes
git diff

# Compare staged changes with last commit
git diff --cached

# Compare between two branches
git diff main..feature-branch

# Check changes to specific file
git diff data_pipeline.py

For data engineers, git diff is particularly useful when:

  • 🔄 Reviewing changes to SQL scripts
  • 📊 Comparing different versions of data transformation logic
  • ⚙️ Checking modifications to configuration files

↩️ git revert - Safe Undo

📌 Use case: Create a new commit that undoes a previous commit

# Revert a specific commit
git revert abc1234

# Revert the last commit
git revert HEAD

💡 Key points:

  • 🛡️ Doesn’t rewrite history (safe for shared branches)
  • ✨ Creates a new commit with inverse changes
  • 🚀 Ideal for fixing production issues without disrupting commit history

⏮️ git reset - Rewriting History

📌 Use case: Remove commits from branch history

# Soft reset (keeps changes in staging)
git reset --soft HEAD~1

# Mixed reset (keeps changes unstaged)
git reset HEAD~1

# Hard reset (discards changes completely)
git reset --hard HEAD~1

🔧 Data engineering scenarios:

  • --soft: When you want to recommit with additional changes
  • --hard: When you need to completely discard experimental changes

⚠️ Warning: Only use --hard on local branches, never on shared branches

🧹 git rebase - Clean History

📌 Use case: Maintain linear project history

# Rebase current branch onto main
git checkout feature-branch
git rebase main

# Interactive rebase (last 3 commits)
git rebase -i HEAD~3

🎯 Benefits for data pipelines:

  • 🧼 Eliminates unnecessary merge commits
  • 🧩 Allows squashing related changes
  • 🔍 Makes bisecting easier for debugging pipeline issues

📦 git stash - Temporary Storage

📌 Use case: Switch contexts without committing

# Stash current changes
git stash

# Stash with message
git stash save "WIP: data validation"

# List stashes
git stash list

# Apply most recent stash
git stash pop

# Apply specific stash
git stash apply stash@{2}

💼 Perfect for when you need to:

  • 🔄 Quickly switch branches to fix a production issue
  • 🧪 Test someone else’s changes without committing your WIP
  • 🧹 Temporarily remove changes to run clean tests

🍒 git cherry-pick - Selective Commits

📌 Use case: Apply specific commits to another branch

git checkout main
git cherry-pick abc1234

📊 Data engineering applications:

  • 🚑 Porting hotfixes between release branches
  • 🏭 Moving specific pipeline improvements to production
  • 🧪 Extracting experimental changes from feature branches

🏷️ git tag - Version Markers

📌 Use case: Mark important milestones

# Create annotated tag
git tag -a v1.2.0 -m "Release version 1.2.0"

# Push tags to remote
git push origin v1.2.0

# List tags
git tag -l

🚀 Essential for data pipeline management:

  • 🏭 Tagging production releases
  • 📈 Marking dataset versions
  • 🤖 Identifying model training checkpoints

🧩 git submodules - Component Management

📌 Use case: Include external repositories

# Add a submodule
git submodule add https://github.com/team/shared-utils.git

# Clone repo with submodules
git clone --recurse-submodules https://github.com/user/data-project.git

# Update submodules
git submodule update --remote

🔧 Common data engineering uses:

  • 📚 Incorporating shared data validation libraries
  • 🏗️ Managing common pipeline components across projects
  • 🧠 Version-controlling machine learning model repositories

🚀 Putting It All Together: Sample Workflow

# Start new feature
git checkout -b feature-data-cleaning

# Make changes
vim cleaning_script.py

# Stash temporary work
git stash save "WIP: outlier detection"

# Pull latest changes from main
git checkout main
git pull
git checkout feature-data-cleaning
git rebase main

# Continue working
git stash pop

# Commit and push
git add cleaning_script.py
git commit -m "Implement robust data cleaning"
git push origin feature-data-cleaning

# Tag release
git tag -a v1.3.0-beta -m "Beta release for testing"
git push origin v1.3.0-beta

🌟 Pro Tip: Create aliases for frequently used commands in your ~/.gitconfig:

[alias]
    st = status
    ci = commit
    co = checkout
    br = branch
    lg = log --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)<%an>%Creset' --abbrev-commit

Mastering these Git commands will significantly improve your efficiency as a data engineer, enabling better collaboration and more reliable data pipeline management.

📚 Further Reading: