During a routine merge of a feature branch into main, a developer encountered merge conflicts in a critical configuration file. After resolving the conflicts, the code was pushed and deployed, causing a production outage.

• Reviewed the deployment logs and identified the breaking change.

• Used git blame to find the commit that introduced the issue.

• Examined the merge commit and conflict resolution with git show.

• Found that critical configuration settings were incorrectly resolved during the merge.

The developer resolving the merge conflict didn't understand the implications of their choices and selected the wrong version of the code. No code review was required for merge conflict resolutions.

• Rolled back to the last known good deployment.

• Properly resolved the merge conflict with input from domain experts.

• Added tests to verify configuration validity.

Merge conflict resolution is a critical operation that requires careful review.

• Require code review for all merge conflict resolutions.

• Implement automated tests for configuration files.

• Use feature flags to safely deploy changes with easy rollback.

• Adopt trunk-based development to reduce large, complex merges.

• Train team members on proper merge conflict resolution techniques.

Clone, pull, and push operations started taking 10+ minutes instead of seconds. Developers reported that working with the repository had become painfully slow.

• Used git count-objects -v to check repository size.

• Found repository had grown to over 5GB.

• Ran git verify-pack -v .git/objects/pack/*.idx | sort -k 3 -n | tail -10 to identify large objects.

• Discovered hundreds of large binary files (images, videos, compiled binaries) committed to the repository.

Developers had been committing large binary assets directly to the repository instead of using Git LFS or external asset management.

• Used BFG Repo-Cleaner to remove large binary files from history:

bfg --strip-blobs-bigger-than 10M repo.git

• Set up Git LFS for necessary binary files.

• Created a separate asset management system for design files and media.

• Added pre-commit hooks to prevent large file commits.

Git performance degrades significantly with large binary files in history.

• Use Git LFS for binary files that must be version controlled.

• Store large assets in dedicated systems (S3, Artifactory, etc.).

• Implement pre-commit hooks to catch large files before they enter the repository.

• Educate team on Git best practices for handling binary files.

• Regularly audit repository size and content.

Developers reported that git clone, pull, and push operations were taking increasingly long times, sometimes timing out completely. The repository had grown to over 10GB in size despite the codebase itself being relatively small.

• Measured repository size with git count-objects -v -H.

• Analyzed repository history with git log --stat.

• Identified large files with git ls-files | xargs du -sh.

• Used git filter-branch to examine history.

• Found large binary files committed and later deleted but still in history.

Several large binary files (deployment artifacts, database dumps, and machine learning models) had been committed to the repository and later removed. However, Git's history retained these files, causing the repository to grow excessively large.

• Short-term: Created a fresh clone with limited history for immediate relief:

git clone --depth=50 --single-branch --branch=main git@github.com:org/repo.git

• Long-term: Used BFG Repo-Cleaner to remove large files from history:

# First, clone a mirror of the repository
git clone --mirror git@github.com:org/repo.git repo.git
# Use BFG to remove files larger than 50MB
java -jar bfg.jar --strip-blobs-bigger-than 50M repo.git
# Clean up and optimize the repository
cd repo.git
git reflog expire --expire=now --all
git gc --prune=now --aggressive
# Push the cleaned repository
git push

• Implemented Git LFS for managing large files:

# Install Git LFS
git lfs install
# Track large file types
git lfs track "*.bin" "*.model" "*.zip" "*.tar.gz"
# Ensure .gitattributes is committed
git add .gitattributes
git commit -m "Configure Git LFS for binary files"

Git repositories require careful management of large binary files.

• Use Git LFS for large binary files from the beginning.

• Implement pre-commit hooks to prevent large file commits:

• ```bash

• !/bin/bash

• .git/hooks/pre-commit

• Prevent committing files larger than 10MB

• max_size_kb=10240

• files_to_check=$(git diff --cached --name-only)

• for file in $files_to_check; do

• file_size_kb=$(du -k "$file" | cut -f1)

• if [ $file_size_kb -gt $max_size_kb ]; then

• echo "Error: $file is $file_size_kb KB, which exceeds the limit of $max_size_kb KB."

• echo "Consider using Git LFS for this file."

• exit 1

• fi

• done

• exit 0

• ```

• Document version control best practices for the team.

• Regularly monitor repository size.

After a routine update to a feature branch, the CI/CD pipeline started failing with errors about missing files. The failures occurred in a component that hadn't been modified in the recent changes.

• Examined CI/CD logs to identify the specific missing files.

• Checked Git submodule status with git submodule status.

• Compared submodule references between working and failing branches.

• Reviewed recent commits with git log -p -- .gitmodules.

• Tested submodule initialization and update locally.

The repository used Git submodules to include code from other repositories. A recent commit had inadvertently changed the submodule reference to point to a non-existent commit hash in the submodule repository. This happened because a developer had force-pushed to the submodule repository, rewriting history and removing the commit that was previously referenced.

• Short-term: Updated the submodule reference to point to a valid commit:

# Update submodule to latest commit on main branch
git submodule update --remote lib/component
git add lib/component
git commit -m "Update submodule reference to latest main"

• Long-term: Implemented proper submodule management practices:

# Create a script to safely update submodules
cat > scripts/update-submodules.sh << 'EOF'
#!/bin/bash
set -euo pipefail
# Ensure we're on a clean branch
if [[ -n $(git status --porcelain) ]]; then
  echo "Working directory not clean. Commit or stash changes first."
  exit 1
fi
# Update each submodule to the latest on its tracking branch
git submodule foreach 'git fetch origin && git checkout $(git rev-parse --abbrev-ref HEAD) && git reset --hard origin/$(git rev-parse --abbrev-ref HEAD)'
# Stage the updates
git add .
# Create a commit if there are changes
if [[ -n $(git status --porcelain) ]]; then
  git commit -m "Update submodules to latest versions"
  echo "Submodules updated and committed."
else
  echo "Submodules already up to date."
fi
EOF
chmod +x scripts/update-submodules.sh

• Added CI/CD checks to validate submodule references:

# GitHub Actions workflow
name: Validate Submodules
on: [push, pull_request]
jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with:
          submodules: recursive
      - name: Validate submodule references
        run: |
          # Check if any submodule is in detached HEAD state
          git submodule foreach 'git status | grep -q "HEAD detached" && echo "WARNING: $name is in detached HEAD state"'
          # Check if all submodule commits exist in their respective remotes
          git submodule foreach 'git fetch origin && git branch -r --contains HEAD || echo "ERROR: $name references a commit not in remote"'

Git submodules require careful management to prevent reference breakage.

• Use tagged releases for submodule references instead of branch HEADs.

• Implement CI/CD checks to validate submodule references.

• Consider alternatives like Git subtrees or package managers.

• Establish clear policies for force-pushing to shared repositories.

• Document submodule update procedures for the team.

A company's application consisted of a main repository with several Git submodules for shared libraries. Developers began reporting inconsistent behavior where code would build locally but fail in CI/CD pipelines. The failures were intermittent and difficult to reproduce. Investigation revealed that different branches had different submodule commit references, and the CI/CD pipeline was not properly synchronizing submodules before building.

• Analyzed CI/CD build logs for failure patterns.

• Compared submodule references between working and failing builds.

• Examined Git configuration and submodule initialization commands.

• Reviewed developer workflows for submodule updates.

• Tested different submodule synchronization approaches.

The investigation revealed multiple issues with submodule management: 1. Developers were not consistently updating submodule references after changes 2. The CI/CD pipeline was using a shallow clone that didn't properly fetch submodules 3. Some branches had outdated .gitmodules configurations 4. Submodule updates were not part of the code review process 5. No automated validation of submodule consistency was in place

• Updated CI/CD pipeline to properly initialize and update submodules

• Created pre-commit hooks to validate submodule references

• Implemented automated tests for submodule consistency

• Documented proper submodule workflow for developers

• Added submodule status checks to code reviews

Git submodules require careful management and consistent workflows to prevent integration issues.

• Implement proper submodule initialization in all environments.

• Create automated validation for submodule consistency.

• Document clear workflows for submodule updates.

• Consider alternatives like Git subtrees or package management.

• Include submodule changes in code review processes.