Get started
Cheatsheet

Break-fix Challenges

The first ten minutes of an incident: read the symptom, find the fault in logs, processes, disk and permissions, fix it, and prove it is fixed.

Practice this path

Triage

Read before you change anything. Write down what you change.

  • tail -n 50 /var/log/nginx/error.log

    The newest errors first; the last line is usually the one that matters

  • journalctl -u myapp --since "-15min" --no-pager

    A service's log for the last fifteen minutes

  • systemctl status myapp

    State, restarts, main PID and the last log lines of a service

  • grep -n -i -E 'error|fatal|denied|refused' app.log | tail

    The interesting lines of a long log, with line numbers

  • ls -lt --time-style=+%F_%T /etc/myapp/ | head

    What changed most recently (a deploy leaves fresh timestamps)

Processes and ports

  • ss -ltnp

    Every listening TCP port and the process behind it

  • curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:3000/health

    Just the status code of a health check

  • curl -v http://127.0.0.1:8080/

    The full exchange, to see where a request fails

  • ps -eo pid,etime,cmd --sort=etime | head

    The newest processes first; a crash loop keeps them young

  • sudo systemctl reset-failed myapp && sudo systemctl start myapp

    Start a service again after it hit its start limit

  • nohup ./start > app.log 2>&1 &

    Run something in the background so it survives the terminal

502 and friends

502 means the proxy is up but the app behind it did not answer properly.

  • sudo nginx -t

    Test the configuration before reloading

  • grep -n -E 'proxy_pass|server ' /etc/nginx/conf.d/*.conf

    Where nginx sends requests, and the upstream servers

  • connect() failed (111: Connection refused)

    Nothing listens on the upstream port. Is the app running, on that port?

  • SSL_do_handshake() failed ... wrong version number

    nginx speaks https to an app that only speaks plain http

  • [Errno 98] Address already in use

    Another process already has that port

Disk full

  • df -h

    Which filesystem is full

  • du -ah /srv | sort -h | tail -n 15

    The biggest files and folders under a path

  • truncate -s 0 /var/log/app/debug.log

    Empty a log the app holds open (rm would not free the space until a restart)

  • sudo lsof +L1

    Deleted files still held open, still taking space

  • gzip /var/log/app/app.log.[1-9]

    Compress rotated logs in place

  • sudo logrotate -d /etc/logrotate.d/app

    Dry-run the rotation rules and see what they would do

Permissions

  • namei -l /var/www/shop/index.html

    The mode of every folder on the way to a file

  • find /var/www/shop -type d -exec chmod 755 {} +

    Folders others may enter and list

  • find /var/www/shop -type f -exec chmod 644 {} +

    Files others may read but not change

  • find /var/www -perm -002 -ls

    Anything writable by every user (should be nothing)

  • chmod 600 secrets.env

    Private to the owner, as an app or ssh expects

  • chmod +x deploy.sh

    Let a script be run as a program

Cron jobs

  • crontab -l

    The current user's jobs

  • 30 2 * * * /opt/jobs/backup.sh

    Every day at 02:30. A * in the minute field means every minute

  • PATH=/usr/local/bin:/usr/bin:/bin

    Set at the top of a crontab; cron's own PATH is only /usr/bin:/bin

  • 0 3 * * * /opt/jobs/report.sh > /var/log/report-$(date +\%F).log 2>&1

    In a crontab an unescaped % ends the command; write \%

  • env -i HOME=$HOME PATH=/usr/bin:/bin /bin/sh -c '/opt/jobs/backup.sh'

    Run a job in an environment as bare as cron's

  • grep CRON /var/log/syslog | tail

    Did cron start the job, and when

Cheatsheet for the Break-fix Challenges path · Deboistech · All cheatsheets