Break-fix Challenges
The first ten minutes of an incident: read the symptom, find the fault in logs, processes, disk and permissions, fix it, and prove it is fixed.
Triage
Read before you change anything. Write down what you change.
tail -n 50 /var/log/nginx/error.logThe newest errors first; the last line is usually the one that matters
journalctl -u myapp --since "-15min" --no-pagerA service's log for the last fifteen minutes
systemctl status myappState, restarts, main PID and the last log lines of a service
grep -n -i -E 'error|fatal|denied|refused' app.log | tailThe interesting lines of a long log, with line numbers
ls -lt --time-style=+%F_%T /etc/myapp/ | headWhat changed most recently (a deploy leaves fresh timestamps)
Processes and ports
ss -ltnpEvery listening TCP port and the process behind it
curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:3000/healthJust the status code of a health check
curl -v http://127.0.0.1:8080/The full exchange, to see where a request fails
ps -eo pid,etime,cmd --sort=etime | headThe newest processes first; a crash loop keeps them young
sudo systemctl reset-failed myapp && sudo systemctl start myappStart a service again after it hit its start limit
nohup ./start > app.log 2>&1 &Run something in the background so it survives the terminal
502 and friends
502 means the proxy is up but the app behind it did not answer properly.
sudo nginx -tTest the configuration before reloading
grep -n -E 'proxy_pass|server ' /etc/nginx/conf.d/*.confWhere nginx sends requests, and the upstream servers
connect() failed (111: Connection refused)Nothing listens on the upstream port. Is the app running, on that port?
SSL_do_handshake() failed ... wrong version numbernginx speaks https to an app that only speaks plain http
[Errno 98] Address already in useAnother process already has that port
Disk full
df -hWhich filesystem is full
du -ah /srv | sort -h | tail -n 15The biggest files and folders under a path
truncate -s 0 /var/log/app/debug.logEmpty a log the app holds open (rm would not free the space until a restart)
sudo lsof +L1Deleted files still held open, still taking space
gzip /var/log/app/app.log.[1-9]Compress rotated logs in place
sudo logrotate -d /etc/logrotate.d/appDry-run the rotation rules and see what they would do
Permissions
namei -l /var/www/shop/index.htmlThe mode of every folder on the way to a file
find /var/www/shop -type d -exec chmod 755 {} +Folders others may enter and list
find /var/www/shop -type f -exec chmod 644 {} +Files others may read but not change
find /var/www -perm -002 -lsAnything writable by every user (should be nothing)
chmod 600 secrets.envPrivate to the owner, as an app or ssh expects
chmod +x deploy.shLet a script be run as a program
Cron jobs
crontab -lThe current user's jobs
30 2 * * * /opt/jobs/backup.shEvery day at 02:30. A * in the minute field means every minute
PATH=/usr/local/bin:/usr/bin:/binSet at the top of a crontab; cron's own PATH is only /usr/bin:/bin
0 3 * * * /opt/jobs/report.sh > /var/log/report-$(date +\%F).log 2>&1In a crontab an unescaped % ends the command; write \%
env -i HOME=$HOME PATH=/usr/bin:/bin /bin/sh -c '/opt/jobs/backup.sh'Run a job in an environment as bare as cron's
grep CRON /var/log/syslog | tailDid cron start the job, and when
Nothing matches that search.
Cheatsheet for the Break-fix Challenges path · Deboistech · All cheatsheets
