Incident Response

1. Check the status page (coming soon: status.costplusdb.com)
   For now: I'll email you if there's a known outage

2. Verify it's not DNS or network:
   ping your-host.costplusdb.com
   (Should get a response - if not, it's DNS/network)

3. Test the connection:
   psql "postgresql://user@host:5432/db?sslmode=require"

   If you get "Connection refused" → Database is down (P0)
   If you get "Timeout" → Network issue (check your firewall)
   If you get authentication error → Database is up, wrong credentials

4. Email me immediately: jeremy@intentsolutions.io
   Subject: "P0: Database Down - [your company]"
   Include: Error messages, when it started, any recent changes

You don't need to tell me the database is down.

Automated alerts already notified me:
- Betterstack detected the outage within 1 minute
- My phone alerted me immediately
- I'm already investigating or working on it

What happens next:
1. I diagnose the issue (5-15 minutes)
   - Check if PostgreSQL is running
   - Review system logs
   - Check disk space, memory, CPU

2. I fix or restore (15-90 minutes depending on cause)
   - Simple fix: Restart services, clear locks
   - Complex: Restore from backup, migrate to new VPS

3. I notify you when resolved
   - Email with incident summary
   - Explanation of what happened
   - Steps to prevent recurrence

Data loss: Maximum 24 hours (last backup). Usually zero if it's a service issue, not data corruption.

These queries help us diagnose faster:

-- Check active connections
SELECT count(*) FROM pg_stat_activity;

-- Look for long-running queries
SELECT pid, now() - query_start AS duration, state, query
FROM pg_stat_activity
WHERE state = 'active'
  AND now() - query_start > interval '1 minute'
ORDER BY duration DESC;

-- Check for locks
SELECT pid, mode, granted, query
FROM pg_locks
JOIN pg_stat_activity USING (pid)
WHERE NOT granted;

Send me the results when you email. It speeds up diagnosis significantly.

• Too many connections: Check if you're hitting max_connections limit
• Long-running query: One slow query blocking others
• Missing index: Sequential scan on large table
• Disk space: Running low on storage
• Memory: Shared buffers exhausted

I'll check all of these when you contact me.

Email: jeremy@intentsolutions.io
Subject: "P1: Performance Issues - [your company]"

Include:
- When did it start?
- How much slower? (e.g., "queries taking 10x longer")
- Any recent changes? (new feature, traffic spike)
- Results of diagnostic queries above (if you ran them)

- Do NOT run DELETE or UPDATE statements to "fix" it
- Do NOT attempt to restore backups yourself
- Do NOT restart the database
- Do NOT run VACUUM FULL

Why? You might overwrite the data we need to recover.

1. Document what you observe:
   - Which records are missing/wrong?
   - When did you notice it?
   - When was the data last correct? (approximate time)
   - What operations were running? (imports, updates, deletes)

2. If possible, export the current state:
   pg_dump -h host -U user -d db -F c -f emergency-$(date +%Y%m%d).dump

   This preserves what's there now, before we restore.

3. Email me IMMEDIATELY:
   Subject: "P0: DATA LOSS - [your company]"
   Mark as urgent.

Recovery process:

1. Assess the scope (10-20 minutes)
   - What data is affected?
   - When did corruption occur?
   - What's the best recovery point?

2. Point-in-time recovery (30-90 minutes)
   - Restore from backup to specific timestamp
   - Replay WAL logs to exact recovery point
   - Verify data integrity

3. Validation (15-30 minutes)
   - Confirm data is restored correctly
   - Run sanity checks
   - Test application connections

4. Cutover (5-10 minutes)
   - Switch your application to restored database
   - Verify everything works
   - Monitor for issues

Total time: 1-2.5 hours typically

• Failed authentication attempts from unknown IPs (check logs)
• Unusual queries in pg_stat_activity (e.g., attempts to read pg_user)
• Unexpected connections from unfamiliar locations
• Sudden performance degradation (could be malicious queries)
• Your team didn't make changes but something changed

1. Do NOT investigate further
   (Preserve evidence, don't tip off attacker)

2. Note the time and what you observed:
   - What suspicious activity did you see?
   - When did you notice it?
   - Were there any unusual access patterns?

3. Email me immediately:
   Subject: "P0: SECURITY INCIDENT - [your company]"

4. If you suspect credentials are compromised:
   - Don't change them yet (wait for my guidance)
   - I need to review logs first

Security incident protocol:

1. Immediate assessment (5-15 minutes)
   - Review PostgreSQL logs
   - Check authentication logs (/var/log/auth.log)
   - Analyze fail2ban blocks
   - Check firewall logs (UFW)

2. Containment (15-30 minutes)
   - Block suspicious IPs
   - Rotate credentials if compromised
   - Update firewall rules
   - Enable additional logging

3. Investigation (30-60 minutes)
   - Determine attack vector
   - Assess data access
   - Check for data exfiltration
   - Review all recent connections

4. Remediation (varies)
   - Patch vulnerabilities
   - Update security policies
   - Implement additional monitoring
   - Notify you of findings

I take security very seriously. I'll give you a full incident report.

• Email from Healthchecks.io: "Backup check failed"
• Email from me: "Backup verification failed"
• You need to restore data but backup is missing/corrupt

Important: Backup failures are P0 if they put data at risk. If your database is still running normally, it's P1 (I'll fix the backup system).

Email: jeremy@intentsolutions.io
Subject: "P0: Backup Issue - [your company]"

Tell me:
- Did you receive a backup failure alert?
- Do you need to restore something right now? (urgent)
- Is your database still running? (if yes, less urgent)

If you need an emergency backup while waiting:
pg_dump -h host -U user -d db -F c -f backup-$(date +%Y%m%d).dump