Emergency Procedures

← Back to Home

Critical Emergency Contact

Database Down? Data Loss? Security Breach?

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
PRIMARY CONTACT (Jeremy Longshore)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Email:    jeremy@intentsolutions.io
Response: 4-hour SLA (business hours), typically 30-min
          7 days a week, including holidays
          Critical outages: IMMEDIATE (automated alerts 24/7)

EMERGENCY BACKUP OPERATOR (If no response >24 hours)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Activated automatically if primary contact unavailable
You will receive notification if EBO activates
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Severity Levels

Emergency Scenarios

1. Check your connection string
   - Verify host, port, username, password
   - Test: psql "postgresql://user:pass@host:5432/db"

2. Check your application logs
   - Look for connection errors
   - Note exact error message

3. Verify our status page
   - Visit: https://status.costplusdb.dev
   - Check if system-wide outage

1. DO NOT delete anything (preserve evidence)
2. Document what you observed (screenshots, log excerpts)
3. Change your database password immediately
4. Disconnect affected applications if possible

-- Check active connections
SELECT count(*) FROM pg_stat_activity;

-- Check for long-running queries
SELECT pid, now() - query_start AS duration, query
FROM pg_stat_activity
WHERE state = 'active'
ORDER BY duration DESC;

-- Check database size
SELECT pg_size_pretty(pg_database_size(current_database()));

Emergency Backup Operator System

What happens if primary operator (Jeremy) is unavailable?

Dead Man's Switch

We use open source monitoring tools to ensure continuity:

• Healthchecks.io: Primary operator checks in twice daily (8am, 8pm)
• Uptime Kuma: Monitors all database services 24/7
• Automated Alerts: If no check-in for 24 hours, Emergency Backup Operator (EBO) is notified
• EBO Activation: After 48 hours, EBO receives emergency credentials and access

EBO Emergency Access

NORMAL OPERATIONS (Read-Only)
├─ EBO can monitor infrastructure health
├─ EBO can view backup status
├─ EBO can see alert dashboards
└─ EBO CANNOT access customer data or make changes

EMERGENCY MODE (Dead Man's Switch Activated)
├─ EBO receives full credentials from secure vault
├─ EBO can restart databases, restore from backups
├─ EBO can respond to security incidents
├─ EBO notifies customers of incident and status
└─ All EBO actions logged and auditable

Your Protections

• EBO is bound by Non-Disclosure Agreement
• All EBO access is logged (you can request audit logs)
• EBO only accesses customer data when necessary for incident resolution
• You're notified immediately when EBO activates
• All credentials rotated after incident resolved

Full Details: Transparency Hub → Emergency Procedures

Open Source Emergency Tools

CostPlusDB uses industry-standard, open source tools for emergency management:

Why Open Source?

• Free (no vendor lock-in)
• Transparent (you can audit the code)
• Battle-tested (millions of users worldwide)
• Self-hosted (we control the data)
• Community support (massive ecosystems)

Setup:
- Self-hosted on VPS (https://github.com/healthchecks/healthchecks)
- Cron jobs ping every 12 hours
- If ping missed >24 hours → Alert sent to Jeremy + EBO

Cost: $0 (self-hosted)
Alternative: Dead Man's Snitch ($5/mo)

Setup:
- Self-hosted monitoring dashboard
- Checks: PostgreSQL (port 5432), HTTP health endpoint, disk space
- Alert destinations: Email, Slack, webhook

Cost: $0 (self-hosted)
Alternative: Betterstack ($10/mo), UptimeRobot ($7/mo)

Setup:
- Self-hosted Bitwarden vault (or Bitwarden cloud free tier)
- Stores: VPS passwords, PostgreSQL credentials, API keys
- Emergency Access: EBO can request, auto-granted after 48 hours

Cost: $0 (free tier) or $10/year (premium)
Alternative: 1Password Teams ($20/mo), KeePassXC (offline, free)

Setup:
- Prometheus scrapes metrics from PostgreSQL (postgres_exporter)
- Grafana visualizes performance, connections, query times
- Alertmanager sends alerts to Grafana OnCall

Cost: $0 (all open source, self-hosted)
Alternative: Datadog ($15/mo), New Relic ($25/mo)

Status Page

Real-time service status: status.costplusdb.dev

Check here first for:

• System-wide outages
• Scheduled maintenance windows
• Historical uptime data
• Incident reports

Status page powered by Uptime Kuma (open source)

Incident Post-Mortems

We publish detailed incident reports for all P0/P1 incidents:

• Root cause analysis
• Timeline of events
• Actions taken
• Prevention measures implemented

Recent Incidents: View all incident reports

No incidents yet (service launched October 2025)

Escalation Path

┌─────────────────────────────────────────────────────────┐
│ CUSTOMER REPORTS ISSUE                                  │
│ Email: jeremy@intentsolutions.io                        │
└─────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────┐
│ PRIMARY OPERATOR RESPONSE                               │
│ Expected: Typically 30-min (4-hour SLA business hours)  │
│ Critical outages: IMMEDIATE (automated alerts 24/7)     │
│ Action: Acknowledge, diagnose, resolve                  │
└─────────────────────────────────────────────────────────┘
                          ↓
           NO RESPONSE AFTER 30 MINUTES?
                          ↓
┌─────────────────────────────────────────────────────────┐
│ AUTOMATED RETRY                                         │
│ System re-sends alert to primary operator              │
│ Notification sent to EBO (informational only)          │
└─────────────────────────────────────────────────────────┘
                          ↓
           NO RESPONSE AFTER 2 HOURS?
                          ↓
┌─────────────────────────────────────────────────────────┐
│ DEAD MAN'S SWITCH CHECK                                │
│ EBO verifies: When was last check-in?                  │
│ If <24 hours: Wait and monitor                         │
│ If >24 hours: Proceed to emergency activation          │
└─────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────┐
│ EMERGENCY BACKUP OPERATOR ACTIVATED                     │
│ EBO receives credentials, resolves incident             │
│ Customer notified: "EBO is responding to your issue"    │
└─────────────────────────────────────────────────────────┘

Customer Self-Service Emergency Tools

Emergency Database Export (pg_dump)

If you need to migrate away immediately:

# Export your entire database
pg_dump -h YOUR_HOST -p 5432 -U YOUR_USER -d YOUR_DB \
  -F c -f backup.dump

# Export specific table
pg_dump -h YOUR_HOST -p 5432 -U YOUR_USER -d YOUR_DB \
  -t table_name -F c -f table_backup.dump

# Export as SQL (human-readable)
pg_dump -h YOUR_HOST -p 5432 -U YOUR_USER -d YOUR_DB \
  -f backup.sql

You always have superuser access - You can export your data anytime, no permission needed from us.

Emergency Read-Only Mode (If Database Corrupting)

-- Make database read-only temporarily
ALTER DATABASE your_database_name SET default_transaction_read_only = on;

-- Revert to read-write
ALTER DATABASE your_database_name SET default_transaction_read_only = off;

This prevents further damage while we diagnose the issue.

Legal & Compliance

Data Breach Notification (GDPR/CCPA)

If we detect unauthorized access to customer data:

1. 0-4 hours: We notify you immediately via email (P0 priority)
2. Within 24 hours: We provide preliminary incident report
3. Within 72 hours: We provide full forensic report for your compliance filing

Your Responsibilities:

• GDPR: Notify your customers within 72 hours if their personal data accessed
• CCPA: Notify California residents without unreasonable delay
• We provide breach notification template and assistance

Business Associate Agreement (BAA) - HIPAA

Not available yet (Month 12+ roadmap)

If you handle healthcare data:

• CostPlusDB is not HIPAA-compliant at launch
• Planned for Month 12+ with AWS/GCP infrastructure upgrade
• BAA signing available with Enterprise tier ($299/mo) + AWS upgrade (+$91/mo)

Need Help Now?

Email: jeremy@intentsolutions.io

Subject Line Format:

• P0 CRITICAL: [Brief description]
• P1 HIGH: [Brief description]
• P2 MEDIUM: [Brief description]

Expected Response Times:

• Regular support: 4-hour SLA (business hours: M-F 9am-6pm ET), typically 30-min response, 7 days/week
• Critical issues (P0): IMMEDIATE response - automated alerts 24/7
• All customers receive same service level