Day-2 Ops Capstone: Backup & Disaster Recovery
Operate a running Consul datacenter through a real incident: back it up, lose a server, lose quorum, then recover both the cluster and its data.
Lab Overview
This is the Day-2 operations capstone for the Consul 101 course. You start with a healthy three-server Consul datacenter running a demo service and KV data, then drive it through a full disaster-recovery drill exactly as an on-call operator would.
You will:
- Take a point-in-time snapshot backup with `consul snapshot save` and verify it with `consul snapshot inspect`
- Simulate a single-server failure, confirm the cluster stays healthy on 2 of 3 quorum, and clean up the dead peer with `consul operator raft remove-peer`
- Lose quorum entirely, recognize it from an empty leader and a failing write, and rebuild the cluster around the surviving server using a `raft/peers.json` recovery file
- Bring the failed servers back and rejoin them into a full three-node quorum
- Restore the snapshot to roll back data corruption that happened during the incident
The recovery you practice here, quorum first with `operator raft`, then data with `snapshot restore`, is the exact modern playbook taught in Section 12 of the course.
What You'll Learn
Take, inspect, and restore a Consul snapshot backup
Recognize the difference between a survivable minority failure and full quorum loss
Clean up a dead server with consul operator raft remove-peer while a leader still exists
Rebuild quorum around a surviving server with a raft/peers.json recovery file when no leader remains
Apply the quorum-first, data-second disaster-recovery playbook end to end
Prerequisites
Completion of the Consul 101 Section 12 (Day-2 Operations) lessons
Familiarity with the consul CLI (members, kv, catalog)
Basic Linux command line and systemd service management
Technologies Covered
Choose your plan
Simple, Transparent Pricing
Unlock full access to TeKanAid courses, labs, and bootcamps
Just exploring? Start free below. Want the full experience? Try Premium free for 7 days (card required, $0 today).
Pro
All courses, with lab scripts to run on your own machine
Renews automatically. Cancel anytime.
Final price verified at checkout.
- Full access to all courses
- Lab scripts to download and run on your own machine (hosted labs not included)
- Progress tracking
- Certificate of completion
- Community access
- Self-paced bootcamps
- Premium puzzle library not included (5 free puzzles available)
- New content access
Premium
Full access, including unlimited hosted labs
Renews automatically. Cancel anytime.
Final price verified at checkout.
- Everything in Pro
- Unlimited hands-on labs, fully hosted on TeKanAid Academy (nothing to set up)
- Full access to the puzzle library
- Lab AI Assistant
- Priority support
Prefer a single course?
Purchase individual courses for a one-time fee of $79. Full access to course content, quizzes, certificates, and community features, lab access is not included.
Browse CoursesJust exploring? Start free, no account needed
Three free ways to start. All bridge into the paid Premium catalog when you're ready.
Not ready to commit? The crash course is email-only. No academy account required.
Ready to Get Started?
Start this hands-on lab and build real-world Platform Engineering skills
Get Access Now