MongoDB Replication Replica Sets Failover

What is a Replica Set?

A replica set is a group of MongoDB instances that maintain the same dataset. It provides redundancy and high availability. If the primary node fails, an automatic election promotes one of the secondaries to become the new primary - with no manual intervention required.

Role	Description
Primary	Receives all write operations. Only one primary per replica set at a time.
Secondary	Replicates data from the primary via the oplog. Can serve reads (with read preference).
Arbiter	Participates in elections but holds no data. Used to break ties in even-numbered sets.

Setting Up a 3-Node Replica Set

Initializing a Replica Set

// Start 3 mongod instances with --replSet flag
// mongod --replSet "rs0" --port 27017 --dbpath /data/rs0
// mongod --replSet "rs0" --port 27018 --dbpath /data/rs1
// mongod --replSet "rs0" --port 27019 --dbpath /data/rs2

// Connect to the first instance and initiate the replica set
mongosh --port 27017

rs.initiate({
  _id: "rs0",
  members: [
    { _id: 0, host: "localhost:27017", priority: 2 },  // preferred primary
    { _id: 1, host: "localhost:27018", priority: 1 },
    { _id: 2, host: "localhost:27019", priority: 1 }
  ]
})

// Check replica set status
rs.status()

// Check replica set configuration
rs.conf()

// Add a new member to an existing replica set
rs.add("localhost:27020")

// Add an arbiter
rs.addArb("localhost:27021")

// Remove a member
rs.remove("localhost:27020")

Understand A MongoDB Replica Set

A replica set contains one primary and one or more secondaries. Clients send ordinary writes to the primary. Secondaries copy the operation log and apply changes. If the primary becomes unavailable, eligible voting members elect a new primary. Replication improves availability but does not replace backups.

Use an odd number of voting members across independent failure domains. Connect with a replica-set connection string listing multiple hosts so the driver can discover topology changes. Applications must tolerate a short election window, retry safe operations, and use idempotency for writes whose result may be ambiguous after a network failure.

Write concern defines how many members acknowledge a write, while read concern controls the consistency of reads. readPreference determines which members may serve reads. Primary reads are simplest for fresh data. Secondary reads can reduce primary load but may return stale results and should match the product requirement.

Deploy an odd number of voting members.
Use a replica-set-aware connection string.
Expect brief write interruption during election.
Choose write concern from durability needs.
Use secondary reads only when staleness is acceptable.

Read Preferences

Read Preferences and Write Concern

// Read Preferences - control where reads are routed
// primary          - always read from primary (default, most consistent)
// primaryPreferred - read from primary if available, else secondary
// secondary        - always read from a secondary
// secondaryPreferred - read from secondary if available, else primary
// nearest          - read from the member with lowest network latency

// Set read preference in connection string
mongosh "mongodb://host1:27017,host2:27018,host3:27019/mydb?replicaSet=rs0&readPreference=secondaryPreferred"

// Set read preference per query
db.users.find({ active: true }).readPref("secondary")

// Write Concern - control acknowledgment level for writes
db.users.insertOne(
  { name: "Alice" },
  { writeConcern: { w: "majority", j: true, wtimeout: 5000 } }
)
// w: 1         - acknowledged by primary only (fast, less safe)
// w: "majority" - acknowledged by majority of replica set (recommended)
// j: true      - write must be journaled before acknowledgment
// wtimeout     - max milliseconds to wait for write concern

Monitoring Replication Lag

// Check replica set status and replication lag
rs.status()
// Look for: members[].optimeDate and members[].lastHeartbeatMessage

// Check oplog size and usage
use local
db.oplog.rs.stats()
db.oplog.rs.find().sort({ $natural: -1 }).limit(1)

// Step down the primary (triggers election)
rs.stepDown()

// Freeze a secondary (prevent it from becoming primary for N seconds)
rs.freeze(120)

// Check if current node is primary
db.isMaster()
// or
rs.isMaster()

Elections, Lag, Backups, And Failure Testing

Monitor replication lag, oplog window, member state, election frequency, network latency, and disk performance. A secondary that falls behind beyond the oplog window needs an initial sync. Hidden or delayed members support specialized recovery patterns but still require capacity and operational care.

Majority write concern and appropriate read concern provide stronger guarantees, but latency rises across distant regions. Place voting members according to failure and latency requirements, use priority and votes carefully, and avoid designs where one site can lose quorum unexpectedly. An arbiter votes but stores no data and does not improve durability.

Backups need consistent snapshots or database-supported tooling and should live outside the replica-set failure domain. Test point-in-time or snapshot restore into an isolated environment. Practice primary failure, member loss, network partition, full disk, and rollback scenarios while measuring application recovery and data correctness.

Monitor lag and oplog retention continuously.
Design quorum across real failure domains.
Understand durability versus latency tradeoffs.
Keep backups independent from replication.
Practice election and restore failure scenarios.

Initialize a three-member replica set

Run this only with correctly configured mongod members and network security.

Initialize a three-member replica set

rs.initiate({
  _id: \"rs0\",
  members: [
    { _id: 0, host: \"mongo1:27017\" },
    { _id: 1, host: \"mongo2:27017\" },
    { _id: 2, host: \"mongo3:27017\" }
  ]
});

rs.status();

Secure inter-member authentication.
Distribute members across failure domains.
Do not expose database ports publicly.

Driver connection with explicit concerns

The application discovers members and requests majority durability.

Driver connection with explicit concerns

const client = new MongoClient(
  \"mongodb://mongo1,mongo2,mongo3/app?replicaSet=rs0\",
  {
    readPreference: \"primary\",
    writeConcern: { w: \"majority\", wtimeoutMS: 5000 },
    readConcern: { level: \"majority\" }
  }
);

await client.connect();

Tune concerns from the operation requirement.
Handle server-selection timeout separately from query failure.
Make retried writes safe for the business operation.

Before you move on