What we want to do is to “merge” these two RS into a single one running mongodb 3.6.
Members of a single replica set need to share a common history of changes via the oplog. For members of two distinct replica sets the only way to merge data would be dumping & restoring data from the replica set you want to retire into the replica set you want to keep.
You could backup using
mongorestore, but may need something more bespoke depending on the overlap of data between your two replica sets. For example, if both replica sets have common databases you may want to merge, rename, or ignore duplicate databases & collections. Merging data would be more straightforward if both replica sets contain different database names.
We also want to update our OS, hardware and all.
I would approach this in several stages so you don’t conflate potential issues due to hardware or O/S upgrades with your major version upgrade and replica set consolidation. Changing many variables at once may save on elapsed time, but could dramatically complicate diagnostics and troubleshooting should anything not go to plan.
For example, you could:
- Provision new servers running the new O/S and latest version of MongoDB 3.4.x.
- Add a 3.4 secondary on a new server to your 3.4 replica set and ensure there are no unexpected issues. You may want to prevent this secondary from becoming primary during your test period.
- Add additional 3.4 secondaries on your new servers.
- Promote one of the new 3.4 secondaries to primary and retire your older 3.4 secondaries.
- Plan and test the replica set upgrade to MongoDB 3.6. Note: you may also need to upgrade your MongoDB drivers before upgrading your server.
- Create a plan to migrate the documents from your 2.6 replica set into your 3.6 deployment.
The upgrade of your 3.4 replica set as well as the major version upgrade from 3.4 to 3.6 should have negligible impact on availability so long as you follow rolling maintenance procedures and keep a strict majority of configured voting members online during replica set upgrades.
You could perform some of these steps in a different order (such as migrating documents from 2.6 to 3.4 prior to the 3.6 upgrade) if that better suits your business requirements.