Upgrading MongoDB 8 to 9: Why Systemd Refused to Start and How to Bridge the FCV Gap

11 hours ago(edited)

Hey everyone,

If you run a Hive-Engine node (or manage a self-hosted MongoDB instance on Linux), you might run into a nasty surprise after bumping your apt repositories for MongoDB 9.0.

You configure the official repository, run sudo apt update && sudo apt dist-upgrade, let the package manager do its thing, and wait for mongod to spin back up.

Except it doesn't.

systemctl status mongod shows the service failed immediately on boot with exit code 62, leaving you staring at an offline database and wondering whether your node's 400+ GB dataset just got toasted.

The good news: your data is completely fine.

The catch: you've stepped right into MongoDB's rapid release compatibility trap.

Here is what went wrong, why MongoDB 9.0 flat-out refused to boot, and the step-by-step process to safely bridge the gap (including the replica set details that Hive-Engine strictly enforces) so you can get your node back online with zero data loss.

The Symptoms: Exit Code 62 on Startup

Immediately after upgrading packages from MongoDB 8.2 to 9.0, systemctl status mongod reported:

× mongod.service - MongoDB Database Server
     Loaded: loaded (/usr/lib/systemd/system/mongod.service; enabled; preset: enabled)
     Active: failed (Result: exit-code)
    Process: 3053614 ExecStart=/usr/bin/mongod --config /etc/mongod.conf (code=exited, status=62)
   Main PID: 3053614 (code=exited, status=62)

Exit code 62 in MongoDB means server initialization failed during startup checks before opening network listeners.

To see the real reason, checking /var/log/mongodb/mongod.log (or journalctl -u mongod) revealed this fatal log entry:

{
  "s": "F",
  "c": "CONTROL",
  "id": 20573,
  "ctx": "initandlisten",
  "msg": "Wrong mongod version",
  "attr": {
    "error": "UPGRADE PROBLEM: Found an invalid featureCompatibilityVersion document (ERROR: Location4926900: Invalid featureCompatibilityVersion document in admin.system.version: { _id: \"featureCompatibilityVersion\", version: \"8.2\" }. See https://docs.mongodb.com/master/release-notes/8.0-compatibility/#feature-compatibility. :: caused by :: Invalid feature compatibility version value '8.2'. Expected one of the following versions: '9.0', '8.3', '8.0'.)"
  }
}

Notice the crucial detail:

Invalid feature compatibility version value '8.2'. Expected one of the following versions: '9.0', '8.3', '8.0'.

Immediately after emitting this error, mongod initiated an orderly rollback checkpoint and shut itself down cleanly without touching or modifying any user collections.

Why Did This Happen? The Rapid Release Trap

MongoDB follows two release tracks:

Major / LTS Releases: (e.g., 7.0, 8.0, 9.0)
Rapid Releases: quarterly minor feature releases between major versions (e.g., 8.1, 8.2, 8.3)

Every MongoDB database contains an internal setting called featureCompatibilityVersion (FCV) stored in the admin.system.version collection. FCV controls which on-disk features, file structures, and internal behaviors the engine is permitted to use.

MongoDB enforces strict rules when upgrading:

You cannot skip versions.
A major release binary only knows how to handle datasets from the previous LTS release (e.g., FCV 8.0) or the final rapid release immediately preceding the new major version (e.g., FCV 8.3).

Because my machine was previously tracking the rapid release channel on MongoDB 8.2 (8.2.12), its internal FCV was set to "8.2".

When the MongoDB 9.0 binary checked the metadata at startup, it saw an FCV of "8.2". But 9.0 only knows about '8.0', '8.3', and '9.0'. It does not support jumping directly from 8.2 to 9.0.

Furthermore, you cannot simply tell 8.2 to downgrade its FCV to 8.0. MongoDB only allows setting FCV to its current minor version or the immediately prior one (8.2 or 8.1).

The only supported bridge to 9.0 is MongoDB 8.3.

The Hive-Engine Factor: Replica Sets Are Mandatory

There is another crucial detail every node operator must know: Hive-Engine strictly forces MongoDB to run as a replica set (typically replSetName: "rs0" in /etc/mongod.conf), even when running on a single local server.

Hive-Engine relies heavily on multi-document ACID transactions and change stream / oplog tailing to process smart contracts deterministically. A standalone MongoDB instance simply will not run Hive-Engine.

This requirement introduces three essential rules during this upgrade process:

Startup is not instantaneous: Every time mongod starts, it must initialize its replication coordinator, replay the oplog, and run an election to become PRIMARY.
FCV commands require PRIMARY state: You cannot execute db.adminCommand({ setFeatureCompatibilityVersion: ... }) while the node is in STARTUP, STARTUP2, or SECONDARY. Running it too quickly after starting systemd will fail with an error like node is not primary or not master and slaveOk=false.
Always verify replication health: Before shutting down 8.3 (and after bringing up 9.0), you must check rs.status() to ensure the replica set is fully caught up and healthy.

The Solution: The 8.3 Stepping Stone

The fix is a sequential upgrade:

Temporarily install MongoDB 8.3.
Boot the 8.3 service (which understands FCV 8.2).
Raise featureCompatibilityVersion to "8.3".
Shut down 8.3.
Upgrade to MongoDB 9.0.
Boot 9.0 (which accepts FCV "8.3").
Finalize featureCompatibilityVersion to "9.0".

Here is the exact playbook to execute this safely on Debian/Ubuntu systems.

Step 1: Temporarily Add the MongoDB 8.3 Repository

Create a temporary APT sources file for MongoDB 8.3. MongoDB 8.3 uses the 8.0 signing key:

sudo tee /etc/apt/sources.list.d/mongodb-org-8.3.sources << 'EOF'
Types: deb
URIs: https://repo.mongodb.org/apt/debian/
Suites: bookworm/mongodb-org/8.3
Components: main
Architectures: amd64
Signed-By: /usr/share/keyrings/mongodb-server-8.0.gpg
EOF

(Note: Adjust the suite name if using Ubuntu or another Debian distribution codename.)

Update your package index:

sudo apt update

Step 2: Downgrade Packages to MongoDB 8.3

Before downgrading, back up your MongoDB configuration just in case:

sudo cp -p /etc/mongod.conf /etc/mongod.conf.bak

Now install the MongoDB 8.3 packages (specifying version 8.3.11 or the latest available 8.3 build) and pass --allow-downgrades:

sudo apt install -y --allow-downgrades -o Dpkg::Options::="--force-confold" \
  mongodb-org=8.3.11 \
  mongodb-org-server=8.3.11 \
  mongodb-org-database=8.3.11 \
  mongodb-org-database-tools-extra=8.3.11 \
  mongodb-org-mongos=8.3.11 \
  mongodb-org-shell=8.3.11 \
  mongodb-org-tools=8.3.11

Confirm that the binary is now on 8.3:

mongod --version

You should see:

db version v8.3.11

Step 3: Start MongoDB 8.3 and Advance FCV to 8.3

Start the systemd service:

sudo systemctl start mongod
sudo systemctl status mongod

Because MongoDB 8.3 supports datasets with FCV 8.2, it will start up cleanly.

Wait for Replica Set Election (Mandatory for Hive-Engine)

Because Hive-Engine enforces running as a replica set, give mongod a few seconds after startup to initialize the replication coordinator and elect itself PRIMARY. If you attempt to update the feature compatibility version while the node is still in STARTUP2 or SECONDARY, the command will be rejected.

Confirm that your node has finished election and is PRIMARY:

mongosh --quiet --eval "rs.status().members[0].stateStr"

You should see:

PRIMARY

Once confirmed PRIMARY, inspect the current FCV:

mongosh --quiet --eval "db.adminCommand({ getParameter: 1, featureCompatibilityVersion: 1 })"

Output:

{
  "featureCompatibilityVersion": { "version": "8.2" },
  "ok": 1
}

Now upgrade the feature compatibility version to 8.3:

mongosh --quiet --eval "db.adminCommand({ setFeatureCompatibilityVersion: '8.3', confirm: true })"

Verify that the transition succeeded:

mongosh --quiet --eval "db.adminCommand({ getParameter: 1, featureCompatibilityVersion: 1 })"

It should now display:

{
  "featureCompatibilityVersion": { "version": "8.3" },
  "ok": 1
}

Confirm replica set health and write availability one more time before stopping the daemon:

mongosh --quiet --eval "rs.status().ok"

Step 4: Stop 8.3 and Clean Up the Temporary Repo

Stop the service cleanly:

sudo systemctl stop mongod

Remove the temporary 8.3 repository so APT doesn't get confused:

sudo rm -f /etc/apt/sources.list.d/mongodb-org-8.3.sources
sudo apt update

Step 5: Upgrade Back to MongoDB 9.0

Ensure your 9.0 repository file (e.g. /etc/apt/sources.list.d/mongodb-org-9.0.sources) is active:

cat /etc/apt/sources.list.d/mongodb-org-9.0.sources

Types: deb
URIs: https://repo.mongodb.org/apt/debian/
Suites: trixie/mongodb-org/9.0
Components: main
Architectures: amd64
Signed-By: /usr/share/keyrings/mongodb-server-9.gpg

Run the upgrade back to MongoDB 9.0:

sudo apt upgrade -y -o Dpkg::Options::="--force-confold" \
  mongodb-org \
  mongodb-org-server \
  mongodb-org-database \
  mongodb-org-database-tools-extra \
  mongodb-org-mongos \
  mongodb-org-shell \
  mongodb-org-tools

Verify the binary is back on version 9.0:

mongod --version

db version v9.0.2

Step 6: Start MongoDB 9.0 and Finalize FCV to 9.0

Start mongod.service:

sudo systemctl start mongod
sudo systemctl status mongod

Check the status: the service should now report active (running).

Confirm Replica Set Primary State

Once again, give the daemon a moment to finish its replica set election. Before attempting to update the FCV to 9.0, make sure the node is in PRIMARY state:

mongosh --quiet --eval "rs.status().members[0].stateStr"

Output:

PRIMARY

Next, connect with mongosh and inspect the current FCV:

mongosh --quiet --eval "db.adminCommand({ getParameter: 1, featureCompatibilityVersion: 1 })"

It will show that 9.0 is currently running in backward-compatibility mode with FCV 8.3.

Finally, lock in the 9.0 features by advancing the compatibility version:

mongosh --quiet --eval "db.adminCommand({ setFeatureCompatibilityVersion: '9.0', confirm: true })"

Verify the final status:

mongosh --quiet --eval "db.adminCommand({ getParameter: 1, featureCompatibilityVersion: 1 })"

Output:

{
  "featureCompatibilityVersion": { "version": "9.0" },
  "ok": 1
}

Lastly, verify that your replica set and Hive-Engine databases (hsc, hsc_history, etc.) are healthy and reporting correctly:

mongosh --quiet --eval "rs.status().ok"
mongosh --quiet --eval "db.adminCommand({ listDatabases: 1 })"

Everything is now fully migrated, running MongoDB 9.0 natively, and all your databases and collections remain intact.

Summary of Key Takeaways

Never skip MongoDB intermediate rapid releases when targeting a new LTS major. MongoDB 9.0 only supports incoming datasets with FCV 8.0 (from the prior LTS) or 8.3 (from the final rapid release).
Exit Code 62 with Wrong mongod version is non-destructive. MongoDB refuses to touch database files when an FCV mismatch is encountered. Do not panic and do not run mongod --repair blindly.
Hive-Engine forces replica sets: Because Hive-Engine requires a replica set for multi-document transactions and oplog processing, MongoDB nodes must complete election and achieve PRIMARY status on every restart before FCV commands can be executed. Running FCV updates during STARTUP2 will fail.
The bridge technique works smoothly with APT: By temporarily pulling in the missing intermediate release (8.3), you can boot the daemon, advance the internal FCV document, and then proceed with the target major release upgrade cleanly.

As always,
Michael Garcia a.k.a. TheCrazyGM

dev hive-engine pimp tribes archon proofofbrain arcadecolony

0.14958255 BEE

0 comments