PR #141: Letting Hive-Engine Lite Nodes Catch Up Before Cleaning House

Hey everyone,

In my last update, I broke down the Knobs A, B, and C changes and teased what was coming next. The latest piece of that puzzle, Knobs D, is now built, tested on live mainnet traffic, and submitted as PR #141 against the qa branch.

This round focuses squarely on lite-node operations: specifically, teaching the node to hold off on pruning local database history when it has more urgent work to do.

The Problem: Housekeeping During a Fire

Lite nodes stay small and efficient by periodically running cleanup jobs (database.cleanupLightNode()) to purge old blocks and transactions they no longer need. Under normal conditions, this background housekeeping runs quietly and everyone is happy.

The trouble starts when a node falls behind. Whether it's scheduled server maintenance, an unexpected reboot, or an upstream network hiccup, a node that wakes up thousands of blocks behind the chain head needs all the CPU cycles and disk I/O it can get just to ingest and validate incoming blocks.

Having MongoDB stop what it's doing every few minutes to execute heavy delete operations right in the middle of a catch-up sprint is the database equivalent of vacuuming the living room while the house is on fire. It causes unnecessary I/O contention and drags out the time it takes to get back to the head block.

The fix is straightforward in concept: make cleanup lag-aware. If the node is struggling to catch up, let it defer cleanup for a bit and focus on block ingestion.

Crucially, this doesn't touch block processing order, smart contracts, or consensus rules. It strictly changes when local pruning takes place.

The New Configuration Knobs

The adaptive cleanup behavior is completely opt-in and controlled under the lightNode configuration block:

{
  "lightNode": {
    "enabled": true,
    "adaptiveCleanup": true,
    "cleanupLagThreshold": 1000,
    "maxConsecutiveCleanupSkips": 3,
    "lastVerifiedBlockCacheTTL": 900000
  }
}

Here’s how these pieces work together:

• adaptiveCleanup: Enables the lag-aware logic. If left false (the default), the node behaves exactly as it always has.
• cleanupLagThreshold: How many blocks behind the node needs to be before it starts skipping cleanup passes. In this example, if the node is more than 1,000 blocks behind head, it postpones pruning.
• maxConsecutiveCleanupSkips: The safety valve. We can't let a node defer cleanup indefinitely while syncing a massive backlog, or local disk usage would balloon. Setting this to 3 means the node will skip at most three consecutive cleanup cycles before forcing a cleanup pass anyway, regardless of lag.
• lastVerifiedBlockCacheTTL: Caches the lookup for lastVerifiedBlockNumber in memory for 15 minutes (900,000 ms).

That last setting is a quiet efficiency win. Previously, every single cleanup pass did a full database lookup against witnesses.params to find the last verified block height. Since round verification only advances every ~40 minutes (~1,200 blocks), re-querying the database every cleanup cycle was pure waste.

Because verified block height only moves forward, a cached value is inherently safe: the absolute worst case is that the node retains an extra 15 minutes of old blocks before pruning them. It can never cause the node to delete data too aggressively.

Live Testing on a 19,000-Block Catch-Up

Unit tests are good for checking logic, but I always prefer seeing how these changes behave under actual pressure. I've been running this branch on my live witness setup since October 5th.

As luck would have it, my test node ended up falling about 19,000 blocks behind: giving me a far more realistic stress test than any tidy benchmark I could have staged.

Watching journalctl during the catch-up confirmed the safety limits were working exactly as designed. The node fell into a predictable rhythm: three skipped cleanup passes followed by one forced cleanup pass, repeating continuously while it processed the backlog.

Over the course of the run, the logs recorded 90 skipped cleanups and 30 forced cleanups. The lag steadily dropped from 19,000 blocks down past the 1,000-block threshold, at which point normal periodic cleanup resumed smoothly. The node is now sitting happily locked to the Hive head.

Designing for Failure: The Fallback Guardrail

One detail I was very deliberate about: what happens if the streamer encounters an RPC blip and can't accurately report chain lag?

A naive implementation might assume zero lag and clean up constantly, or worse, assume infinite lag and stop cleaning up forever.

Instead, if the streamer can't determine current lag, the node logs a clear warning and proceeds with the cleanup pass anyway. It’s always safer to fall back to the established default behavior when an input is uncertain. A kind of defensive edge-case handling that prevents an optimization from turning into a late-night operational headache.

Status and Verification

All of this work is wrapped up in PR #141, open against the qa branch of hive-engine/hivesmartcontracts.

The PR includes full configuration docs alongside focused database unit tests covering both the cleanup skip decision tree and the cached verified-block lookup (all tests passing cleanly).

No complicated magic here: just giving the database a breather when it has higher-priority work on its plate, backed by firm guardrails so the garbage still gets taken out.

Give it a look on qa, and let me know your thoughts!

As always,
Michael Garcia a.k.a. TheCrazyGM