Helping Hive Engine Nodes Catch Up Without Getting Too Clever

Hey everyone,

I have been working through the next small performance bundle for the Hive-Engine node, and this one is about the part of the node that fetches Hive blocks.

The short version is:

I added opt-in adaptive fetch concurrency and dynamic lookahead buffering, tested it on my custom witness setup, and opened the changes toward the QA branch.

Making the node a little less stubborn when it has some catching up to do.

The Problem

The streamer had two fixed values controlling how aggressively it fetched blocks:

• how many block requests it could have in flight per RPC node
• how many blocks it prefetched ahead of the current block

Those defaults are reasonable for a node sitting near the chain head. They are not necessarily the best settings for a node that is hundreds or thousands of blocks behind after maintenance, downtime, or a restart.

A catching-up node should be able to make progress without changing how blocks are processed.

What Changed

The new behavior is opt-in.

When adaptive QPS is enabled, the streamer watches the existing per-node failure and cooldown information:

• healthy RPC nodes can use more fetch concurrency
• one struggling node brings the setting back to the configured baseline
• multiple struggling nodes cause a more conservative reduction
• the healthy-node ceiling is explicitly capped in configuration

I am calling this QPS because that is the existing configuration name, but technically this is a per-node in-flight request limit rather than a precise requests-per-second meter.

The lookahead buffer now adapts to sync lag as well:

• 5 blocks when the node is close to the head
• 20 blocks while catching up
• up to a configured maximum when it is far behind

The defaults remain unchanged unless an operator enables the new settings.

This is important because existing nodes should not suddenly decide to behave differently just because the code was updated.

The Config I Tested

For the live test, I used settings shaped like this:

{
  "maxQps": 2,
  "adaptiveQps": true,
  "adaptiveQpsMax": 4,
  "lookaheadBufferSize": 15,
  "dynamicLookaheadBuffer": true,
  "dynamicLookaheadBufferMaxSize": 50
}

The ceiling is there on purpose. Healthy RPC nodes are not an invitation to send an unlimited amount of traffic at public endpoints.

The Test

I let my node fall roughly 300 Hive blocks behind and watched it catch up.

Once the transition logging was made visible, the journal showed:

Streamer QPS changed to 4
Streamer lookahead buffer changed to 20

The node then worked its way back toward the Hive head. Sidechain blocks continued to advance, and the database and block hashes matched between the main reference values and the locally produced values.

Eventually the lag returned to zero.

One Honest Caveat

There was a transient MongoDB NoSuchTransaction error during one of the restarts. The service remained technically active, but block processing stalled until a second restart recovered it.

I am not treating that as proof that the adaptive fetch code caused it. I have seen this same kind of restart/session behavior before this bundle. Still, it is worth mentioning because production testing should include the ugly bits, not just the clean screenshots.

What This Does Not Change

This bundle changes fetch scheduling, not state transition logic.

It does not change:

• smart contract execution
• transaction ordering
• block hashing
• database state rules
• witness consensus behavior

The goal is simply to get the right blocks into the existing processing pipeline more efficiently while backing off when the RPC environment looks unhealthy.

The Outcome

The code is now in a PR prepared for QA branch.

The live test node caught up from roughly 300 blocks behind, reported the expected adaptive settings, and returned to the chain head without hash divergence in the sampled blocks.

As always,
Michael Garcia a.k.a. TheCrazyGM