Playing Traffic Cop, Part Two: When the Cop Pulls Over Everybody

Hey everyone,

A little follow-up to my original Playing Traffic Cop - Limiting the bandwith Hive-Engine can use post.

In that post, I wrote about using Linux tc (Traffic Control) to keep my Hive Engine node from eating my entire upload connection.

I still like calling it "traffic cop," because it is funnier and it makes the concept easier to explain.

But during the recent Moon migration, my traffic cop got a little too enthusiastic.

It was not just pulling over the Hive Engine traffic.

It was pulling over everybody.

The Symptom

While uploading the Moon server backup to the new Hetzner box, the transfer felt slower than it should have.

Not unusable.

Just suspiciously slow.

I was seeing SSH upload speeds around the same range as the limit I had intentionally set for Hive Engine traffic. After I temporarily disabled my traffic control service, the upload speed jumped immediately.

That was the clue.

My bandwidth limiter was not only limiting the service ports I cared about. It was also catching unrelated SSH uploads.

The Original Mistake

The original script created one HTB class capped at 4mbit:

/sbin/tc qdisc add dev "$DEV" root handle 1: htb default 10
/sbin/tc class add dev "$DEV" parent 1: classid 1:10 htb rate "$RATE" ceil "$RATE"

Then it added filters for the Hive Engine related source ports:

PORTS=(443 5001 5002)

for PORT in "${PORTS[@]}"; do
  /sbin/tc filter add dev "$DEV" protocol ip parent 1:0 prio 1 u32 \
    match ip sport "$PORT" 0xffff flowid 1:10
done

At first glance, that looks reasonable.

Match source port 443, 5001, and 5002, then send that traffic into the limited class.

And for a local service, sport is the right direction. When outside clients connect to my node, the outbound replies from my server have those local source ports.

The problem was this part:

default 10

That tells tc where unmatched traffic should go.

And in my script, unmatched traffic went to class 1:10.

Which was the limited class.

So yes, the filters were targeting the Hive Engine ports. But every packet that did not match those filters also landed in the same speed-limited bucket.

Including SSH uploads.

Oops.

The Fix

The fix was simple: create a separate default class for everything else.

The limited traffic still goes to 1:10, but normal traffic falls into 1:20 instead.

DEV="enp0s31f6"
RATE="4mbit"
DEFAULT_RATE="1000mbit"
PORTS=(443 5001 5002)

/sbin/tc qdisc add dev "$DEV" root handle 1: htb default 20

# Limited Hive Engine / Caddy traffic
/sbin/tc class add dev "$DEV" parent 1: classid 1:10 htb rate "$RATE" ceil "$RATE"

# Everything else
/sbin/tc class add dev "$DEV" parent 1: classid 1:20 htb rate "$DEFAULT_RATE" ceil "$DEFAULT_RATE"

for PORT in "${PORTS[@]}"; do
  /sbin/tc filter add dev "$DEV" protocol ip parent 1:0 prio 1 u32 \
    match ip sport "$PORT" 0xffff flowid 1:10
done

Now the selected service ports are still capped at 4mbit, but unrelated traffic is no longer forced into that same cap.

That means Hive Engine can keep behaving itself without making my SSH uploads crawl.

One Important Detail

This still is not "limit only this one application" in a perfect process-aware sense.

Because I am matching source ports, anything serving from those ports gets limited.

For me, that is fine.

Caddy traffic on 443 can be throttled. The Hive Engine P2P/RPC ports can be throttled. The couple of other local web services on the box are not important enough to need full upstream speed.

If someone needed more precision, they could look at firewall marks, cgroups, containers, or shaping by service user.

I do not need that much machinery here.

I just need the node to stop stepping on the rest of the house.

The Lesson

Linux tc is powerful, but it is also very literal.

It does exactly what you told it to do.

Not what you meant.

In this case, I meant:

throttle these service ports

But I accidentally wrote:

throttle these service ports, and also throttle everything else by default

The good news is that the traffic cop is back on duty now, with a slightly better understanding of who it is supposed to pull over.

Hive Engine stays in its lane.

SSH uploads get to pass.

The internet stays usable.

As always,
Michael Garcia a.k.a. TheCrazyGM