> ## Content Index > Fetch the complete content index at: https://blog.melardenio.com/llms.txt > Use this file to discover other available public pages before exploring further. # The Router Wasn't the Problem - URL: https://blog.melardenio.com/the-router-wasnt-the-problem/ - Published: 2026-09-29T22:55:42.000Z - Updated: 2026-11-09T03:49:08.000Z - Author: Rommell Ardenio - Tags: AI & Automation, Software & Homelab, Broadcast Engineering Twenty-some years in broadcast engineering teaches you one thing above all else: when a signal path has a problem, the first thing you fix is rarely the actual thing that was wrong. You patch the symptom, the alarm clears, everyone moves on — and then three days later it comes back, usually at the worst possible time, and you have to admit you never actually found the fault. That's exactly what happened to me this week, except the "signal path" was a homelab server, and the fault lived somewhere I hadn't even thought to look. ## The fix that wasn't A few days earlier, a self-hosted GitHub Actions runner on my homelab box had started going dark — jobs would just sit there as "Waiting for a runner to pick up this job," forever. Digging into the runner's logs, the story looked like a classic path-MTU black hole: the TLS handshake to GitHub's broker would start and then just... stop. No error, no timeout message, nothing. Just silence. The fix, at the time, seemed obvious and it seemed to work: drop the interface MTU from 1500 to 1400\. Large packets stopped disappearing, the runner came back online, releases started flowing again. I even wrote it up as a permanent fix in my infrastructure notes — persisted the MTU change through netplan so it would survive a reboot, added a note recommending MSS clamping on the router for defense in depth. Case closed, or so I thought. ## It came back Two days later, a Telegram bot on the same host started timing out mid-conversation. Then, worse, a staging deploy pipeline failed outright with `git@ssh.github.com: Permission denied (publickey)` — a full connection failure, not a slow one. Same host. Same general shape of problem. The MTU fix that was supposed to have solved this clearly hadn't solved it — it had just made it happen less often. That's the tell I should have caught sooner. A "fix" that reduces the failure rate instead of eliminating it isn't a fix. It's a smaller version of the same bug, and it always comes back. ## The detail I'd been missing When we went back to actually re-diagnose it, one fact changed everything: this Linux box isn't bare metal. It's a VM running under Hyper-V. I hadn't mentioned that the first time around, mostly because it hadn't seemed relevant to a networking problem — it's just where the VM happens to live. It turned out to be the whole story. Hyper-V's synthetic network adapter (`hv_netvsc`, and you can spot it immediately by the Microsoft OUI on the MAC address, `00:15:5d:...`) has a well-documented history of botching TCP checksum and segmentation offload for traffic that rides inside a tunnel. This box reaches everything over Tailscale — WireGuard, UDP-encapsulated. Encapsulated traffic plus a buggy hardware offload is a specific, known combination: large packets get their checksums computed wrong by the "smart" NIC, the far end silently drops them, and it looks — convincingly — exactly like a path-MTU black hole. Small packets are fine. Big ones vanish. That's precisely what an MTU problem looks like from the outside, and it's precisely what my original fix "solved," by keeping packets small enough to dodge the bug most of the time. ``` rx-checksumming: on tx-checksumming: on tcp-segmentation-offload: on generic-segmentation-offload: on ``` Turning those off with `ethtool -K eth0 tx off sg off tso off gso off gro off` should, in theory, fix the actual corruption instead of just avoiding it. Time to test the theory. ## The part where I locked myself out Testing it properly meant putting the MTU back to normal 1500 and sending an actual 1450-byte packet with the "don't fragment" flag set, to see if it now survived the trip to GitHub. Except the route table had other ideas — `ip route show default` printed two identical default routes, one with the MTU clamp and one without, sitting at the same metric. Trying to cleanly replace one gave me `RTNETLINK answers: File exists`, because the kernel genuinely didn't know which of the two I meant. So I deleted both, meaning to add one clean route right back. In the gap between "delete" and "add," the SSH session died. For about ten seconds I had that specific stomach-drop feeling every network engineer knows — did I just lock myself out of a box with no other way in? (For the record: no. It's a Hyper-V VM, so there's always a console. But it's a good reminder to have that console handy before you start deleting routes on a machine you're connected to over the network you're about to break.) It came back on its own within a few seconds — Tailscale re-established once a default route existed again — and the single clean route went in without a fight. ## Finding the actual number With one clean route and no MTU override on it, I ran the test ping again. Still `message too long, mtu=1400`. Still failing. Which meant the MTU wasn't coming from the route at all — it was pinned directly on the interface itself, and a quick `ip link show eth0` confirmed it: MTU 1400, hard-set, independent of anything in the routing table. The source turned out to be exactly what I'd built three days earlier: `/etc/netplan/60-eth0-mtu.yaml`, my own "permanent fix" file, quietly overriding the interface back to 1400 every time netplan applied. I disabled that file, ran `netplan apply`, and tested again — this time for real, with both the MTU restriction gone and the offload bug patched: ``` PING github.com (140.82.121.3) 1450(1478) bytes of data. 1458 bytes from github.com: icmp_seq=1 ttl=49 time=197 ms 1458 bytes from github.com: icmp_seq=2 ttl=49 time=197 ms 1458 bytes from github.com: icmp_seq=3 ttl=49 time=197 ms 1458 bytes from github.com: icmp_seq=4 ttl=49 time=197 ms 0% packet loss ``` Clean. Consistent. No corruption. That's the actual fix — not a smaller version of the failure, an absence of it. I made the offload-disable persistent with a small `networkd-dispatcher` hook so it reapplies itself on every boot, DHCP renewal, or carrier bounce, instead of resetting the moment the VM restarts. ## A second bug hiding behind the first One more thing fell out of this. The `git@ssh.github.com: Permission denied (publickey)` failure from the staging deploy — the thing that first made me suspect the network was still broken — turned out to be completely unrelated. Once the network was actually clean, that same command connected fine, did a full handshake, and then failed instantly and cleanly with the exact same "permission denied." Instant and clean is not what a network black hole looks like; that's what a missing key looks like. Sure enough: the CI service account on that box had no SSH key pair at all. Not expired, not misconfigured — just never generated. Two people had spent real time chasing a phantom network regression that was actually just an onboarding step nobody had done. Generated a key, registered it as a deploy key on the repo, and the staging pipeline authenticated cleanly on the first try. ## The actual lesson None of this was really about MTUs or Hyper-V specifically. It was about the oldest trap in troubleshooting: a fix that makes the symptom rarer feels like a fix that made the problem go away, and those are not the same thing. If I'd asked "why did lowering the MTU help at all" instead of just confirming that it did, I'd have found the offload bug three days sooner. The router was never the problem. It never even got the chance to be — I just never looked past the first plausible explanation that made the pain stop. ## Related reading - [Cloudflare Tunnel 502 Bad Gateway: Removing a Dead Route, and What Access Hides From Scanners](https://blog.melardenio.com/cloudflare-tunnel-502-bad-gateway-removing-a-dead-route-and-what-access-hides-from-scanners/) - [Docker Build Cache on a Homelab: 152 GB Used, 17.7 GB Safely Freed](https://blog.melardenio.com/docker-build-cache-on-a-homelab-152-gb-used-17-7-gb-safely-freed/)