Field notes

Field note: chasing a network loop across a multi-story building

A network loop kept knocking things offline. Loop protection didn't stop it. Tracking a single MAC address switch by switch finally did.

In a multi-story public building, the network started logging loop warnings, and parts of it went unstable. A router kept complaining that it was receiving packets carrying its own address as the source, a classic sign that traffic is circling back on itself through a loop.

The first loop: two trunks

The first time, the cause turned out to be two switch ports configured as trunks carrying every network, both connecting the same pair of switches. Traffic had two paths to take and circled endlessly. Fixing the switch configuration so only one path carried the traffic resolved it, though shutting down the wrong port briefly took part of the building offline first. That was a lesson in itself.

The second loop: when loop protection wasn’t enough

Weeks later, a loop showed up again, this time on the phone network. I enabled Loop Guard on the suspect switch port, and it did disable that port when it saw the loop. But the router kept logging loop errors. The loop wasn’t where I thought it was.

Tracking the MAC address

The loop warning included a specific MAC address. Following that address through each switch’s address table showed where it was being learned: one port on the main switch, and one port on a second switch. Disabling that port on the second switch stopped the log messages immediately.

That port turned out to be the uplink feeding the phones on another floor, through a small switch nobody had documented well. Then I enabled Rapid Spanning Tree (RSTP) on the switch, so it could detect and block redundant paths on its own going forward.

What I took away from it

  • Loop Guard protects a port. It doesn’t find a loop somewhere else. If errors continue, the loop is elsewhere.
  • The MAC address in the error is your map. Follow it through the switches’ address tables.
  • Turn on spanning tree (RSTP) wherever the switches support it. It’s the protection that keeps an accidental loop from taking the network down.
  • Label your ports. Half the time in this job went to figuring out what was plugged in where.
  • Before shutting a port, know what’s behind it.

All articles