In today’s tale, we find ourselves dealing with a recently replaced Meraki switch which is, once again, showing offline.
Previously, this site had an issue where the management of the switch was inoperable, and we assumed the only fix was to swap it out with a known good switch. Today, we are finding that the same issue cropped up. Any devices connected to the switch that have static IPs are completely fine as the VLAN configs of the ports are still intact.
The findings
We also use MX devices for routing and security at the sites, so we better start there.

There’s a handy-dandy API call we can leverage to figure out what’s plugged into the MX ports: https://api.meraki.com/api/v1/devices/<serialNumberHere>/lldpCdp
{
"ports": {
"port10": {
"cdp": {
"sourcePort": "port10",
"deviceId": "W60",
"address": "192.168.1.7",
"portId": "WAN PORT"
},
"lldp": {
"sourcePort": "port10",
"systemName": "SIP-W60B",
"portId": "00:11:22:33:44:55"
}
},
"port11": {
"cdp": {
"sourcePort": "port11",
"deviceId": "0c8d00001111",
"address": "192.168.1.4",
"portId": "Port 0"
},
"lldp": {
"sourcePort": "port11",
"systemName": "Meraki MR33",
"portId": "0"
}
},
"port12": {
"cdp": {
"sourcePort": "port12",
"deviceId": "T43U",
"address": "192.168.1.8",
"portId": "WAN PORT"
},
"lldp": {
"sourcePort": "port12",
"systemName": "SIP-T43U",
"portId": "80:5e:44:55:66:77"
}
},
"port3": {
"cdp": {
"sourcePort": "port3",
"deviceId": "ac1788889999",
"address": "192.168.0.5",
"portId": "Port 24"
},
"lldp": {
"sourcePort": "port3",
"systemName": "Meraki MS120-24",
"managementAddress": "192.168.0.5",
"portId": "24"
}
}
},
"sourceMac": "e0:55:dd:dd:ee:ee"
}
Lo and behold, there’s our missing switch, plugged into port 3. This is not the IP scheme we use internally, so something is definitely amiss, and furthermore, this IP scheme matches what we find below. This also tells us that we’ve got something plugged into port 4 & 7 (per the dashboard UI) which are not reporting LLDP info back. Due to the way we have the VLANs setup, I know what is in port 7. Port 4 is the interesting client.
What about the event logs? Let’s take a look back at the time period in question, when it was last online.
MX:

MX with event type filter:

Earlier in the day another device connected and reported an IP of 192.168.0.1. Generally, I’d expect a device with that IP to be the default gateway on the network, usually some type of routing device capable of running a DHCP server. Also, this IP scheme matches what we’re seeing the printer obtain. Taking our search to a MAC OUI lookup provider – https://www.wireshark.org/tools/oui-lookup.html – we find that the manufacturer is Zyxel. If you’ve been around DSL modems in the last decade or so, this should ring a bell. CenturyLink was a big client of theirs for residential and small business services.
Smoking Gun
Once the onsite team had been contacted, we sent them on the chase to find the mysterious Zyxel device. Within a few minutes they had identified an old CenturyLink modem that was never pulled from the network cage. Powering this device down caused the management interface of the Meraki switch to pick up an IP from the MX on the network and we were on our merry way. We had a few other DHCP enabled devices which we had to cycle switchports for, but nothing major.
Conclusion
I think the Meraki and Mist platforms get a fair amount of undeserved hate from the larger networking community. Are they capable of running complex networks or advanced routing protocols? No. But what they are good at is providing a reliable cookie-cutter network for branches or remote sites. Though the API is really what I love about Meraki, having the ‘single pane of glass’ view (when you run a full stack) is really convenient for troubleshooting sessions like this.


