Wednesday, May 24, 2017
Paradise ponders: routers, sprinklers, and ARP proxies edition...
Yesterday I ripped out our two older MikroTik routers and replaced them with newer, (much) faster models (RB/1100AHX2). I brought up the one in my barn office first. That was a tedious, but fairly straightforward affair, and by 1 pm I had that up and running. After returning from a very pleasant lunch with Debbie and our friend Michelle H., I started on the house side. I had it all installed by 6pm, and it was talking between the house and the barn just fine (over my new radio link). But it wouldn't connect to the Internet at all.
I troubleshot it for three hours, with (to me!) very puzzling symptoms. The new MikroTik router could talk to the cable modem just fine, but nothing connected to the router could do so. Sounds like a routing problem, right? I inspected and re-inspected all the address and routing configuration, and found no problems at all. No reason for it not to work! So then I did some packet sniffing, using a constantly-running ping session on Debbie's workstation as a source of known data. The outbound ping got routed to the cable modem just fine, but then the cable modem never responded. Tried the same thing with a ping from the router, and the cable modem responded just fine. My tired brain couldn't process that information, so I went to bed and hoped that with fresh, caffeinated neurons in the morning I could figure it out.
Round about 2 am I woke up, visions of router configurations dancing in my head. After thinking about it some more, it occurred to me that it might be a problem with ARP (the Address Resolution Protocol). Unless you're a networking geek, you probably have no idea what that is. In technical terms, it gives networked devices a way to translate an IP address into an Ethernet destination (a MAC address). This isn't a great analogy, but it's a bit like a service that translates ZIP+4 codes into a street address. In terms of my problem, if the router was trying to send a packet to an IP address (in this case, Debbie's terminal) that it didn't know the corresponding MAC address for, it would broadcast an ARP request to all the devices directly connected to the modem, and then my new router should reply (because it already knows how to send something to that IP address). If that ARP request was never sent, or if my new router never replied to it, the symptoms would match what I was seeing.
Amazing what a little sleep will do for your troubleshooting capability!