Unfortunately, the active coaxial port was not located near enough to our computer desks -- Ben and I both require an ethernet connection directly to the modem, so going wireless wasn't an option. The kids' computer was in their room and so would never be close enough to the router for a hardwired connection, but we planned to use a powerline adapter to bring the internet in for them. In the meantime, I connected using a powerline adapter myself until a tech could come out to switch the active port to one in the same corner as our desks, which is exactly when the trouble would begin.
The internet was working after the tech left, but it was much slower than it had been before. A quick check on Speedtest showed that not only were we only getting somewhere around 16mbps down, we also had AT&T as our internet service provider -- which was extremely strange, considering that our ISP is actually Cox.
A false positive
Commence contact #1 to tech support. The representative confirmed that he could see the modem and was able to issue a reset from his end, meaning it was definitely connected to Cox's network; Cox could see us, but we couldn't see them. He did notice that the IP address I was seeing on my computer didn't match the IP address he had listed for us. After about an hour of working with him I noticed that the ethernet cable for my computer was loose in the modem port, and when pushing it back in, the problem seemed to resolve itself. I very sheepishly apologized and gave him permission to use me as an "idiot customer" story on his lunchbreak and figured everything was fine.
Alas, it was fine for about five minutes. Note that I'm still not sure why this seemed to temporarily fix the issue, but I was never able to reproduce this fix, even when unplugging and plugging it back in.
Enter Ricky and Vicky
It was at this point that we also noticed we were unable to reach the router on our local network via the typical 192.168.0.1 address. We were, however, able to access the address listed as the default gateway in my ipconfig results, which brought us to the control page for an AT&T network owned by "Ricky and Vicky." GeoIP initially showed this as being a network located in Georgia. Frighteningly enough, this page listed plaintext passwords for their network, the MAC addresses and IPs for every device connected to it, and the ability to control those devices. In a previous life, I would have had lots of fun running a MitM attack to monitor keystrokes and maybe score some sweet banking credentials or just cause some childish mayhem. Luckily for Ricky and Vicky, my hat has faded a few shades in the wash.
The theory, developed with lots of help from my telecom friends on Twitter, was now that the equipment used to belong to Ricky and Vicky had been wiped improperly, possibly because an AT&T tech had overridden the base configuration of the modem to pull down the MD5 from the same TFTP every time -- they're not necessarily supposed to do it, but it's still a fairly common practice. As to how the modem then ended up in Cox's leasing pool was anyone's guess, but probably they switched ISPs at some point and it got taken to the wrong place.
Ruling out devices
Contact #2 involved going to the Cox store, giving a very high-level overview of the problem, and retrieving a brand-new, never-before-leased modem. If the problem was indeed an improperly wiped device, this would be an immediate fix.
No such luck. Upon setting up the new modem and plugging everything in, Ricky and Vicky were once again flashing up on the screen. It was statistically possible that we'd been given two bad modems from the same location back to back, but very, very improbable.
The issue was confirmed to be showing up on both computers on the hardwired connection, ruling out any strange DNS or proxy settings on one machine or the other. Virus scans came up clean. The final evidence that the computers themselves were not the culprit would come during the next tech's visit, when upon plugging his laptop directly into the modem, he saw the same exact behavior.
Plugged into the wall below Ben's desk, a small white box sat quietly, innocently, going largely unnoticed during the troubleshooting process.
Renew and release: an exercise in futility
Contact #3 was another hour or two on the phone to a customer support representative who was just as stumped as everyone else was. He insisted there couldn't possibly be anything wrong because he could see our modem and issue remote commands to it; we informed him as patiently as we could that we were aware of this, but that we were still getting AT&T's service through a Cox router.
This is where I'd like to make a controversial statement: ipconfig /release and ipconfig /renew have never once solved anyone's problem. Ever. I have been using, fixing, and building computers for more than three quarters of my life and renew/release has never once been the solution.
Although this representative was surprisingly hesitant to send out a tech, I finally put on my sternest Mom voice, explained that I was four days into not being able to work reliably -- the reason that there was no blog post last week, in fact -- and that the 20 minutes of dead air punctuated with "ummmmm" and "I don't know" indicated to me that he had done all he could to help and that the next step was sending a tech out to look. He finally agreed.
A networking nightmare
My next theory was that the issue had something to do with the cabling. I was confident it wasn't an issue with the main line, since multiple representatives had confirmed there were no outages in the area and no similar reports had come in from anywhere, let alone our apartment complex. This was actually good news, as an issue with the main line takes a lot more time and manpower to fix. It seemed like two cables were resting against each other somewhere in such a way that they were causing AT&T to override Cox with regards to the information being sent to the modem. Or, more likely, the first tech had grabbed the wrong cable from the box when switching the active port.
When the second tech showed up, he helped me rule out the "wrong cable" theory by explaining that AT&T isn't actually cable internet, it's DSL, and thus doesn't use the coaxial port like Cox does. This actually made the entire situation even stranger, as if the modem wasn't receiving information from Cox, it should have just shut off completely. We also noticed that somehow, even when the modem was restarting, I still maintained a connection to Ricky and Vicky's network, which was now showing up in the list of available wireless networks -- GeoIP was incorrect when it told us Georgia. It now seems that they're either our upstairs or downstairs neighbors.
The issue persisted even when running the cable line straight from the external box into the modem. After two hours, the tech brought another brand-new modem up from his truck as a last ditch effort to solve the problem. To our shock, it worked. I was back on Cox and getting my blazing fast speeds. My connection persisted through a computer restart and power-cycling the modem.
Must have been that statistical improbability of two bad modems, we thought. As he was packing up his tools, I noticed that my computer was the only one hooked into the modem. Ben's computer and the powerline adapter for the boys' room were unplugged.
The powerline adapter.
Not wanting to keep the tech out any longer than the poor guy had already been held captive, I saw him out, then came back to the computer area and plugged the powerline adapter in.
I unplugged it again.
Cox. All without having to even restart the modem or my computer.
Telecom Twitter to the rescue again as I explained the ultimately nonsensical situation -- although I don't have a way to crack open the wall to verify, it's most likely that we have poorly shielded copper wires in the wall where our computers and the newly-activated port reside. Cross-talk is occurring between the electrical and phone lines, and then when the adapter is plugged straight into the modem, it's allowing it to use the modem for the DSL line instead of the cable while still maintaining a connection to the Cox network.
As to why the wires are not shielded correctly, it's a front wall and our computers are in what would usually be a dining nook. I'm sure it wasn't planned that people would be using this area of the apartment for heavy-duty telecommunications. There's also a very good possibility that the wires were initially shielded just fine, but ants -- of which we have many due to all of the trees and plants around -- have chewed through the physical shielding. The destructive power of ants is often underestimated, but we had a main cable line in one of my previous residences actually go completely down because of an ant infestation that had taken hold.
It's something that is incredibly unusual, but I guess we're just that lucky. At least the solution didn't require ripping out drywall or anything else messy, expensive, and difficult; we can just go with a wireless card for the boys' computer. Even still, I wanted to make sure to document the process for anyone in IT or customer service who may have a customer with the same issue, as everyone I spoke to along the way stated that this was the first time they'd ever heard of such a thing, even those who had been working in the industry for 10+ years.
I've also done some research regarding the particular modem we have, manufactured by Arris, and found that there are multiple complaints of incorrect IP addresses being assigned by them (although not as far as incorrect ISPs). I'm frankly a bit surprised that Arris wouldn't have included something in their firmware to prevent this kind of override. I'm also shocked at AT&T for the ease with which I was able to access my neighbor's sensitive information -- there was no initial login page, just an automatic connection to their network controls.
At any rate, it looks like this is the end of our adventures with Ricky and Vicky, until such a time that I muster up the courage to knock on their door and give them a quick lesson in network security.