I have a story to tell. It was years in the making. Hopefully it won't take that long to tell. I can't promise you'll enjoy it, but it has a (sort of) happy ending and there are some lessons in it, if you make it to the end.
Did I ever mention I helped start the first Internet service in Camden County, GA? Back in the day there was no local dialup service, so me and some friends at the computer store where I worked got a wild hair to bring Internet to our area, and we did it (with some funding from a partner and a lot of help from a provider in a non-competing area). After two years we had 2000-plus paying customers! We also had a pant-load of debt, and we were really lucky to get bought out before we got into real trouble. I walked away from that experience with no debt, and a crash course in how the Internet works which served to get me my present job as a network engineer with a big company.
Our local phone company is TDS Telecom. TDS mainly serves Wisconsin, but they have gotten into a number of other markets, and they had bought the local mom & pop phone company here in the early 1980's. A few years after I had moved on to my new career, they began offering DSL service. I live a few miles outside of town in a sparsely populated area, so I knew it would take a while to reach my neighborhood, but eventually I got 1.5 mbps service - yay! It was incredible! After a year or two they bumped the speed to 5 mbps. Life was great. Then they began offering 10 mbps and I was in heaven.
Around that time the first set top streaming boxes were starting to hit the market. I was an early adopter of the Boxee Box - a little box design by D-Link, running Boxee software (which was a fork of XMBC). There was a decent amount of content and it was fun to play around with. One of the apps for the Boxee was MLB.TV, which I was really excited about. I signed up and started watching baseball on it.
But it wasn't long before I started noticing problems. The games would stutter and buffer a lot. The MLB app on the Boxee supported automatic quality adjustments, and the picture would get really bad, and the constant buffering made it really hard to enjoy a game. Oddly enough it seemed the service was fine on the weekends, especially in the mornings, but in the evenings on weekdays it was terrible.
I did some speed tests and found that my evening speed was pretty bad - almost always below 5 mbps, sometimes dropping down to 1 or 2 mbps. The MLB.TV FAQ said I needed a pretty solid 4 mbps to watch a game, so that seemed to be my issue. I called TDS tech support and told them I was having some speed issues. They checked my line, "reset my port", pronounced it all clean...but I kept having issues.
Over a period of some six months, I dorked around with everything I could. I got a new router. I ran a clean phone line from the box outside the house straight to my modem, no splices. I ran wired ethernet to the living room for the Boxee. And when 15 mbps service became available I signed up for that. Oh yeah, I also called tech support a half dozen more times - checked my line, reset my port, etc. But no matter what I tried it just didn't improve.
Finally I called tech support and insisted they escalate to the next level of support. After a long wait, a nice lady came on the line to explain that my problem was oversubscription. Basically TDS had sold more service than the equipment serving my area could support. She said they had to order a new switch, and that currently it was scheduled to take about 3 months to fix. She assured me that there was a running ticket on this problem and that I should just wait it out.
So I did. I waited 3 months, and then I waited an extra month for good measure. And when it still hadn't gotten better, I called back. At first, the tech tried to tell me TDS would never give out information about their installation schedules, but when I insisted on talking to a supervisor, I got an answer - it would be another 3 months, there had been a backlog of some kind, the switches were all custom built, whatever.
I waited 4 months again, and it still wasn't any better so I called back. And the answer was, again, about 3 months. Given that TDS had no competition for high speed Internet in the area, I had no real choice but to wait it out. Another 4 months went by, and still things didn't improve. It was now 12 months since I'd been told about the oversubscription, more like 18 since I'd been calling in about the problem, and it looked like it would never get better.
Around this time I was talking to a friend at work who suggested I contact the Public Service Commission. I decided that before doing so, I would call TDS one more time, but I would tell them that I was GOING to call the PSC. And this time, I got a little different response. A supervisor offered to give me the name and phone number of a guy I'll just call "Chuck" (not his real name), who was in charge of scheduling all the big installation work for our area. "Chuck" would be able to at least tell me the real scoop on the schedule.
The first time I got to talk to Chuck, I was sure this was a quality guy who would get things straightened out. He told me the current switch was only capable of handling a couple of DS3s (45 mbps connections) and couldn't be upgraded. However, they were building an entirely new switch and when it was in, they would have a couple of gigabit uplinks - something like 20 times the available bandwidth. He couldn't give me a date but thought it would be ... about 3 months. Oh well, I thought, at least now I know what's going on.
It turned out to be more like 6 months, but the new switch got installed, and one day a technician showed up to let me know they were moving my connection to the new switch. Once it was done, I checked the speed...and I was now getting a whopping 5 mbps. What happened to my 15 mbps service? Well, the new switch was a mile further away, and the way DSL works is very distance-dependent. The good news was that the speed no longer dropped out in the evenings, but I was barely getting the kind of speed you need for good streaming.
And that was a real problem, because by then I had chosen to become a "cord-cutter." I had canceled my DiSH Network service - no sense paying $100 a month for a bunch of channels I didn't want to watch - and I had gotten a Roku box to replace the Boxee, and well, 5 mbps was just barely getting the job done. That mattered because as a network guy, I wanted to use my Internet service for more than one thing at a time, not to mention I was now in a long-distance relationship and had become very dependent on Skype.
Chuck told me he was very sorry, this was not the solution he had envisioned. He told me there was still a chance he could get a new switch put in to replace the one closer to my house, and that he would try, but it was going to take a while and this time he was not willing to give me a projected date. I thanked him for everything he had done, and gave up...for a while.
About 6 months along, I decided to check in with Chuck and see if there had been any news on the new switch. And I was astonished to hear that he had actually already gotten it done! However, he cautioned that it would be a while before they could cut anyone over to it. There was some story about new software at TDS for managing customer orders and it was going to be, you guessed it, about 3 months before he could move me to the faster connection. By now I understood that "about 3 months" was TDS code for "no freaking clue", but it looked like there was light at the end of the tunnel.
I waited 5 months this time before bothering Chuck again. I was really getting tired of this. My struggle to simply be able to order, pay for, and enjoy the service TDS was advertising had been going on for YEARS. I wrote my frustrations into a note for Chuck, doing my best not to burn bridges but letting him know I felt I was getting jerked around. I don't know if that made any difference, but a few days later, something happened.
I came home from work to find my dial-tone cut off. I actually wrote another blog post about this so I won't detail it all here, but the next day my DSL went offline as well, and it was a couple of days before it got sorted out. They had cut me over to the new switch but hadn't bothered to configure service on the line.
With this fixed, I now had pretty good speed - somewhere around 12 mbps, which is decent for DSL. I was happy for a few months, until the dropouts started. This was a new problem. I could surf, or watch TV, for minutes or maybe even an hour or two at a time, and then...dead air. The DSL lite on my modem was never going out, but I couldn't get any data in or out. Resetting the modem would immediately fix the problem, or sometimes it would resolve itself without me doing anything, but it wouldn't stay on for good.
I was quite certain I wasn't having a problem with local equipment in the house, and when I called TDS tech support, I tried to convince them that they didn't need to send a technician. However something at TDS had changed - they now had their phones being answered by an off-shore outfit, maybe in Jamaica judging from the accents. These techs simply wouldn't even consider looking at anything until someone was dispatched to the house. I got an agreement from tech support that their field technician would only look at the line outside the house, and would not enter the premises, and I let them schedule the visit.
When I got home from work the next day, the field tech was there. My girlfriend was there talking to the guy out in front of the house, and he informed me he had gone into the house and replaced my modem! It turned out my girlfriend thought I expected this and would be OK with it. The tech said my old modem was no longer supported, he had failed to find a problem hooking up to the line outside the house, and so assumed the problem must be inside. I tried to explain that in order to see the problem he would have had to stay connected outside for up to an hour, but no dice - and he was not giving my old modem back. I'm not a physical kind of guy but I had a fleeting thought of wrestling the old modem from his grasp.
He had replaced my old "modem" with a new wireless plus 4-port ethernet router. He had pulled my old router out - because he didn't have any idea how to set all my stuff back up - and left me with nothing working at all except his new router. I was furious. I spent the next couple of hours disabling the wireless on the new router, disabling DHCP, putting it in bridge mode, and getting my own router back online. And within an hour, the connection was dropping again.
Just to make sure it wasn't my equipment in the house, I removed my router, re-enabled wireless and DHCP on the router they had provided, and basically made everything just like a standard customer. I made another call to the Caribbean tech support people, insisted they escalate to the next tier, and got a guy who could actually troubleshoot. He quickly diagnosed a DHCP problem causing their system to give me addresses that had already been given to other customers, and he got it fixed. This should have happened on the first call, and my equipment should never have been touched.
Still, I left the phone company's router (an Actiontec V1000H) in place as my primary router for the time being. My girlfriend was visiting for an extended stay, and every time I tore everything out and reconfigured things it would keep her from being able to watch TV or use her computer on the Internet, so I decided to wait until her visit was over. It turned out to be a good thing - because within a few weeks, the connection was dropping out again, and I was able to point out to TDS that none of my own equipment was at fault. These dropouts were different - with the DHCP problem, the dropouts lasted for quite a while, but with these new ones, the connection would come back quickly. They would sometimes come in waves of several dropouts in a row within a few minutes, then things would settle down.
Looking at the router's status page I could see that we were getting numerous retrains, a symptom that the combination of errors and weak signal-to-noise ratios were causing the router to renegotiate the speed. When rebooted, the router would sync up at about 12-13 mbps, but after a set of these retrains it would drop down to about 6 mbps. Another call to tech support, another site visit - I stayed home for this one to make sure they didn't enter the house or touch my equipment. The field tech replaced the box on the outside of the house, checked some things at a junction box at the street, made a couple of trips down the road to the switch...but when he was done, he said he didn't think it was any better and wanted to get a more experienced technician to come out the following day.
I couldn't stay home again, so I left my girlfriend in charge and went to work. It's too bad, really - I have no idea what this guy did, but whatever it was worked wonders. Ever since he left, I have had a solid 15 mbps connection (I NEVER got 15 before) and it never drops out.
I had one final issue with all these changes. I recently bought a new printer, and my desktop computer uses mDNS protocol to discover printers on the network, and it wasn't working. It turns out the Actiontec doesn't pass mDNS (or at least I couldn't find an option anywhere in the configuration to allow it). Once again I disabled all features of the Actiontec router (turned off WiFi, turned off DHCP) and put it in bridge mode, and I hooked up my trusty Asus router. The printer problem was solved.
As I have thought through this entire affair in the process of writing it out, I have tried very hard to come up with some lessons learned. The one clear thing in all of this is that when things aren't working, you have to be persistent. It may not be possible to force a provider to do their job and make things right, but you can bug the crap out of them until they do. My biggest regret was allowing them to put me off for months at a time with their repeated promises that they were fixing things. I really should have called the Public Service Commission, and early in the process. If I had had the option of switching to another Internet provider I'd have done so years ago, but I didn't have that choice. It's taken years to get things right, and it's difficult to say I'm satisfied given the painful process. But I have what I'm paying for, and I suppose that's better than where I was.
Now...TDS is starting to run fiber to the area, and will soon offer 50 mbps service. I wonder if I will have the guts to sign up for it?
No comments:
Post a Comment