Some Things Shouldn't Have To Be Hard

Part of my job involves supporting the network for a business unit with government contracts.  We have a connection to a private government extranet, over which our users connect to several websites required to fulfill the contract work.

Monday morning the senior director of IT operations for this business unit called me to say that his users couldn't log into one of these sites.  He had already been in touch with tech support for the site, and they had confirmed that it was up and running, and suggested we had a problem on our end.

I started my troubleshooting by logging into the perimeter router connecting to the private extranet, and saw that the connection was up.  Next I logged into a perimeter firewall and checked that there was live traffic passing in both directions - everything looked healthy there as well.

Finally I logged into a PC on the affected network and tried connecting to the external site myself using a web browser.  I was unable to connect.  Browsers these days do a pretty poor job of indicating what the problem is if a site can't be reached.  I was using IE 11, and it gave me a list of possible causes that covered just about every possible issue.

I decided to look up the IP address of the remote site so that I could trace the path through the network and double check firewall rules.  Using nslookup at the command prompt, I got a good indication of the problem right away - I was unable to resolve the IP address of the site.  My computer was configured to point to our internal DNS servers, which in turn forward certain domains to DNS servers located across the private extranet.  

Since I was unable to resolve the IP address, I suggested that we needed to get the on-call DNS administrator to check things out.  In the meantime we also started a conference call with the tech support people for the remote network.  While waiting for our own DNS administrator to join, I described the issue I was seeing.

The remote technician asked me, "Well, what did you change?"  I told him we hadn't made any changes.  He asked, "Did you do anything to your network connection?"  No, we hadn't.  "Did you make any firewall changes over the weekend?" was the next question.  No, we didn't.  I reiterated to the remote tech that our connection was up, everything seemed to be working, but we just couldn't get DNS resolution.

After a short while our local DNS admin joined the call.  In short order he confirmed that the DNS servers were working properly, no changes were made on our end, and we seemed to be getting "denied" messages back from the remote DNS server.  The remote tech repeated just about every possible iteration of the question about what WE had done to break things.

Only after more than an hour of this line of questioning did the remote technician finally reveal that the remote DNS servers had been changed over the weekend - completely replaced with entirely new devices.  It took a little longer, but it was eventually discovered that the new devices had a built-in ACL which was blocking our requests.  The old servers hadn't had this capability, and the ACL which the remote DNS admins had put in place didn't allow our servers to talk to theirs.

So riddle me this, Batman - you know you changed out your DNS servers, but when I call and tell you my DNS queries are being refused, you spend an hour making me repeatedly assert that I didn't change anything?  I lost two hours of my time, and more importantly my business lost two hours of productive work for dozens of users trying to fulfill their quota of work on a government contract because some bozo didn't want to admit that his change broke the system?  Priceless.

Attack Of The Clones (Microsoft Takes On Chromebooks)

Tech news and business websites are posting lots of stories these days suggesting that Microsoft is "targeting" Chromebooks with a new line of cheap laptops.  The HP Stream, priced at about $200, was the first major entry in the category, and now we're seeing reports that Microsoft is partnering with hardware vendors to create laptops well below that price.  The term "Chromebook killer" is cropping up all over.  Microsoft seems to be particularly worried about the adoption of Chromebooks in school systems, where the combination of utility and ease of administration makes them an obvious choice.  A new generation of computer users is getting regular exposure to Chromebooks, learning that they are capable devices through daily use, and those young people will grow up to be consumers one day.  It's easy to see why this is a real concern.

Here's why I think it's a losing game for Microsoft:

1. Chromebooks aren't about "cheap": It's true that many Chromebooks are inexpensive.  But their low price isn't necessarily the primary goal - rather, it's the logical result of the architecture.  The point of a Chromebook is to leverage cloud server-based computing and storage, more or less turning the laptop into a terminal device.  Chromebooks have cheap, low-power processors simply because they can.  And not all Chromebooks are "cheap" - my Toshiba Chromebook 2 with 1080p IPS display costs over $300, due largely to the spectacular display and better-than-average sound hardware.  In building cheap Windows laptops, Microsoft is missing the point - lots, maybe even most of the people buying Chromebooks aren't buying them because they're too poor to afford anything else.  They're buying Chromebooks because they like what they offer.
2. Chromebook buyers see the lack of Windows as an advantage: Windows is bloated.  It requires a lot of resources to run well.  It's popularity, and aspects of it's design, make it a favorite target for hackers.  It has to be patched constantly, and the patching process is cumbersome and fraught with problems.  It can't be run safely without add-on applications to guard against viruses and other malware.  It gets fouled up with use over time.  And in environments where IT staff have to administer large-scale deployments, it takes a lot of work to manage. Chromebooks aren't like that.  They are relatively stable.  The OS is replaced on a regular basis rather than patched, and the process is very smooth and quick (simply requiring a reboot that takes, on average, about 10 seconds).  Chromebooks are relatively safe from malware, and offer a simple way to remove all customizations and return to factory configurations (called the powerwash).
3. Windows needs resources: Microsoft likes to point out all the things you can do with Windows, but not with Chromebooks (warning - the link above contains several factual errors).  And it's true - you can't install full versions of Microsoft Office on a Chromebook.  You can't install Photoshop on a Chromebook.  You can't install commercial, power-hungry games on a Chromebook.   What these comparisons fail to point out is that while you can install these things on a cheap, sub-$200 Windows laptop, running them with acceptable performance is another matter entirely.  The last time Microsoft decided to try this, the market was flooded with cheap "netbooks" which many users found frustrating to use.  In some ways, these new, cheap Windows laptops will be even less capable than those netbooks - Microsoft is pushing the use of super-low-power CPUs like the Bay Trail Celerons (which do fine on tablets and Chromebooks, but which are not predicted to run intensive applications like Photoshop all that well).  The new laptops are coming with internal storage reminiscent of a Chromebook - like the 32gig SSDs in the HP Stream - and offering free cloud storage for 1 year, similar to Chromebooks - but that storage is paltry for the installation of applications like Photoshop and Microsoft Office.  They come with 2GB of RAM, barely enough to get Windows up and running, and in no way adequate to run Photoshop or advanced games.  The best way to summarize the problem is that in order to hit the desired price-point, Microsoft has had to lower the hardware resources to such a low level that the Windows experience is likely to be sub-par.  They may garner some sales in the short-term, but people expecting to run Office, Photoshop, or triple-A game titles will be disappointed.  My guess is this will hurt Microsoft in the long run.

It isn't my conclusion that Chromebooks are going to kill off Windows, and few Chromebook users think so.  However I do believe that Chromebooks fill an important niche, and I just don't see any way that Microsoft can play in that space without sacrificing Windows' strengths.  It will be interesting to see how this plays out.

The Case Of The Silence On The Wire

I have spent a lot of my career as a network engineer in front of packet sniffers.  I've often heard it said that "the wire doesn't lie", and that's true - as far as it goes.  But packet sniffers (and other analysis tools) don't show you the "truth" either, unless you define truth as just a set of data points.  "Truth" as most of us understand it requires deriving meaning from the facts, and sniffers are pretty limited in this aspect.    The following story will (hopefully) illustrate the sometimes difficult process of extracting the truth from the facts, how different people sometimes draw different conclusions from the same facts, and the importance of persistence in the pursuit of the truth.

My company has a big print and mail facility, and years ago some smart person realized that we had enough excess production capacity to offer our services to other companies.  They lined up their first prospective customer and things got underway.  We set up a VPN over the public Internet between the other company's network and our own.  Our Internet connection came into our primary datacenter, and from there we had a private connection to our print and mail facility.

The print server was a Unix system running the standard line printer daemon (LPD), and the client was running the line printer remote (LPR) protocol on a Windows server with Microsoft Print Services for Unix.  Their system would create a print job and connect to our server over the VPN, whereupon the job would be queued and printed.  We went through a short POC phase, and when everything worked to the satisfaction of the print facility and the customer, contracts were signed and work got underway.

Not long afterwards, my manager was contacted by the print facility folks and asked to look into a problem - the customer was reporting occasional problems connecting to the print server.  We didn't have a lot of VPNs at that time, and the combined mistrust of the Internet and VPNs had led to a suspicion that the VPN was the culprit.  We therefore combed through our VPN logs for evidence of problems, as did an engineer at the customer's network.  Neither we, nor the customer's engineer (who I will refer to hereafter as "Steve") found any evidence implicating the VPN.  However, the problems continued.

We had recently gotten new packet capture devices with large amounts of storage, designed for more or less permanent installation in potentially high-value locations on the network.  We deployed one of these to be able to watch all traffic on both sides of our VPN concentrator - one side would see the encrypted stream, while the other would be able to see the unencrypted stream.  We deliberately chose this spot on the network because it was the "furthest out" on the perimeter network - if we didn't see problems here, we could start looking further into the network, while on the other hand if we did, we could safely ignore our internal network.

We didn't have long to wait until the problem resurfaced, and the packet traces were instructive.  We saw the communications from the client, and every one of these communications looked perfectly clean.  Every connection included a complete TCP three-way handshake, what appeared to be normal communications between the LPR and LPD, and a standard four-way teardown of the session.  Not once during the entire time troubleshooting this application did we ever see anything like a failed connection - TCP handshakes were always complete, the client and server communications were always successful, there were never any sessions that died midstream, and the teardowns were always textbook clean.  It's been a long time but if memory serves, I am nearly certain we never even saw a single TCP retransmission.

What we DID see were inexplicable absences of connections - periods of time usually lasting several minutes in which there were no packets of any kind coming from the client's systems.  During these periods there were no TCP attempts at all.  The trace taken outside the VPN box was similarly devoid of traffic.  In effect, the wire was silent.

There was a chance - a very slim one - that something was going on in what little of our network existed outside the VPN box, so we looked for evidence of that.  The VPN box plugged into an ethernet switch, as did our Internet routers.  The switch was clean, as were the routers.  We were not experiencing any interruption of other Internet traffic, our other VPNs were all running clean, and the VPN to this partner was not having problems.

Given that we saw NO issues of any kind, I concluded that the issue was occurring at the customer's end.  I reasoned that intermittent issues on the Internet, or within our infrastructure, would not "respect" the boundaries of TCP sessions - in other words, I would have expected to see problems occur within the TCP sessions.  I might expect to see failed TCP handshakes, or some irregularity within the print jobs streaming over.  The fact that this never happened led me to believe that something was preventing our customer's systems from even attempting to connect for short periods of time.  I should also mention that our print server was handling lots of internal jobs with no issues.

I packaged up my sniffer traces and forwarded them to Steve, outlining my conclusions and the reasoning behind them, and asking him if he could take local traces of his own and confirm whether his systems were making any attempt to connect.  He promised to do so.  I didn't hear from him for a while, and we continued to get reports through the print facility that the customer was complaining about the connection problems, so I reached out to Steve again.

I asked directly whether Steve had taken traces.  He said he had.  I asked if he had seen any irregularities in the traces on his side.  He said he had not.  This might have been an error on my part - maybe I should have asked whether he saw anything at all when the dropouts were occurring, the same "silence on the wire" evident in my traces, but I didn't think of it.  I did ask if I could get copies of his traces to compare with mine, and Steve said he would share them, but they were never forthcoming - so to this day I do not know if he actually took any.

I reported my research to my management, along with my conclusion that the problem must be at the client's end, and that there was not likely anything we could do about it.  Their response was to urge me to keep looking, and so I did.  I took a dozen or more traces, all containing perfect, complete sessions, and usually also containing some of these weird silent periods.

I dug into the traces and started looking at everything that was there in the communications.  Let's see - client SYN packet, source port 721, server port 515, server SYN-ACK, client ACK, some kind of "hello" packet from the client and a response from the server, print job streaming over, client FIN-ACK, server ACK, server FIN-ACK, client ACK.  All perfect.  Next job - identical except for client source port which is now 722 (and the sequence numbers, of course).  And another - client side port 723, etc.  It did occur to me that the client-side port numbers were almost TOO sequential - there were never any skips, say from 721 to 725 - which made me think the client's system must not been too busy, or that it might be passing through some device that was altering the client ports on the way our of their network.  I also thought most clients should pick their ephemeral ports from a range 1024 and above, but I wasn't too bothered by it.

I looked at this for a while - several times, on various days, over several weeks.  I couldn't find anything that I thought would help, and I slowly lost interest, especially as the rate of complaints dwindled.  After a while I just didn't think about it any more.  It wasn't exactly a matter of "giving up" - I thought I'd done a good job of isolating the issue to the customer's end, and lacking visibility into their network I just didn't see that I could do more to help.

For several months it dropped completely off my radar.  I guess I should have known it would come back though, because unresolved issues never really go away.  One day my manager and director both asked me to get back on the case.  They had been contacted by the manager over the print and mail facility.  The issue was still going on, the customer was complaining more than ever, and now the print facility was having to reboot the server on a regular basis to clear up the issue.  To make matters more urgent, they were hoping to add another customer, but until this issue was resolved they were unable to take that next step.

I started by calling our print server operator - I wanted to know why we were rebooting the server.  He told me that after countless "outages", the customer had requested that we try rebooting the server.  This had been done, and the customer was then able to connect and send print jobs.  Ever since then the customer had gotten into the habit of calling and requesting a reboot whenever they had trouble connecting.

I have to tell you that this made me really angry.  The lack of logic involved here was staggering.  I had clear, indisputable evidence that when the customer was "having trouble connecting", we weren't getting anything from them at all.  I had evidence that every time we did receive a connection request, we answered appropriately - our server always responded.  And in addition, our print server never failed to pick up and handle internal print jobs, which, by the way, were now being interrupted by these frequent reboots.  The whole thing made no sense.

I then talked to Steve.  He told me that since the problem had persisted, they (the customer's company) had started using a command-line utility to check the print server.  They would fire up a command prompt on the system creating the print jobs and run a command that would connect to the print server and display the jobs.  When "the problem" was occurring, the command-line utility would also be unable to connect.  Their operators would sit there rerunning the utility every couple of minutes until they got a successful connection, and then try to restart the print jobs.  Usually after this, they could print again, but sometimes not.

What I got from all of this was that the issue had some sort of time component to it, resolving itself within a few minutes.  I was still convinced that the issue was on the customer's end - there was never any evidence otherwise.  The server reboots simply gave time for the problem to correct itself, but that had always happened anyway.  To test this theory, I advised the print server operator not to reboot the server any more.  I suggested that he didn't have to tell the customer he wasn't rebooting the server - he could just say "OK, try again in a few minutes."  He began doing this, and sure enough there was no difference in the behavior of the whole system.  After a wait of a few minutes, their print jobs would start coming through.

Now that I had gotten the "reboot monkey" off our backs, I went back to the traces.  I took a bunch of new ones and started going through the connections again and again, looking for anything out of the ordinary.  They looked just like the ones from before - client sends SYN packet with client port 721 (or something similar), server side port 515, server sends SYN-ACK, etc.  The connections were as perfect as ever.  In fact, they looked so familiar that I began to wonder if I was looking at my old trace files.  Nope, these were new.  I pulled up some of the old original files to make sure, and it was at this point that I began to grasp the faint outlines of the problem.

The client side port numbers had always bothered me a little bit.  Aside from the fact that they were all under 1024, the range of port numbers was always very consistent - and very small.  The client side port numbers were always within the range 721 - 731.  Eleven port numbers, always in succession, reused again and again.  I would see a connection from port 728, 729, 730, 731, then it would loop back around to 721.  And every so often, usually after a bunch of successful connections, silence on the wire.

I began to wonder if there was some issue with port-exhaustion - this thing was using such a small pool of client-side ports.  I wondered, how quickly is a client-side port allowed to be reused?  I dug out my trusty copy of Richard W. Stevens' TCP/IP Illustrated Volume 1 and found the TCP state diagram.  I saw something called the "2MSL" wait state which occurs before the socket is fully closed.  The MSL is the "maximum segment lifetime" which is supposed to be two minutes.  The standards for the protocols we still use today were created back when computers and the Internet were MUCH slower, and back then there might be conditions on the network that could cause a packet to arrive late - very late indeed.  Anyway, the standards also said that the partner in a TCP session which initiates an active close (through the use of a FIN-ACK packet) MUST then hold the connection for two times the maximum segment lifetime (2MSL) before it can consider the socket closed.

In other words, the connection doesn't truly close for four minutes after all the teardown messages have been exchanged.  If you do a "netstat" command on a system you will often see sessions in something called the "TIME_WAIT" state.  These are sessions waiting out the 2MSL period so the system can close them.  Basically, from the time I observed the client FIN-ACK and other teardown packets, four minutes would have to elapse before the client-side port number would again be released to the operating system for reuse.

With respect to our customer's printing problem, the issue was now in pretty sharp focus.  For some reason, the client was only using 11 port numbers (721-731).  After 11 successive print jobs, if the timespan of those jobs was less than 4 minutes, all of the TCP sessions involved would be in the TIME_WAIT state.  Until the oldest sessions completed the 2MSL wait, there would not be any available ports for new sessions.  But why were they using such a small pool?

The answer to this is in the RFC which defines the LPD/LPR services.  RFC 1179 says that "The source port must be in the range 721 to 731 inclusive."  To be honest, I didn't actually find the answer in the RFC - but some Googling led me to a Microsoft Knowledge Base document which described exactly the problem we were seeing, which mainly occurred on a specific version of Windows Server, and with a suggested fix - a registry setting to cause the Print Services for Unix to use standard ephemeral ports from the much larger pool above 1023.  The document outlined that the command-line utility they were using also drew client ports from the same range, so effectively, if all the ports were tied up in TIME_WAIT sessions, the utility would similarly fail to connect.  In fact, when they DID connect with the utility, they were actually putting an available port into a 2MSL wait again!

I sent an email to Steve, asking what version Windows Server was in use.  He confirmed the affected version.  I then sent an email telling him what I thought was happening - that his server was using a very limited range of client side ports, that the speed/volume of print jobs was outpacing the system's ability to clear the sessions for reuse resulting in client port exhaustion, and that there was a suggested fix involving a registry setting.  I even sent a link to the Knowledge Base document.

Steve responded that he would look into it, but my interpretation of his response was that he wasn't sure he believed me.  I couldn't do much about that - whenever you are dealing with another company, and that company is your customer, and when you are telling a peer engineer that you have remotely diagnosed a problem in his systems...well, there may be some resistance to the idea.  So I waited to see what would happen.

What happened was exactly zilch - the problem persisted, and again we were being begged by our print facility manager to intervene.  But this time, there really was nothing more we could do - except that I now understood I would have to force the issue with Steve.  I wrote - to my management, and copied to Steve - an exhaustive (and at times pointed) accounting of the entire troubleshooting effort, including my early work and their (accurate) conclusions about the problem being in the customer's network, my difficulties getting information out of Steve, my work in understanding and putting a stop to the reboots, and finally my conclusion - backed up by sniffer traces and documentation from the customer's server OS vendor - that the problem was caused by port exhaustion.  I included the Knowledge Base document for reference, and stated that my department was finished working the issue, once and for all.

Within a week of sending that email, Steve (or someone else at his company) had made the registry changes to their system, and the problem never surfaced again.  It had taken six months, dozens of hours looking at traces, emails back and forth with an incompetent or unhelpful peer, a lot of pain and suffering on the part of our print facility, and research into another company's network and systems, but the problem was finally resolved.

Whenever I work a problem - especially when it's such a challenging and painful one - I always look for "lessons learned."  This one was particularly fruitful:

1. Persistence, persistence, persistence - over and over throughout my career troubleshooting, I have run into problems where it seems like I just stare at packet traces or logs until my head is about to burst, and then, like a ray of sunshine coming through the storm clouds - the solution appears.  This case was somewhat rare in that there was a period where I accepted that the problem was "solved" even when it wasn't.  I had felt that in proving the problem was on the customer's end, that my work was done.  Figuring out when to stop, when enough is enough, is part of maturing as a troubleshooter and I may do a blog post about that later - but in this case, my real customer was always the people at my company's print and mail facility, and until things were completely resolved, my work was not truly done.
2. Troubleshooting a problem that exists on a foreign network is really hard, but NOT always impossible - this probably doesn't require much more explanation than what is available in the story above, but I've often seen network engineers focus on this sort of "us VS. them" strategy in problem solving.  The idea is that if we can prove it isn't US, then it must be THEM, and we can't do anything more.   A lot of times there is a bit of shaky logic employed, something like "Well, our printer works fine for everybody else, and we don't have any other VPN or Internet problems, so it's not us."  Of course, Steve always insisted that they were not having problems sending print jobs to anywhere but us, which is possible if we were the only LPD server they were targeting.  All of this may have been true - and in my case I had even better evidence from my traces that we weren't even receiving communications from the customer's network - but the fact remains that the answer was always right in front of me, in the packet traces I was so proud of analyzing. 
3. Dealing with "peers" on other teams or at other companies can be just as challenging as the technical act of troubleshooting - "Steve" is a prime example of something I've dealt with many times over the years.  He was either unwilling to really look at his network and systems, or incompetent, or both.  I believe that if he had actually performed network sniffer traces he would have noticed that there was NO communication coming from his print system during the outages, which would have led him to the same conclusion I had reached.  The fact that this didn't happen, combined with his apparent unwillingness to share the traces he claimed to have taken, leads me to believe he never did them at all.  Of course it's possible he did the traces but just didn't interpret them properly.  I don't suppose I will ever know.  I also strongly suspect that the Windows Server in question would likely have been writing event log messages regarding the connection problems, had anyone over there cared to take a look.  But Steve was what I had to work with - I had to continually reach out to him, probe for information and prod for action, while trying not to upset or insult him, in order to finally get the action required.  It was neither easy nor pleasant.
4. No matter how much you know, there's always room for more - going into this problem I thought I knew the basics of socket communications pretty well.  But it took far too long for me to notice the oddly low port numbers, or the small pool in use.  There's a lot of detail in packets and packet traces, and it takes diligence to spot patterns like these.  

And so ends the tale of the Silence On The Wire - hopefully you made it here to the end, and that it was worth coming along for the ride.

Order-taking vs. Selling (or maybe just a bad metaphor)

In the world of sales, there is a big distinction between being (just) an order-taker and being a salesperson.  The distinction comes down to just going through the motions (order-taker) and going out prospecting and nailing down real sales (salesperson).  I am not entirely sure where I heard it first used in this way, but the term "order-taker" can also be applied to people in other professions, and generally refers to doing a very passive (sometimes passive-agressive) and reactive job, rather than figuring out what your customers need and giving it to them.

My team does networking - architecture, engineering, installation, maintenance, etc.  We also manage all the network firewalls in our company.  Most firewall requests are generated by other people, and come to us in the form of work tickets.  The on-call person from our team has to do these and it's something no one looks forward to doing - which is why it makes a perfect example for demonstrating the order-taking mentality.

In general we tend to think of a firewall rule as a pretty simple set of four things - a source IP address, a destination IP address, a protocol, and a port number - with some obvious exceptions, that set of details makes up the vast majority of the rules we create.  It would seem like simplicity itself for the requester to gather these items for us and properly document the request, and that's what we want - we basically want to get a "form" with everything filled out, so we can just go execute the request.

The sad fact is that it's just not that simple in the real world, and a lot of the requests we get can't be executed "as is".  Here are just a few of the issues:

• The requester doesn't understand ANYTHING about networking and doesn't know what an address, protocol, or port is
• The requester doesn't know how to use the order tool (in our environment it's a sort of web order form for technical services)
• The requester asks for something impossible (some parts of the network can't talk to one another for technical reasons)
• The requester asks for something inappropriate (we have rules about what we let in and out of the network, you know?)
• The request is rejected by the security department (they have to sign off on these) or the change management department

The question of whether a highly paid professional network engineer is reduced to being an order-taker comes down to the manner in which he/she handles these kinds of challenges.  Here are some examples of order-taker behavior I have seen more than once in my career:

• Engineer knows the request can't be worked due to incorrect or impossible combination of items - but instead of contacting the requester proactively so they can correct the issue, just doesn't do the work and allows the change to expire
• Engineer refuses to explain anything to requester who clearly needs a little education (sometimes with a snide comment about how dumb "those people" are)
• Engineer goes ahead and implements an impossible rule (such as a firewall rule that doesn't work because the network routing doesn't ever bring the traffic to the firewall) because hey, they were dumb enough to request it and the security guys were dumb enough to approve it, right?

I absolutely understand on the deepest personal level this kind of behavior.  We're all overworked, the firewall changes are already a pain in the ass, and we'd all rather be doing something fun like building out a new network.  But those who rise to the level of networking professional (I like that better than salesperson) will do the following, or something like it:

• If the requester clearly doesn't know what something is or how something works, the professional will take the time to explain it
• If the requester asks for something that will never work, the professional will be proactive in pointing this out, will find out what the requester was trying to accomplish, and help them refine the request so that it represents what they actually needed
• The professional will contact the security team if something has been approved which should not have been, because even the security guys can make mistakes, and the company's security is too important to play games

I think the above examples illustrate what I'm talking about.  In our work as network engineers, we can be order-takers - waiting for someone to tell us what to do, passive-aggressively carrying out stupid or impossible requests, and refusing to take an interest in our customers....OR, we can try to be more than technicians - we can rise to the level of networking professionals, an integral and indispensable part of making our companies successful.

I know which I'd rather be.

What A Week

Last week I was "on-call" for work.  That meant I was responsible for watching our monitoring systems and problem queue, working problems as they arise if possible, coordinating efforts if it's something I need help resolving.  The first couple of days were pretty slow, a couple of failed power supplies in systems with redundant power, no biggie.

Thursday I got a call that users in our Mechanicsburg office were experiencing a lot of performance degradation.  A quick check of their primary MPLS circuit (from Level3) showed a lot of packet loss.  We have BGP configured to switch them over to Centurylink if Level3 fails, but the circuit hadn't actually dropped, so we forced it - shut BGP to Level3, and opened a problem ticket with them.  

A short time later, our monitoring tools reporting trouble reaching a router in Williamsport - another Level3 circuit, this time the backup circuit, normally only used when connecting to that one router.  We began thinking Level3 was having a bigger issue.  But before we could contact them to add the info to our ticket, we heard users in Harrisburg were having performance issues.  Level3 again, and the primary circuit - so we shut BGP there, forcing them over to a backup circuit from Verizon.  Finally we got the Level3 ticket updated with all the circuit information and waited for their response.  

About 3:00PM a bunch of us were supposed to go out to celebrate a teammate's birthday.  Right when I get to the bar, the phone rings - network admin requested to look at an application issue.  So I went back in and launched into one of those 3-hour marathon sniffer sessions. Fun!  I finally got out about 6:00PM and headed home.

On the way home I got a text message from Bank of America - fraudulent charge suspected on my debit card, please call or login to online banking to check.  Peachy.  As I walked into my house around 7:00PM, my cellphone rang - a guy at work who was going to swap some potentially bad GBICs on a fiber, wanted me to make sure we had traffic off the link.  

I decided to call back from my landline because cell coverage at home is spotty.  I picked up the phone, and...no dial-tone.  Luckily I still had DSL service.  I got logged in, called him from my cell, and got that one worked.

In the meantime I opened a chat session with the phone company's tech support.  They wanted me to swap phones or try the test jack outside the house.  No good - I didn't have a spare phone, and the one I did have was a cordless that requires power for the base station.  I would have to wait until I could get another phone on Friday to find out if it was my problem or the phone company.

Finally I logged into BoA's web site.  Yep, somebody tried to access my account from a Publix supermarket down in Florida.  Of course as soon as I marked the charge fraudulent, BoA promptly canceled my debit card and notified me it would be 5 - 7 days to get a new one.  You just have to love the modern world, right?  I checked my wallet - $5 cash, maybe with that and the change I keep in the jar at work I would be able to eat on Friday.

Friday morning, we had an email from Level3 waiting for us.  They had found a problem with a core router serving a bunch of their customers in the northeast, and routed around it.  After talking it over with my director and teammates, we decided to keep Mechanicsburg and Harrisburg on their backup circuits for the day and watch the Level3 circuits.  If everything held up we would re-enable BGP over Level3 sometime Friday night.

Two hours later the Verizon circuit to Harrisburg died.  Just plain died.  And with BGP shut over the Level3 circuit, they were cut off completely.  We dialed into a modem on an emergency backup router and got BGP going again on Level3 to get them back online.  Total time of that outage was maybe 5 minutes.

Friday afternoon rolls around and I got talked into trying another social outing.  But just when it was time to leave, I got asked to look at another issue - a file transfer running over a point-to-point circuit between Florida and Pennsylvania was running slow.  In fact, it had been running slow all week, but no one had asked for help until Friday afternoon.  AAAUUUGGGHHH!  So another night not getting off until 6:00PM, not getting home until 7:00PM.  And to make it more interesting, it looked like there was packet loss going from us to the remote site - on a Level3 circuit.  Not MPLS, true, but another Level3 circuit in Pennsylvania?  They claimed to have routed around their other issue, but at this point we were getting gun-shy about putting anything else on their network if we didn't have to (Harrisburg notwithstanding).

On the way home I stopped at Target and bought a plain-old telephone that doesn't need external power.  When I got home I plugged it in inside the house - no dial-tone.  I took it out to the box outside - no dial-tone.  Ok, it's the phone company's problem.  I went in to do another online chat session with tech support, but now I had no DSL.

I got on the cell phone to call the phone company and halfway through one of the half-dozen prerecorded messages, the call dropped.  I dialed back, worked my way through the menus - and got dropped listening to the same message.  Now, they say that doing the same thing over and over and expecting a different result is one definition of insanity.  I must be insane, because I tried a third time.  And got dropped during the same message.  Finally I called in and just kept hitting "0" on every menu and eventually got a live person.  Of course, all they could tell me was they didn't see any trouble in my area, couldn't call my house phone (duh) and couldn't see any signal from my computer.   That, and they couldn't send anyone to the house to fix it during the weekend unless I paid, otherwise I would have to wait until Monday for a visit from a tech (I was still on-call for the weekend), and I would have to stay home from work to meet the tech or they wouldn't come (despite the fact that the issue was clearly NOT inside me house).

So today is Monday.  The tech came.  They had moved my circuits last week to a new switch and somehow failed to configure my service.  

The good news is, I'm not on-call again for about 7 weeks.  

Yeesh!

Bloody Turnips

“You can’t squeeze blood from a turnip.”  This old saying is a way of expressing that some things are so obviously impossible that they aren’t worth trying, that they are a waste of time.  But sometimes the problem isn’t that we’re trying to squeeze blood from a turnip - the problem is assuming that we’re looking at a turnip in the first place.

The other day I got “the call.”  “The call” usually comes late in the day, and frequently on a Friday.  It’s when someone has been working at a problem all day, or all week, realizes they are running out of time, and in a last ditch effort at a resolution they ask for a network admin to take a packet trace.  And I’m the person that frequently gets “the call.”

This time it was an application which picks up files from a server, the application was locking up, and the people troubleshooting it explained that this is frequently a sign that there was a delay in picking up the files (this application was said to be super time-sensitive).  Server admins had found nothing wrong on the file server.  I was asked to see if there was anything causing network-based latency, or if I could at least see something in the trace that might account for the issue.

I have to admit that I did not approach this problem with any enthusiasm.  I have a life.  I do not like getting called at 3:00PM to start a multi-hour troubleshooting session on something this vague.  But it’s part of the job, these were my customers, and apparently nobody else was making any headway (including the vendor of the application, who had been called in to work on it).

Now despite being pretty good with the sniffer - and sometimes enjoying the challenge - I know that it can be a hard way to get to the root of a problem, so I made an effort to do things the easier way.  I asked the usual questions - when did the problem start, did something change, could I get a more technically accurate description of the problem, etc.  I looked at the basics - located and checked for errors on the switch ports of the file server and application system and so forth.  And then, reluctantly, I fired up the sniffer and got started.

About an hour into the session, one of my teammates came up to watch, and he asked the obvious question - “Do you really think you’re going to find the problem by looking at the packet contents?”  He was, in essence, asking me if I was trying to squeeze blood from a turnip.  And honestly I did not know how to answer him.

It’s something I’ve thought about often over the years.  I am very interested in troubleshooting - the thought processes that go into it, the practice of it, the techniques that are used.  I think that the act of trying to reverse-engineer an application by staring at the sniffer until it feels like my head is bleeding is a really hard way to do things.  But while I have not come up with a lot of amazing answers to those questions, I have learned one thing:

I can’t solve a problem if I don’t try.

There are a lot of times it feels like I’m squeezing a turnip.  But the truth is I don’t know what I’m squeezing.  It’s like sticking my hand in a bag and grabbing something, and squeezing it, and after a long time I get some blood out of it - in which case I find that it wasn’t a turnip.  And sometimes I get nothing but a turnip guts.

So I just said to him - “I have no idea.”  And I kept on squeezing.

I’d like to conclude this post by telling you about the amazing discovery I made in the packet trace.  Unfortunately that didn’t happen.  What did happen is I was able to determine that when the application freezes up, it isn’t waiting for anything from the file server.  The application was getting a response that looked “complete” (for you packet monkeys, it had the PUSH flag set on the last packet of the response), the application system responded with an immediate ACK, and then sat there for a long time before doing anything else.  Then the application system sent a packet and things started up again.  I saw this happen multiple times during “freezing” episodes.

What does it mean?  Well, it means the problem isn’t a delay in getting information from the file server.  There could be a problem in the contents of the response, and being unfamiliar with the application itself I couldn’t speak to that.  Or there could be something happening on the application system causing it to freeze that has nothing to do with the network traffic.

This information didn’t solve the problem for the application folks.  It did get the file server admins off the hook, and it pretty well proved the network infrastructure wasn’t at issue, and it gave the application admins and their vendor a little push in the direction of looking at their own system a little harder.  I hope it helped.

If there is a message here, it’s this - troubleshooting can be a painful, frustrating, and sometimes ultimately unrewarding process.  Problems can be really complicated, the tools can be hard to use, and the whole thing can just be a lot of work.  Even when you try your best you don’t always come up with a big win.  But if you don’t try, you don’t stand a chance.  I think a lot of people - including a lot of network people - think that problems can't be solved with a sniffer, or maybe that they can't solve them, so they don't try.  All I can say is, I've done it often enough to know it's not impossible.  Working a problem with a sniffer isn't always fruitless.  So the moral of the story?

Keep squeezing.