By Scott Carson, Network Engineer
Tell me if you have heard this one before…
You have been working diligently on a project. The time has come for the final deployment. The grand finale if you will. Everything is in place. You power it on. It works flawlessly! All your validation steps check out and everyone goes home happy. Another successful deployment in the books.
Without getting into the actual statistics, it feels like this describes at least 80-90% of IT projects. If you do the work ahead of time and try to anticipate any potential issues, everything should simply fall into place.
What about that other 10-20%? You can try to get ahead of potential issues, yet sometimes there is still that special surprise waiting for you when you we are set to deploy. This can be a challenge, but over my time as a Network Engineer I have learned to love and embrace this challenge.
This is where my skills get put to the test. Lots of people can follow instructions in a knowledge base article. What happens when things do not go your way? EMBRACE TROUBLESHOOTING!
It can certainly be frustrating, especially when you do not initially know why it is not working. In the right scenario I can get the same rush from solving a complex issue as I would from finally finishing that boss battle in <insert favorite video game here>.
An example is a remote office deployment I was working on this summer. Long story short, we had packets that needed to route back to HQ and, even with our config in place, it still was not happening. We had just established the IPSec tunnel. I confirmed static routes looked good. Security policies were allowing the traffic. All the basics checked out. I would have to dig deeper.
Firewall logs were not telling me much – in fact, the lack of data told me more. Was the traffic even being generated in the first place? If the logs are not telling me what is happening, I will have to dig even further and look at the packets themselves.
I took some packet captures so I could get more detail on what was happening in the network. Sure enough, on the remote side I can see the traffic generated. However, on the HQ side I am not seeing any of the same packets. While this is clearly not the result I am looking for, I consider the packet capture a huge success. Based on these results, I know the issue is coming from the remote side. I just eliminated half of my suspects.
Next, I tried to understand why I am seeing the traffic but not seeing it logged. If I can determine that, it should also point me to the root cause of my issue. I thought to myself, “What needs to happen in order for that traffic to be logged?” Two specific requirements came to mind: 1) traffic needs to match a security policy before it is logged and 2) traffic is logged at the end of a session. I had checked and double-checked my security policies before. A triple-check and comparison to my packet capture confirmed that my policies were configured correctly. That leaves me with one final culprit… the session table.
I was relieved to find in the session table a “stuck” session. We initially made the change to the internal network before the IPSec tunnel was established. The packets I wanted to route to HQ were instead being sent out the Internet. Even though I had made the change to the config, the stuck session caused those changes to be ignored. Once I cleared the session, it was as if the flood gates had been opened. We immediately saw the connectivity we were looking for, and balance was restored to the universe.
There are certainly lessons to take away from this. My example above could have potentially been avoided, so it is always important to take note of what could have been done better. The real take-away for me is to embrace troubleshooting.
My example above took place over the course of no more than an hour. The roller coaster of emotions in that hour was exhilarating.
- Confident – Everything should proceed as planned.
- Confused – Why didn’t this work as I expected?
- Panicked – This isn’t working and, as of yet, I do not know why.
- Frustrated – The usual suspects all check out. What on earth could it be?!
- Elated – I AM THE SMARTEST MAN ALIVE!
When troubleshooting it is easy to get caught up in the confused, panicked, and frustrated emotions. Still, it is imperative to stay focused on the issue and not the emotions connected to it. Collect your potential suspects. Work through your testing and troubleshooting to eliminate each suspect, and eventually you will find your resolution. Trust me, the payoff is worth it.
Do you have something Atom Creek can help you troubleshoot? Let us know how we can help alleviate some of those confused, panicked, and frustrated feelings that can come with outstanding IT issues. We would love to leave you elated!
