Don’t Be Troubled with Troubleshooting

by | Dec 16, 2020 | Cybersecurity

By Scott Carson, Network Engineer


Tell me if you have heard this one before…

You have been working diligently on a project. The time has come for the final deployment. The grand finale if you will. Everything is in place. You power it on. It works flawlessly! All your validation steps check out and everyone goes home happy. Another successful deployment in the books.

Without getting into the actual statistics, it feels like this describes at least 80-90% of IT projects. If you do the work ahead of time and try to anticipate any potential issues, everything should simply fall into place.

What about that other 10-20%? You can try to get ahead of potential issues, yet sometimes there is still that special surprise waiting for you when you we are set to deploy. This can be a challenge, but over my time as a Network Engineer I have learned to love and embrace this challenge.

This is where my skills get put to the test. Lots of people can follow instructions in a knowledge base article. What happens when things do not go your way? EMBRACE TROUBLESHOOTING!

It can certainly be frustrating, especially when you do not initially know why it is not working. In the right scenario I can get the same rush from solving a complex issue as I would from finally finishing that boss battle in <insert favorite video game here>.

An example is a remote office deployment I was working on this summer. Long story short, we had packets that needed to route back to HQ and, even with our config in place, it still was not happening. We had just established the IPSec tunnel. I confirmed static routes looked good. Security policies were allowing the traffic. All the basics checked out. I would have to dig deeper.

Firewall logs were not telling me much – in fact, the lack of data told me more. Was the traffic even being generated in the first place? If the logs are not telling me what is happening, I will have to dig even further and look at the packets themselves.

I took some packet captures so I could get more detail on what was happening in the network. Sure enough, on the remote side I can see the traffic generated. However, on the HQ side I am not seeing any of the same packets. While this is clearly not the result I am looking for, I consider the packet capture a huge success. Based on these results, I know the issue is coming from the remote side. I just eliminated half of my suspects.

Next, I tried to understand why I am seeing the traffic but not seeing it logged. If I can determine that, it should also point me to the root cause of my issue. I thought to myself, “What needs to happen in order for that traffic to be logged?” Two specific requirements came to mind: 1) traffic needs to match a security policy before it is logged and 2) traffic is logged at the end of a session. I had checked and double-checked my security policies before. A triple-check and comparison to my packet capture confirmed that my policies were configured correctly. That leaves me with one final culprit… the session table.

I was relieved to find in the session table a “stuck” session. We initially made the change to the internal network before the IPSec tunnel was established. The packets I wanted to route to HQ were instead being sent out the Internet. Even though I had made the change to the config, the stuck session caused those changes to be ignored. Once I cleared the session, it was as if the flood gates had been opened. We immediately saw the connectivity we were looking for, and balance was restored to the universe.

There are certainly lessons to take away from this. My example above could have potentially been avoided, so it is always important to take note of what could have been done better. The real take-away for me is to embrace troubleshooting.

My example above took place over the course of no more than an hour. The roller coaster of emotions in that hour was exhilarating.

  • Confident – Everything should proceed as planned.
  • Confused – Why didn’t this work as I expected?
  • Panicked – This isn’t working and, as of yet, I do not know why.
  • Frustrated – The usual suspects all check out. What on earth could it be?!

When troubleshooting it is easy to get caught up in the confused, panicked, and frustrated emotions. Still, it is imperative to stay focused on the issue and not the emotions connected to it. Collect your potential suspects. Work through your testing and troubleshooting to eliminate each suspect, and eventually you will find your resolution. Trust me, the payoff is worth it.

Do you have something Atom Creek can help you troubleshoot? Let us know how we can help alleviate some of those confused, panicked, and frustrated feelings that can come with outstanding IT issues. We would love to leave you elated!

See What Our Partners Are Saying

Atom Creek is an exceptional managed services provider. Customer experience is a priority. My experience with Atom Creek has been that the engagement and focus of the top levels of leadership and the owner of the company on projects is second to none. Broad knowledge and being at the forefront of new technologies and innovations is a key characteristic of the firm and one of many reasons so many clients stay with them for so long. Highly recommend.

Brian Bybee

We have worked with Atom Creek for 7 years, they are always on the forefront of security and compliance for our company. When you need computer help from the techs they know us like they are your own internal department and not the random call center help person. Atom Creek is our IT department and is Amazing.

Kirsten Berger

Atom Creek is great they took care of everything we needed in a timely manner with zero issues. Paid great attention to exactly what was needed and delivered everything perfect. Highly recommend!

Tyler Holt

Excellent managed service provider with a highly skilled and customer-focused team!

Jon Taylor

Great folks to work with. Honest work and honest price.

Stephen G

Great business, knowledge and customer service.

John Schliep

Take the Next Step with Atom Creek

Contact us and learn how to create a harmonized IT environment that makes achieving your business goals faster and easier.