In the last few days we have learned of a very serious cyber attack launched against many organizations by a still unconfirmed attacker. Some suggest nation states are behind it, but whomever has done it, was very very smart. This will go down as one of the most genius hacks of all time. Solarwinds one of the largest and most prolific network/server management platforms was hacked from the inside, their supply chain of software actually distributed malware, to the inside of the network, behind firewalls, into a system that is the nerve centre of networks. The hack itself is frankly genius. It’s pure evil – but it’s genius.

There are many blogs out there taking on the task of taking apart the hack itself – I will not do this here, I am not a security expert. I would encourage you to seek those out.

There is a 100% chance your organization will be attacked, and even odds that it will actually be breached. This is something that you need to understand and accept – have plans for each of these, but, what if you do not? This attack happened from the inside out from a trusted software vendor. No firewall could have protected against this.

We were all new once

Do you remember that? I do. Wow did I do some stupid things while learning. Thankfully in my early career days I had some more senior engineers who helped (or took pity) me learn. If it wasn’t for those people, I wouldn’t be where I am.

Even the experienced make mistakes

I have a confession to make. Even after years of experience, I have done dumb things. I have taken down large environments in the middle of a business day, I have broken things – all things that could have been avoided had I made the right choice. Most of those mistakes were a single command, or mouse click – followed by “Oh Sh–” and an immediate learning moment. Understand that technology engineers often have to make split second decisions that can result in either very positive or very negative outcomes, and planning time is not always on our side. Sometimes we are also working alone, with nobody to discuss the plan with, and are working against some kind of imposed timeline.

Some manufacturers say “You can replace this CPU card in the middle of the operating day” — the so called “Hot Swap”. Ok, would you do this though? Ask yourself one question.. The manufacturer says I can – but should I? What are the repercussions if it goes down when I am doing that? Remember, you are the one making that decision – do not blame the manufacturer if it goes wrong.

Limited resources result in complacency and cut corners

When people are pushed to the limit, especially those in smaller IT organizations that is where corners get cut. Maybe you used Telnet when you should have enabled SSH, or used the same passwords for a bunch of accounts. For years and years, service accounts were built with full-admin rights to the entire network and even given interactive logon rights. We did these things and didn’t think anything of it. Now we know better, but did we go back and fix every single little thing we ever did wrong?

Everyone at some point – did it wrong. The question is, did someone else go back and do it right?

We need to stop the “Blame Game” which is so prevalent in our industry. My cloud application went down, blame the cloud provider, the switch went down when I typed this command that should have been ok, blame the manufacturer.

We need to learn, and grow, and make better decisions and understand that sometimes things happen. The question is, did you think about what the consequences were, and weigh the positive and negative outcomes of what you were about to do. Now imagine doing that calculation in your brain – with – every – keystroke.

Help each other, avoid standing in the ivory tower

This week, many people will lose their jobs, as a result many will lose their homes and some their livelihoods, because someone needed someone to blame. For what? Possibly a human error. That’s right, behind every one of these electron switching super computers, firewalls, switches and lines of code, is a real fallible learning human being. Who makes mistakes, and grows from them.

Instead of being critical of each other, now is the time to band together and help everyone remediate the damage, and take this as a serious warning that we need to take certain things more seriously. Password management, inter-domain firewalls, security posture on operating systems.

So in that light, I urge you that if you, this week, are looking at your IT team, who right now might be panicking about what to do, questioning if they were attacked, questioning if they were impacted and worried for their job, and you are wondering if they know what they are doing about this. The reality is that almost nothing could have been done to protect against this.

Ask them these 3 simple questions.

1. Are we affected by this?
2. What are we doing about it?
3. How do we prevent this from affecting us again, or in the future.

Then tell them you know their job is difficult – they have literally thousands of hackers, and government bad actors against them – and they are just a small IT team running your business. It is a hard job and they need your support.