Search
Close this search box.

A real IT team’s response to a global IT incident

banner-static


When it became apparent that rapidly spreading malware was in the wild, the first question 1E’s Internal IT Team needed answering was, “Are we vulnerable?”
Details of the attack vector were very sketchy for the first few hours but it looked likely that the malware exploited a known vulnerability that was patched by Microsoft in March.
At 1E, we patch frequently and we upgrade to the latest OS within a year of its release. We knew we only had to focus on a limited variety of operating systems and we definitely didn’t have any archaic Windows XP machines to worry about. Therefore, we were confident that most systems would be up to date and not vulnerable.
Why do I say “most” not all? Well, anybody who uses WSUS and ConfigMgr knows that it isn’t a perfect system and we had, in fact, found a bug that affected a few PCs so we couldn’t absolutely guarantee 100% compliance – and we wanted certainty.
So, step 1 – we needed to know if we had been compromised. Using our Tachyon product we quickly checked to see if we had any machines with a .wcry file on them. Details were sketchy as the news broke on Wanna Cry 2.0 – but we knew that encrypted files had this extension. With Tachyon we were able to check all fixed disks across all of our machines in India, UK, Ireland, and the US – including the 20% or so of our machines which are out of the office, but online, on any given day. We knew that we had no infected machines. The previous version of WannaCry also created files with extensions .wnry and .wncry so we checked for them too.
Step 2 was to ensure it stayed that way. It was confirmed that patches provided by Microsoft since March would prevent the spread of WannaCry 2.0 and that disabling SMB v1.0 would further reduce the risk of this particular malware spreading.
To make sure we were fully patched, we needed a script that would change the registry so that updates would go to Microsoft Online rather than WSUS, then launch Windows Updates. I know this potentially means that all PCs will try to update at the same time and potentially slow the network. However, as I said, I knew only a few PCs would actually need to download updates. Also, the alternative would be to remain at risk of being a victim of ransomware.
We created a Tachyon Product Pack that ran a PowerShell command. The command was actually a sequence of commands in one line which stopped the Windows Update service, changed the registry key to force Windows Updates to run from Microsoft Online, start the service, then run the update with an exit code letting us know it worked.
We targeted every PC and server. The servers we would restart manually as we wanted this level of control and users would be told if a restart was needed – but more on forcing a restart later. We hit around 95% of the PCs connected to our global network within 3 or 4 seconds.
At this point, more information about the first indicators of compromise was emerging so we used Tachyon to perform the following:

  1. Tell me if any of the executables known to be used by the ransomware were running. Responses back within a few seconds. Again, none found.

  2. We also asked Tachyon to see if these executables were stored anywhere on the hard drives. Within 10 seconds we were confident that there weren’t any.

  3. Within a couple of minutes of the publication of the file hashes for files associated with the ransomware, we had verified no files with these hashes existed on any of our PCs or servers.

At this point, and within a few hours of the malware first hitting the news, we were confident we were not compromised and all available patches were either installed or would be soon. Tachyon would also continue to check any PC that connected to the network for the next 24 hours so we could confidently leave the office. We also needed a decent night’s sleep before tackling the other vulnerability that this malware exploited – SMB v1.
We knew that the malware exploited SMB version 1 – a protocol only needed if you use older OSs such as Windows XP. Knowing we had mitigated our risk by patching against WannaCry 2.0, we could afford a little more time to research the impact of turning off SMB v1 and understanding how best to achieve that.
Tachyon was again the answer. We extended the Tachyon Product Pack to add or update the Registry key that disabled SMB1 on all OSs. This was preferable to deactivate the feature, as, by default, it would be enabled again if the feature were ever reactivated.
In some situations, it needs a system restart to take effect. We could have instructed Tachyon to perform this restart immediately as part of the instruction, but this may have caused our users to encounter problems and we definitely wanted to have more control over restarting our servers. This was a personal decision and the fact that we were confident that we were fully patched helped us to make an informed decision. However, we still wanted to know the restart had happened and SMB1 was disabled. There isn’t an easy way – or possibly anyway – to do this, so as part of the Product Pack, Tachyon would tag the endpoint so we could compare the time stamp of when SMB1 was disabled and when the endpoint was subsequently restarted. If the latter field was empty, the endpoint hadn’t been restarted. We gave our users a deadline within which to restart their machines, otherwise, we would use Tachyon to force the restart. Other restart methods are available that can delay the restart, inform the user, etc. but they all carry a degree of uncertainty, and we needed guarantees!
On Monday (15th May) Microsoft finally released a comprehensive list of the KB’s that contained the patch. You may think this would have been easy to work out but every cumulative update since March has the patch (one KB per month per version of OS) plus individual patches. Confirming that what we had performed the right steps comprehensively was very important to us as we wanted to also advise our customers on how to achieve the same successful outcome. This check is also now embedded in the WannaCry 2.0 pack – available for Tachyon customers to download to perform their own checks and update here.


Tachyon gave us the information we needed incredibly quickly so we could make informed decisions during a fast changing situation. It then gave us the ability to act on these decisions just as quickly. Need more? You can always contact us to learn more.
 

Report

The FORRESTER WAVE™: End-User Experience Management, Q3 2022

The FORRESTER WAVE™: End-User Experience Management, Q3 2022