Wargaming: running incident response plan tests

Part two finished with our defenders heading in the right direction to find the simulated infection. They'll have used a combination of tools in order to get this far, likely including firewall and network logs. As a reminder, for this exercise there was a simulation running that reached out to a server on the Internet at regular intervals.

Locate the "infection"

One of the injects in my scenario included the remote IP address that the infected machine was connecting to. Using logs from the firewall it was possible to find the source IP for the connection - the infected machine. Logs will have looked something like this [1]:

TIME                 SRC-IP        DST-IP        SRC-PORT     DST-PORT
2021-04-19 10:12:56  192.168.12.15 192.0.2.234   23445        80

Taking the source IP (SRC-IP above) of 192.168.12.15 the defenders were able to track down the infected machine to the workstation (end-user computers) address range. There were then a few options to pinpoint the device more accurately:

DHCP [2] logs will show the MAC [3] address of the computer, they may also show the computer name
Logs on managed network switches will show what port the infected computer is plugged in to
Asset management software may allow you to obtain more information on the device, including its user
Authentication logs would allow you to find the user that's logged on from the infected IP

Using a combination of factors the defenders were able to locate the infected machine, both geographically (physically) and logically (its location on the network). They could then isolate the infected machine.

Isolating the machine

In the interests of protecting the network and other devices, disconnecting the device from the network is an excellent first step. This could take the form of disabling the switch port, meaning the user would receive a "network cable unplugged" message, or kicking them off the VPN so the user would lose remote access to resources. If that's not practical for some reason this would be the point to call the user and ask them to power off (not shut down) their device. We'd be calling the user anyway, but our primary goal is to protect the network - customer service can wait!

Customer service takes a back seat here because in incident response time is critical. If this was a ransomware infection recovery can cost into the millions of pounds, and years to recover from, so upsetting a user because their computer is encrypting everything seems a fair trade to me. Redcar and Cleveland Council's attack has cost over £10 million.

Obviously this is a simulation, so the user to contact was the facilitator (me). Earlier I mentioned the computer should be powered off "not shut down" and this is to preserve forensic evidence. Malicious code could be hooked into the shut down routine to remove its presence or useful logs so by yanking the power cable (or holding down the power button) the infection doesn't have opportunity to run any clean up routines.

When things go wrong

Much like in a real incident, things can go wrong during a war game exercise. I'm going to address two issues here, one that impacted the facilitator and another that effected one of the participants.

Starting with the facilitator, after releasing the final inject to the team they created a firewall to block traffic to the "malicious" remote IP. Confusingly there were no hits on that firewall rule and the infected machine continued to communicate with the server successfully. Closer inspection showed the IP address the computer was connected to differed from the inject (it had previously been the same). Obviously it was crucial for the IP address to be correct so an additional inject was provided with a further IP. It's unclear what changed (it wasn't DNS) and I got around this problem for future simulations by changing the code to point to a web server that I control.

One of the defenders had his work laptop die on him part way through the exercise. This could happen during a live incident but we gave him a brief time to recover before carrying on without him. After he'd managed to switch to a backup computer, and contacted us, we brought him back into the incident via a conference call. You may argue this was unfair, but it was useful to demonstrate what we would do in the event of a problem. He was fully debriefed and didn't miss anything as a result - it was important he could learn from the exercise too.

If something had gone significantly wrong, say the simulator had stopped working and couldn't be recovered, it would be necessary to abort the exercise and reschedule. Of course, if a real incident occurred during the exercise that'd be time to stop too!

Reviewing

A bit like reviewing your science experiment in school (well, I had to anyway!), it's important to evaluate how the exercise went in order to both learn for running future exercises and so the defence team can reflect on their experience. Start by discussing what went well, and what didn't, but focus on the positives. As a facilitator you should have made some notes that you can use as prompts here. For example, I commented on how the teams knew where to look early on (their first port of call was the firewall logs) and that discussion was good among them.

Discussing what went badly, the pain points, is equally important so that changes can be made - granting access to tools, identifying additional tools or training to purchase, updating plans. Importantly this is not a blame exercise and there is no benefit to saying "well if Alex had done this at the beginnning...." - leave any personal grievances outside.

One request I received was for a more documented plan the team could follow in this scenario. That's in the works and we're looking at producing a number of playbooks that can be made available to response teams.

Be prepared to ask for feedback on yourself as a facilitator, as you need the opportunity to improve too. If you ask that question, make sure you do so in a way that people feel they can answer truthfully, without any form of penalty!

As a final question, check that the team are happy to be involved in a wargaming exercise again. Hopefully they'll say yes as these exercises can be quite fun (honest!).

Tidy up

It's important to tidy up after yourself - a good lesson in life and in IT! Potentially you'll have the following to dispose of or undo:

Firewall rules that blocked connections to the mailicious IP
Virtual Machines (VMs) that acted as your infected computer
Reports stored on any log appliances
Network switch ports that have been shutdown to contain the infection

Make sure you undo everything related directly to the exercise so there's not something left that someone will find in several months when everyone's forgotten. Worst case scenario someone would convene a live incident as a result...

Conclusion

I really hope this short series has shown you the benefit of wargaming / testing your incident response processes. As I mentioned, not only are these educational and a good tool for testing how prepared you are, they can also be fun. I recommend arranging some wargames of your own soon (and I know time is limited, but this is something it's worth making time for).

Banner image: The DaVinci virus from Hackers the movie ( (c) MGM / United Artists Entertainment 1995. ), lifted from this tweet to save me a job :) .

[1] In this log example, the destination IP address is taken from TEST-NET-1, a documentation IP address range of 192.0.2.0/24.

[2] Dynamic Host Configuration Protocol, DHCP, gives out addresses to computers so they can access network resources.

[3] Media Access Control, MAC, is the hardware address of a network card. A MAC address looks like this: 00:1B:44:11:3B:B7.