In 2019, a bank in the Isle of Man was the subject of a major incursion into its systems, which led to an exfiltration of data on a large scale. Kurt Roosen FBCS, Head of Innovation for Digital Isle of Man, who was the Head of IT for the bank at the time, shares his experiences and the lessons learnt.
"We have credible information that your systems have been breached and data have been taken. We also believe that this is ongoing..."
This is not the ideal message to receive from enforcement authorities at just before 5pm on a Friday afternoon, or indeed any other time. In complete isolation, this is enough information from a credible source that you need to take immediate action, but at the same time so little information that you have to think where to start and what to do. You’re given the challenge of finding the proverbial needle in a haystack, or in this case, a huge number of haystacks, each concealing an array of sharp implements.
We needed to find time for consideration. Our only real choice became to close ourselves off and all our internet connections, too. We were fortunate that as we were a business focused bank with predominantly local customers, transactions did not really happen at weekends. The timing of the call on a Friday evening worked in our favour in that respect.
We also had externally hosted telephone and information websites and a proxy system for our email so customers could stay in contact and we would not lose any communications. At 6pm we closed ourselves off from the world.
Using a military mindset
Once isolated, we had to look for signs of incursion, starting with the logs across our systems. Nothing obvious resulted from this, so this is where we took a tactical decision to bring in people with a military mindset to treat this as a surprise war and to coordinate a response to discover, contain and eliminate.
The first part was actually the most controversial – how to discover the ‘what’ and ‘how’. We were approaching the end of the weekend and, as a bank, needed to be open to our clients on Monday. But, at this point, we had no way to provide a defence or know the scope of any data loss.
Opening up our systems in the knowledge that we had a flaw, potentially exposed us to a wider issue, a legal quandary about responsibility from this point forward. On the other side, we did not know the problem (just that one existed) so any action we took could not be verified as the resolution.
If we remained without definitive knowledge, we would never be able to issue any definitive assurance going forward, so it was essential that we found out what had happened and how the incursion took place. The only way to do this was to observe an incursion by creating a honey trap.
The opening dilemma
In doing this, we had to be very confident that we could both observe and contain the problem. We didn’t want to make our situation worse. Initially, we installed a security information and event management system (SIEM) from scratch.
We did this in a couple of days and turned up all logs on all systems to their most verbose level. We also installed application firewalls on every server and every workstation and segmented application usage such that explicit access was required to perform any function.
Not a light undertaking. However, we did not change any of the security structures, virus, intrusion detection systems or the location of data so that the route of incursion would not alter.
Informing the Information Commissioner’s Office
At this point we needed to inform our regulator and Information Commissioner’s Office. We laid out the situation to them: we did not have facts at this time. The technical steps we were taking were specifically to get that clarity. Thus began a daily, highly transparent, dialogue that ultimately proved to be very helpful and productive. However, the issue was not disclosed to our general staff or our clients at this time, as this would have potentially made our honey trap ineffective.
We also pursued the original route that alerted us to the fact that we had an issue. Three weeks later, we managed to get access to the original source, the Cyber Security Unit of an Eastern European Police Force. A plane trip and journey into the concrete basement of an Eastern European police station allowed us (rather sickeningly) to see our own data on a screen and verify it was ours.
This also confirmed the scale of the exfiltration, which in multiple copies involved some 2TB of data in total, using our virtual server hosts as the target. This allowed us to determine the ‘what’; we had lost almost everything, over multiple occasions going back over a year. However, whilst we had part of the method, we did not know the point of incursion and the full methods deployed, although we now had some suspicions.
Looking darkly across Europe
It was also easy to assume, because of the location of the data, that the perpetrators were also Eastern European – in fact this was not the case. There had been an arrest of a hacker who extracted funds from a local bank, and this led to a wider investigation around a set of people that had been very active in Europe and beyond.
Their activities included the CIA, Barcelona Police and even Hacking Team, a Milan based company who wrote spyware for governments. In fact, the data that we were shown originated from a data centre in Paris and the main participants of the hacks appeared to be from Spain.
So, we were faced at this point with a sophisticated, multinational operation whose members were sought by worldwide enforcement organisations and had remained elusive for a number of years. Their modus operandi were as hacktivists, operating for a noble cause, but by extorting financial recompense from their targets. We were told to expect an attempt to divert money and then possibly a ransom demand.
Cat and mouse
Armed with this knowledge, we returned to the vigil, waiting for the inevitable return, but quietly confident that we had built up our defensive barriers such that we would contain the perpetrator and be able to observe what they were doing. A few days later they duly obliged.
For you
Be part of something bigger, join BCS, The Chartered Institute for IT.
Early on a Sunday morning we saw the entry into the system via patterns detected using the AI capabilities of our SIEM. We watched as the perpetrator bounced around our newly erected firewalls, attempted to use a number of, now disabled, profiles and then used stored toolsets to take away an encrypted password file using an elevated authority service profile, presumably for future decryption.
We now knew the point of incursion, had seen the toolsets used and the method of extraction. It was obvious that the hacker noticed the changes to the environment and knew that we were on to them, and so we prepared for the ransom demand – but this did not come.
Less than two hours after the monitored incursion, all of our data was published behind a prepared online article in a US based hacktivist magazine together with a 21 page manifesto from a person claiming to be Phineas Fisher. There were very detailed descriptions of what they had done and why.
Behind the sentiments of protecting the poor from evil, corrupt government and financial institutions, there was a need to create a furore to cover their tracks. They offered a $100,000 prize to the next hacker of a bank. This led to our systems lighting up and all our logging becoming overwhelmed.
Exploring the logs
By this time, we were analysing the data we had stored safely away in our SIEM and comparing them with the original logs of our systems. The first, most prominent, feature was that our original logs were devoid of any information relating to the hack.
Similarly, our virus, anti-malware, firewall and intrusion detection systems (IDSs) were also silent. Although we physically watched the attack take place in real time, only minutes after they left, none of our systems showed a single trace of any of this activity. It was as if they had never been there.
Our anti-virus and IDS’s silence was strange. They were both high end products and had been installed in the last 12 months. We were very curious to say the least.
These were administered using a web based console hosted by the software manufacturer. These had been quite separately hacked and our policies changed to render them useless. The IDS had its policy changed to ignore spurious PowerShell commands and the anti-virus / malware system had been changed to only assess files/programs with a new date stamp.
This meant malware tools sat in clear sight in directories with deliberately altered past date stamps. What was more disconcerting was that the policies were changed within a month of our creating them and there were no change logs in the provider systems.
The hack’s nature becomes clear
We had been a very early victim of a supply chain attack. At least three external systems had been separately hacked in a coordinated manner. This made us vulnerable and once the bridgehead was established the hackers began researching: watching communications, reading procedure manuals and reacting to changes in security posture. This gave the perpetrators a clear view of our systems and their potential.
In addition, the source of the very original seeding incursion was identified as a commercial firewall that had a flaw in its firmware which allowed it to be remotely placed into console mode where the firmware could be updated. The hacker had installed amended firmware that performed the original functions but also mirrored passwords used on this remote access firewall, to a remote site.
This captured one privileged access password which was then used to legitimately access the system and establish a bridgehead which, as long as they entered the system once a month and updated their passwords, remained open with no additional effort.
The firewall itself reverted to its original firmware when it was power cycled, but significantly, the flaw in the code that allowed this to happen survived all patching by the manufacturer and was still possible to do in a lab some four years after it had originally been reported.
What did we learn?
We became a very introvert organisation with almost zero external trust. We also significantly layered our systems such that one reporting system would never give us an answer in isolation; we had different opinions as they had different methods to ascertain security.
We also dispersed systems and data so that it was never evident what parts had to be reconstructed to make something that makes sense, and we put two factor authentication and encryption everywhere, again in multiple layers. We also learnt that security is impossible without a reactive SIEM.
Beyond the technology, we had to accept that we could not ultimately have prevented the incursion. That sounds a very dramatic statement but the coordination and sophistication were beyond the defensive capabilities that we could have reasonably deployed.
The time and effort devoted to hacking into our tiny bank with no financial reward was not performed by a back room hacker, but a sponsored group with infinite time and resources that was not seeking a financial return.
When looked at in terms of available capability, the motivation can be anything from reputational harm to the collection of a very small amount of information to a type of financial denial of service where the organisation cannot afford the correct remediation. This is not always about a financial loss to the clients. Neither our clients nor the bank lost any money directly.
In the recent Solarwinds supplier chain attack, Ciaran Martin, former head of Britain's National Cyber Security Centre, said in The Register: ‘A lot of state-sponsored hacking work consists in essence of picking the lock, opening the door, and then trying to figure out what you've just found. This contrasts with the popular view that fiendish adversaries pick their targets with ruthless precision and then execute a surgical cyber-strike to get what they're after.’
Cyber security is no longer about if you’ll be hacked, but when. We don’t necessarily need to spot every hack, rather we need to detect them early and slow the incursion down sufficiently. If we do this, we can buy time to act on intelligence. It is about surviving attacks financially and emotionally.
So, accept the assertion that everything can be compromised, at least for some period of time, if the adversary is determined enough. Then, we have to look at digital systems. We should seek to wrap actions around them. The most vulnerable will be those that are open to the public internet. We should also start thinking now about the increasingly influential crypto space.
Nothing is insurmountable. But, the final message here is to expect the worst and plan for it. We are entering an era where nothing is immune to attack. That sense of being able to keep everything out must be severely challenged and attitudes must change to match accordingly.
