Negotiating Ransoms: When to Play and When to Fold

When Colonial Pipeline was struck with ransomware last month, many were surprised at how quickly the company paid the $4.4 million ransom. Surely a business that big and critical to the economy had sufficient resources and plans in place to recover quickly without needing to capitulate to extortionists.

But Colonial Pipeline CEO Joseph Blount told lawmakers on Capitol Hill this week that although his company had an emergency-response plan in place, it didn’t include plans for responding to a ransomware attack. The company did have insurance to pay for ransomware attacks, however, so the decision to pay was swift.

A ransomware notice first appeared on a machine in Colonial Pipeline’s control room around 5am on May 7, Blount testified. By 6am the company had shut down its 5,000-mile pipeline. Within another hour the company had contacted outside legal counsel and engaged digital investigations firm Mandiant to begin a forensic assessment of the damage. By late afternoon that day, Blount had decided to pay the bandits, and on May 8 the money was sent.

The rise in ransomware as a business for criminals has produced a parallel rise in companies engaged in helping victims negotiate ransoms and recover. Negotiating ransoms is a fraught process that can take more than a week and change rapidly, depending on the whims of the extortionists and the state of the victim’s backups, according to Bill Siegel, CEO and co-founder of Coveware, a company that negotiates ransomware payments for victims. His firm also aggregates statistics and other data about ransomware incidents to help the government track the scourge.

Coveware has negotiated a “few thousand” ransomware cases since its founding in 2018, and each case is different, Siegel says. He declined to discuss his customers or the specifics of negotiations, to avoid giving ransomware actors insight into negotiating tactics. But he did say that his company won’t negotiate any ransomware attack conducted by the Darkside group that hit Colonial Pipeline.

Last November Darkside bragged that it was planning to start using servers in Iran to store data it steals from victims, in order to make it harder for U.S. law enforcement to get access to the servers to seize the data or take the servers offline. But Iran is a sanctioned country, which makes paying Darkside legally risky for victims. Last year the U.S. Treasury Department's Office of Foreign Assets Control warned that victims are potentially at risk of violating OFAC regulations if they pay ransom to a group on OFAC’s list of sanctioned entities. Darkside isn’t currently on that list, but Siegel says any ransom paid to Darkside could potentially be used by Darkside to pay Iranian companies for hosting services. Darkside later walked back on its plans to store stolen data in Iran, but Siegel says his company just doesn’t want to take any risks that Darkside might change its mind and his company could inadvertently violate sanctions.

I spoke with Siegel about the calculations victims make in deciding when to pay ransoms, the things that can go wrong with decrypting ransomed data and why victims in Europe often pay less in ransom money than victims in the U.S. The interview has been edited for length and continuity.

We’ve seen outrageous ransoms lately for tens of millions of dollars. Many of them get negotiated down to lower payments. How does a victim who can afford to pay decide what they will pay?

The threat actors that do this … started just testing the waters [with large ransoms] and so you get these crazy demands — $3 million, $10 million, $50 million. They just make these crazy numbers up, so then when they cut it in half they can sound generous. What matters [for the victim] is, What is the financial impact to the business that could be averted by hastening the recovery?… What is this worth? It’s a very hard question to answer, but most of the time … an enterprise can boil this down: “This is costing us this amount per day or this amount per hour.” And so they know what shaving a day or a week off of their recovery [time] will save them.

Handling one of these negotiations, everything moves. The business value — Why should we do this? What’s the value of it? — that changes every twelve hours for the company…. The final decision to actually pay, that only happens at the very end, [after you’ve determined if you can recover from your backups and avoid paying.]

[But] it can be very difficult to ascertain the integrity of your backups.… It can take several days…. So if a negotiation goes five or seven days, it typically means that the company isn’t sure if they actually need to pay or not. What you want to do is … complete the negotiations so that we’re at the finish line. And at that point in time, you will probably know whether or not you need to [pay].… If someone has to pay very fast, it’s typically because they know they have no other means to recover.

You talked abut the integrity of backups and that it’s not sufficient just to have backups.

When you have the properly configured backups you’ll be okay. The issue is, most companies don’t have properly configured backups, or they haven’t tested their resiliency or the ability to recover their backups against the ransomware scenario.

What do you mean if they don’t have the right configuration?

It can be [that they] have 50 petabytes of backups … but it’s in a … facility 30 miles away.… And then they start [restoring over a copper wire from those remote backups] and it’s going really slow … and someone pulls out a calculator and realizes it’s going to take 69 years [to restore what they need]. And then it’s like, “Oh god, how did we never think of this?” Well, you never practiced [restoring your data].

Or there’s lots of software applications that you actually use to do a restore, and some of these applications are in your network [that got] encrypted.  So you’re like, “Oh great. We have backups, the data is there, but the application to actually do the restoration is encrypted.” So there’s all these little things that can trip you up, that prevent you from doing a restore when you don’t practice….

In addition … there is so much pressure to have recovery-time objectives — the downtime that the business side and the technology side agree that they can sustain. It’s typically measured in hours. So it’s like … “We have to be fully recovered within three hours.” [But] … with very large networks and lots of data…it’s not physically possible to … restore … in hours, [even] over the fastest connectivity or the fastest machines…. For a big network, it can take days, sometimes weeks just to restore backups and decrypt everything…. The only way to make it fast is to draw everything inside the network and … put the [backup] servers … on the same network [as the primary systems]. [But] then [the backups] just get wiped or encrypted because the bad guys can find them super easily.

One of the reasons I think that European companies actually end up paying a lot less [in ransom] is they don’t put this amount of pressure on their technology teams for these tight recovery-time objectives … and so, accordingly, they properly segment things and they’re okay with tape [backups]. In the U.S., people are like, “Tapes? [Those are] ancient, too slow.” But I can’t tell you how many companies [hit by ransomware in the U.S.] would have killed for the option to have their tape backups show up on a FedEx truck three days later…. In Europe, most companies have tape backup and it’s fine. They’re like, it’s okay, we’ll be down for a week and … it will be embarrassing … but we won’t lose any data.

I spoke with someone who works in ransomware recovery and he told me they advise customers to build an entirely new network and to never use the ransomed systems again because they’re not trustworthy.

A server that gets impacted with ransomware has to be, at a minimum, heavily remediated to ever be trusted again. The best practice is to stand up a green network where you’re re-imaging all of the servers, you’re re-installing all of the applications. And then for the data, you have backups that you’re restoring to the green-network machines. With [desktop computers], … most of the time the desktops are just re-imaged [not restored]. It’s like, “Look, if you saved your family photos on there, sorry they’re gone.” You don’t pay for a decryptor to decrypt desktop machines…. If you work for a big company, they will tell you, if it’s not on the file-server don’t expect to ever get it back. Don’t save anything locally; save it on the network where it’s backed up.

How often have you run into a problem with the decryption key not working — either because the hackers badly coded it or because it’s not compatible with the victim’s systems? How often is it useless?

Useless? I would say … less then 5 percent of cases. But there are always issues [with the decryption], because ransomware messes up computers. Most of the time, though, if it’s the correct key and the data was properly encrypted by the ransomware, it’s going to be recoverable. Most of the time when you see data-loss it’s because the malware had a math error in it and … it just corrupted the file, or it overwrites some bytes, or it bricks the server in a certain section and it just breaks the data.

If you discover that the data was corrupted during the encryption process, is it game over?

Most of the time, yeah. If it’s database files, typically they’re gone. If it’s a text file or a picture, sometimes it’s just a couple of characters missing at the end or a couple of pixels are off. But if it’s database files, they’re typically toast.

The software that the bad guys wrap [around] the key … is also garbage…. It’s always a crappy Windows executable, and people are like “What if it has other malware on it?” Which is a valid concern.… So when we get the decryptor, we extract the actual key [from it]. We don’t need the software that’s wrapped around it. We extract the key and we embed it in our software tool, so we give [the victim] our tool, [which] works like a piece of real enterprise software….

[To determine if files will decrypt properly] you need to do scans on all of the [encrypted] files to look at the integrity of the encryption. A properly encrypted file will normally properly decrypt. [But] every type of ransomware stripes the data differently with its encryption.… If it’s Ryuk [ransomware], we know that it saves the key on this section of the file, so we make sure that the key is there. Sodin [ransomware], it saves it in the footer, so we have to make sure that the footers are all there. [Our software] will rip through every single file on the network and make sure that every file has those components to it. And if it’s not there, then it will say that out of 20 terabytes [of encrypted files], you’ve got 150 gigs that are already corrupt and you’re not going to get those back…. If that’s your critical data, then that’s a point where you keep negotiating [on the ransom]. If that’s all junk data and you don’t care, [that will help determine if you want to pay the ransom].

What was interesting about the Colonial Pipeline outage is that it had cascading effects that the company had no direct control over — that is, the reaction of the people who started hoarding gasoline. Even though there wasn’t actually a fuel shortage, people created a fuel shortage from panic. So there are all of these follow-on effects you might not anticipate that can put pressure on you to pay the ransom.

That’s a special case because gas prices are obviously a major economic trigger, which means they’re a major political trigger. Cases like that are somewhat rare because there’s not a lot of infrastructure that has the ability to have that sort of broad impact to consumers. But it’s a very healthy wake-up call. And to be candid, I’m glad that the outrage has been there… Because you need to get people fired up and caring about this….

At the end of the day, as scary as it all was, we will look back at this and say this was a really important incident to witness and have happen, because it’s a major wake-up call that we’re not as safe as we thought we were.

Related Stories:

Anatomy of a $2 Million Darkside Ransomware Breach

Ransomware Infection on Colonial Pipeline Shows Potential for Worse Gas Disruption

US Gov Issues Emergency Order While Colonial Pipeline Is Down

Darkside Retreats to the Dark

If you like this interview, feel free to share with others.

If you’d like to receive future articles directly to your email in-box, you can also subscribe: