PRESENTER OPINION : ARRL IT Security Incident

/

The ARRL released a report to members on August 21st, 2024, and then posted it publicly on August 22nd, 2024, in reference to their IT security incident in May of 2024. I reviewed the document, and have some reactions to it, as well as some opinions on “opportunities lost.” A typical mental experiment I routinely perform is “What would I do if I was involved in the process.” Some of these are opinions I would have given the ARRL leadership if I was engaged with them on the Incident Response team.

ARRL IT Security Incident - Report to Members

08/22/2024

Sometime in early May 2024, ARRL’s systems network was compromised by threat actors (TAs) using information they had purchased on the dark web.

The first sentence is a huge statement. My first reaction, it’s probably not a bad idea to expand this more than a sentence for the particular audience: the members of the ARRL.

Something like this: “Sometime in early May 2024, ARRL’s computer network and systems were compromised by criminal threat actors (TAs) using information they had purchased on the dark web. The dark web is an encrypted portion of the internet not visible to the general public via traditional search engines such as Google and is largely used by criminal organizations to conceal their activity online. Learn more here: https://en.wikipedia.org/wiki/Dark_web”

Random note, this summary was generated by google search’s AI thing: “The dark web is an encrypted portion of the internet not visible to the general public via traditional search engines such as Google.” One of the few examples of AI that actually wasn’t too bad. Bravo Google!

The TAs accessed headquarters on-site systems and most cloud-based systems. They used a wide variety of payloads affecting everything from desktops and laptops to Windows-based and Linux-based servers. Despite the wide variety of target configurations, the TAs seemed to have a payload that would host and execute encryption or deletion of network-based IT assets, as well as launch demands for a ransom payment, for every system.

This part seems genuine, however, I would have recommended to give another sentence about “ransomware as a service” offerings since this behavior of having payloads for a very diverse IT environment is usually one indicator that I generally consider part as a “ransomware as a service.”

This serious incident was an act of organized crime. The highly coordinated and executed attack took place during the early morning hours of May 15. That morning, as staff arrived, it was immediately apparent that ARRL had become the victim of an extensive and sophisticated ransomware attack.

I agree with this timeline based on previous information. One item that some organizations miss on public notices is contradicting information without a retraction / clarifying update.

The FBI categorized the attack as “unique” as they had not seen this level of sophistication among the many other attacks, they have experience with.

Seriously? This is not unique since 2021. I usually share this website with folks to make them aware: https://www.cisa.gov/news-events/cybersecurity-advisories/aa22-040a

“Cybersecurity Advisory: 2021 Trends Show Increased Globalized Threat of Ransomware”

There are other examples, I’m sure folks will email me tons of links I probably already have read; however, this 2021 Trend summary is something I felt was useful enough to make a bookmark in my “ransomware info” bookmark folder. I reference it often as I assist folks.

Within 3 hours a crisis management team had been constructed of ARRL management, an outside vendor with extensive resources and experience in the ransomware recovery space, attorneys experienced with managing the legal aspects of the attack including interfacing with the authorities, and our insurance carrier. The authorities were contacted immediately as was the ARRL President.

Great! This was a critical detail I wish they shared earlier – What actions did you take in that critical 72-hour period to prevent the incident from getting worse.

The ransom demands by the TAs, in exchange for access to their decryption tools, were exorbitant. It was clear they didn’t know, and didn’t care, that they had attacked a small 501(c)(3) organization with limited resources. Their ransom demands were dramatically weakened by the fact that they did not have access to any compromising data. It was also clear that they believed ARRL had extensive insurance coverage that would cover a multi-million-dollar ransom payment. After days of tense negotiation and brinkmanship, ARRL agreed to pay a $1 million ransom. That payment, along with the cost of restoration, has been largely covered by our insurance policy.

Ransomware payments are ALWAYS huge – the criminals are looking for the huge payday on every network they attack. So that is not unusual. What surprised me, they paid the ransom? I consistently recommend folks I work with on ransomware incidents to never pay the ransom, you are rewarding them for this and giving them more money to hack the next organization. I also am aware in my small circle of professionals that I routinely discuss Information Security topics with, paying the ransom is roughly a 50/50 bet. Half the time the decryption key works, other half of the time it doesn’t. So, between those two data points, I consistently recommend to not pay the ransom. This also is the advice given by multiple agencies: “Remember: Paying ransom will not ensure your data is decrypted or that your systems or data will no longer be compromised. CISA, MS-ISAC, and federal law enforcement do not recommend paying ransom.” (quoted from: https://www.cisa.gov/sites/default/files/2023-01/CISA_MS-ISAC_Ransomware%20Guide_S508C.pdf, page 12 – Another document I routinely reference when helping folks.)

From the start of the incident, the ARRL board met weekly using a continuing special board meeting for full progress reports and to offer assistance.

Weekly? Seriously? My experience we met daily with board updates, questions and answers from the board, after that initial 72 hours where we had touchpoint every few hours with board and key stakeholders.

We also had various collaboration tools open to share real time details of findings and recovery efforts and status updates from there.

In the first few meetings there were significant details to cover, and the board was thoughtfully engaged, asked important questions, and was fully supportive of the team at HQ to keep the restoration efforts moving.

Reading this I would ask the folks why does this need to be stated? Was this included to dispel a rumor going around? This was the only sentence that I found odd – maybe I need more details to understand this disclosure?

Member updates were posted to a single page on the website and were posted across the internet in many forums and groups. ARRL worked closely with professionals deeply experienced in ransomware matters on every post.

I stated something like this on the ICQ Podcast Episode 436 - Cyber Security in a Ham Radio World – Parts of the updates posted had indicators that were based on best practices standard letters to be tailored for the organization.

It is important to understand that the TAs had ARRL under a magnifying glass while we were negotiating. Based on the expert advice we were being given, we could not publicly communicate anything informative, useful, or potentially antagonistic to the TAs during this time frame.

BECAUSE YOU WERE NEGOTIATING THE RANSOM AND THEN PAID THE RANSOM!

Can you tell I’m annoyed? I work off a ransomware checklist shared by an incredibly large cybersecurity end point detection and recovery vendor, early in the checklist, cut off access to the threat actors, remediate the vector that they used, and verify that they can’t get back into your network. At that point, I DON’T CARE what you say about the threat actor, you’re NOT going to pay the ransom. If they have stolen data, they’re going to leak what they have anyway. However, ARRL did appear to pay the ransom anyway.

Today, most systems have been restored or are waiting for interfaces to come back online to interconnect them. While we have been in restoration mode, we have also been working to simplify the infrastructure to the extent possible. We anticipate that it may take another month or two to complete restoration under the new infrastructure guidelines and new standards.

WAIT, hold up – your infrastructure wasn’t simplified before? I know that sounds snarky, however in almost all cases, less is more when you’re trying to secure an environment from threats. This implies that there hasn’t been any external audits or reviews of IT operation in general and they ended up with something called “IT sprawl.” A good article to explain IT sprawl: https://jumpcloud.com/blog/what-is-it-sprawl.

It is a very positive sign that they stated that they are simplifying the infrastructure, and they still have more work to do, and they need more time. Time spent now to prevent the attack of tomorrow is always a good return on time investment.

Most ARRL member benefits remained operational during the attack. One that wasn’t was Logbook of The World (LoTW), which is one of our most popular member benefits. LoTW data was not impacted by the attack and once the environment was ready to again permit public access to ARRL network-based servers, we returned LoTW into service. The fact that LoTW took less than 4 days to get through a backlog that at times exceeded over 60,000 logs was outstanding.

Cool. What about DXCC processing? I would have mentioned something in this space, because it seems like a logical question a member would ask.

The board at the ARRL Second Board Meeting in July voted to approve a new committee, the Information Technology Advisory Committee. This will be comprised of ARRL staff, board members with demonstrated experience in IT, and additional members from the IT industry who are currently employed as subject matter experts in a few areas. They will help analyze and advise on future steps to take with ARRL IT within the financial means available to the organization.

This is great news, I volunteer! How does someone outside for the staff and board get invited?

We thank you for your patience as we navigated our way through this. The emails of moral support and offers of IT expertise were well received by the team. Although we are not entirely out of the woods yet and are still working to restore minor servers that serve internal needs (such as various email services like bulk mail and some internal reflectors), we are happy with the progress that has been made and for the incredible dedication of staff and consultants who continue to work together to bring this incident to a successful conclusion.

This information was shared with ARRL Members via email on August 21, 2024.

This is a decent letter overall; however, they missed what I consider a huge opportunity to share lessons learned. I also have questions about their cyber insurance – the last one I helped folks fill out, they had quite a few questions about the cyber security infrastructure and practices that had to be disclosed before the insurance would be priced and issued. What changes did they have to make to get cyber insurance in the first place, and now that they have been attacked, what changes are they going to have to make to get a renewal at the next insurance term? Sharing lessons learned could also discuss insurance as well, since it appears they were able to use the insurance to pay the ransom and pay for recovery post incident. What was the root cause? Knowing this and sharing this information could save the next organization that reads about this incident and makes a change to avoid being next.

As far as lessons learned, I would have end ed with something like this:

“This serious incident was a dramatic wake-up call for all small organizations and our constituents in the Amateur Radio community. As you are discussing this with your local clubs, field organizations, and served agencies that you volunteer for with ARES®, please keep cyber hygiene and best practices in mind so your organization doesn’t fall victim like our organization.”

I would then offer “learn more” links with brief details to highlight actions everyone should take: (assuming this is what could have been done to prevent the incident.)

Some places to get started:

  • Four Things you can do to keep yourself cyber safe

  • Turn on Multifactor Authentication

    • Multifactor authentication is a layered approach to securing data and applications where a system requires a user to present a combination of two or more credentials to verify a user’s identity for login. One is something you know – your password, two is something you have – a challenge/response or one time use code app on your cell phone.

  • Update Your Software

    • Criminals will exploit flaws in the computer systems. Vendors are working hard to fix them as soon as they can, however, we need to take action to update our software with their latest fixes

  • Think Before You Click

    • A very common trick to fool you will be to get you to perform some action, and example would be clicking on a link in email that takes you to a site that looks like a “known good” site, however, it is a trick to get you to type in your username and password. Spend a little time thinking about the link before you click. More details about this are in the link above.

  • Use Strong Passwords

    • Use long – at least 14 characters, unique – never used anywhere else passwords.

    • Implement a password manager, which will generate and track long random unique passwords for you.

  • Learn More about Ransomware - Ransomware guide: https://www.cisa.gov/sites/default/files/2023-01/CISA_MS-ISAC_Ransomware%20Guide_S508C.pdf

  • Stop ransomware online guide - https://www.cisa.gov/stopransomware

We would also recommend discussing with your members about cybersecurity. Offer a cybersecurity training session as a club talk, there are various good videos online you could use or solicit volunteers from folks you know who are knowledgeable in the topic of cyber security. Robust offline (backups that are separate from your environment that are not accessible without someone performing an action) and encrypted backups of critical data, both personally and club data, and regularly test these backups will enable all organizations to recover from an incident like this.”

Sharing some lessons learned would be a powerful statement of transparency at zero risk to the ARRL. Maybe this could be a follow-up communication later?

Protecting ourselves, clubs, and organizations we volunteer with starts with each of us. Educate yourself, encourage all organizations to take cyber threats seriously - implement IT and cybersecurity best practices, and keep robust backups that are separate from your environment is a good list of first steps that all of us could do to combating this constant threat.