by Konstantinos Xanthopoulos | Senior System Engineer
Background
Nearly before Christmas, Citrix publicly announced the CVE-2019-19781 vulnerability in Citrix Application Delivery Controller, Citrix Gateway, and Citrix SD-WAN WANOP appliances that could lead to arbitrary code execution affecting the majority of the NetScaler products that are under active maintenance.
This development has led to disarrays amongst Information Technology professionals given that in most IT departments, annual leaves were already in effect; thus, Incident Response procedures have been a bit harder to follow. It doesn’t seem so promising to have to deal with this while your leave has just started, right?
The days following the announcement, the company urged customers to apply the mitigation steps they have provided as workarounds to stop exploitation attempts until they release an official patch that addresses the vulnerability. However, at the same time, malicious actors have been lurking around the corner planning their hits seizing the opportunity with many appliances worldwide being susceptible to compromise.
The facts
The period between vulnerability announcement and the official patch release, a lot of things have happened from both sides, those who were in charge of defending the networks and those who had bad intentions towards the vulnerable appliances.
In the meantime, the researching community started setting up honeypots to verify how real is the danger of having vulnerable appliances compromised. It was only a matter of time until the malicious actors started attacking the honeypots successfully; thus, it was then official that the enemy was at the gates!
In the meantime, while Citrix was working on issuing patches to the affected appliances, they partnered with a group of Cyber Security experts to release the Indicator of Compromise (IoC) scanner tool which would be able to tell whether a vulnerable system has been hacked or not. The script accomplishes that by checking for any modifications on several system files.
Shortly after, Citrix started to gradually roll out official patches for the affected appliances, starting from the 19th of January 2020 for 11.1 and 12.0 versions and ending on the 24th of January 2020 with the patch release for 10.5 version.
According to the community feedback, a significant number of susceptible appliances have been attacked with many ultimately compromised. Since it doesn’t make any sense to patch a system that is compromised already, the only way moving forward has been to rebuild those systems from scratch.
Lessons learned
As with every similar incident (remember Ransomware panic back in 2017), the IT industry becomes a bit wiser both for being proactive as well as reactive. Regarding the first, defends in place get re-evaluated while concerning the second Incident Response procedures get improved by adding elements that were previously missing (e.g. distinct assigned roles to IT personnel).
If you are curious about what could you do yourself to align with best practices for deployments across every organization, regardless of its size, then, the below is for you!
-
Segment your network
In today’s hybrid/cloud world, companies are having their assets spread across different locations while, at the same time, the working landscape also changed rapidly; with trends like telecommuting being on the rise.
This reality adds complex challenges to the IT departments since they now must pay attention not only to their datacenter(s) but also to every location where company data reside and from where users can access corporate resources.
Well, that might be known already but still, how to deal with this?
The answer is, segment your network! Isolate your workloads by creating secure zones and protect each separately, thus, adding security layers to your infrastructure.
Let’s dig a bit more on this using a real-world example:
Presumably, you just got your costly next-gen firewall to protect your company’s network. At the same time, you ensured to configure it according to best practices (or even better, you brought us in to do it on your behalf!). So, you are now protected, right?
Well, indeed you are protected against most of the threats but what happens in case of the vendor, on a later time identifies a significant security flaw, rolls out an urgent patch to address it but you fail to apply in time? You got it; a breach is around the corner!
Now here comes the real value of this architecture: Your unpatched device may be hacked, compromised or whatsoever but the resources that the perpetrator could access would be significantly less than otherwise.
As a result, the chances that the break-in could lead to a potential compromise of most of the infrastructure parts or even worse of the entire infrastructure, are significantly limited.
-
Monitor your infrastructure and your published services
So, you have your services published securely and, you are confident nothing is going to happen. Or at least nearly nothing, besides cyberattacks are on the rise, remember? However, in case something happens; chances are you would want to be notified as earlier as possible so you can work your remediation actions.
That is why you need to have the right tools in place to detect any abnormal activity throughout your infrastructure. Products based on Machine Learning and Behaviour analysis can be an ally fulfilling this purpose hence consider to at least play around with some of them to get better overall perception and see in how well they could fit in your infrastructure.
-
Have an Incident Response procedure in place with distinct assigned roles per person
Most of the organizations, sooner or later have to deal with a less or more severe incident. The strategy and procedures an organization has in place concerning Incident Response; play an important role both for dealing with the alarming situation as well as for the affection the incident could have on its reputation publicly.
The two key futures when dealing with such incidents; is firstly discover the attack as soon as possible and secondly to effectively contain the damage. Both will make integrity restoration for the entire infrastructure more comfortable and practical.
For the plan to succeed, it is crucial to assign specific roles and duties to personnel, hold relevant training sessions regularly and last but not least ensure to document all the procedures for everyone to be aware of their responsibilities.
And as you might have guessed already, the last one on the list is …
-
Backup, backup, backup!
You’re right; I am referring to this “old school” practice that nearly “anyone” knows a thing or two about, which is often considered less significant between IT professionals.
In reality, backups are a few of the last (for most Small Businesses most likely the last) cards an affected organization has to restore its infrastructure partially or even completely.
However, this invaluable option comes with strings attached for the personnel that is in charge of backups since they will have to ensure they are following the principles of a proper backup strategy. Such one typically includes periodic checks of the integrity and validity of backups.
Although we will discuss more on this on a future post, it is necessary to mention that as a rule of thumb, one who would want to feel confident for the ability to restore from backups, should have no less than two copies of their backup sets if not more.
Correct, that is both onsite as well as at an offsite location!
If, while reading this you felt you have to reconsider some of your plans and try to improve them, please feel free to let us know so we can bring our experience in to support you on this expedition.
Otherwise, if you found your self following some or all of the above; kudos to you but still, we’d love to discuss how we can improve your current setup as there is always space for improvement!
Since you made it that far, I would like to thank you for attention, we will meet again in another blog post but until then, take care!
