Providers of non-detection-oriented security solutions need objective, transparent methodologies to evaluate and frame their value propositions. This need was the driver behind the launch of MITRE Engenuity ATT&CK® Evaluations Trials, an exploratory new program designed to help niche solutions providers realistically describe their strengths in defending against known adversary behavior.
Results from the first Trials project, which focused on deception solutions, are now available and published on the ATT&CK Evaluations website. In keeping with CounterCraft’s commitment to transparency and trust, here’s a behind-the-scenes look at how we approached the evaluations, our methodology, and findings.
Deception is the only way to provide advanced warning of how an organization is being targeted by their adversaries. It provides high-fidelity alerts on attack patterns, tactics, techniques, and procedures and is the only way to build a detailed profile of the who, what and why of an attack, as it relates directly to you. This detailed threat intelligence is timely, focused, and above all actionable because the intel being collected is directly related to your own organization and your attack surface.
Deception is the new kid on the block, and CounterCraft has consciously designed our platform to fit seamlessly into an existing security infrastructure. With over 45 integrations to third-party security systems out of the box, fitting in with incumbent systems is simple, either to add valuable threat intel feeds to existing analysis systems or to interact with orchestration systems to provide automated cross-platform attack response.
MITRE Engenuity
CounterCraft participated to compare our deception tech with other vendors in a clear and neutral way. Standard minimal functionality and performance criteria do not exist in the deception space, making it confusing for end users when evaluating deception technology. We hoped these trials would change that. Standard testing criteria would benefit end users and allow different vendor approaches to shine in different ways. We believe MITRE is uniquely positioned to create this common and fair testing framework, and we are proud to invest in the ATT&CK Evaluations as a way to provide value to the entire deception user and vendor community.
The CounterCraft expectation was to collaborate with MITRE Engenuity in the creation of the testing methodology, in order to fully define metrics that measure the different benefits of a deception platform and help provide a way of identifying key strengths of each technology. The goal of this is to educate potential users of how to best deploy deception according to their needs. This expectation has not been reached yet, but this initial Trial is a start and provides many areas to build upon.
CounterCraft hopes that in the future the evaluation process will provide an independent analysis, on common ground from a known impartial source, to help identify how best to deploy deception. It should provide a benchmark to measure all deception platforms equally, highlighting strong areas and helping potential consumers identify the best tool for their specific deception deployment. At the forefront of our minds was the possibility of providing an open and impartial analysis of deception technology, both to raise awareness and to showcase the CounterCraft approach to deception deployment.
CounterCraft Deployment
The CounterCraft approach to deception deployment is based on zero-impact to the production environment. In keeping with this, we approached the testing environment as the customer production environment. For the purposes of the evaluation we instrumented the test hosts [Scranton, New York, Utica Warehouse, Nashua, Yonkers and Schrute Farms] with the CounterCraft DeepSense Agent, in order to monitor activity, but no additional deception software was deployed.
The deception environment deployed by CounterCraft comprised a number of addtional deception hosts [Sabre, Buffalo, Stamford and GitLab] set up so that the adversary could connect to them as part of their attack plan. All the deception assets were real operating systems (running in Azure) instrumented by the DeepSense agent that is responsible for collecting the telemetry data in a stealthy manner.
The ActiveLink network provides a full command and control network for the deception environment and allows completely cloaked exfiltration of DeepSense telemetry and also allows real-time control over hosts, services and breadcrumbs for instant response to adversary activity.
Custom or template-based CounterCraft breadcrumb technology was deployed across the hosts and MITRE servers. Both passive and active breadcrumbs allow you to tailor them to attract exactly the adversary you are targeting.
The main job of the ActiveLures is to attract adversaries into the CounterCraft ActiveSense Environments. The follwoing deception hosts and ActiveLures were deployed:
- Sabre (Windows 2019 Server / 10.0.1.9): was exposing a RDP and a shared folder. If an adversary connects via RDP, there are some PowerShell scripts that include the backup AD Domain Admin credentials.
- Buffalo (Windows 2019 Server / 10.0.1.1): was a real Active Directory that pretended to be a backup AD. It had a one-way trust relationship with the production AD. It was vulnerable to the CVE-2020-1472 (ZeroLogon) vulnerability.
- Ubuntu20 (Ubuntu 20.04 / 10.0.1.11): was accessible via SSH and it had a vulnerable Gitlab server (CVE-2021-2205)
- Stamford (Windows 10 / 10.0.1.12): was accessible via RDP and included credentials and information about the AD.
Evaluation results
Deception products are not as straightforward to evaluate as detection-oriented security systems, and they are far more nuanced than detection products. Before comparing results and weighing participants’ solutions against one another, note again that the two products have fundamentally different approaches.
As previously mentioned, the CounterCraft approach to deception deployment is based on zero impact to the production environment. The core of our technology is to provide high-fidelity, actionable threat intelligence in real time. Our fully instrumented deception environments and stealth communication channels allow monitoring of attacker behavior, detecting activity, providing enriched threat intel, and allowing the defender to manage the attack and protect their assets.
The evaluation focused on how each deception is capable of impacting the adversary. However, it did not evaluate how we could engage with the adversary or how we could collect threat intelligence (TTPs and IOCs) about them.
Assessing these different approaches is extremely valuable as you determine how to read through the results and determine which solution may fit your security program. For those of you who are familiar with the Enterprise Evaluation results, keep in mind that there are no Detection Categories for the deception results. Instead, Deception Categories were used that are based on the terminology found in the MITRE Engage framework. For these reasons, It is even more important than usual to assess the gaps you are trying to fill before you look to examine the results.
During the evaluation, CounterCraft Deception Director was able to impact the adversary when performing the following ATT&CK techniques:
OS Credential Dumping: LSASS Memory (T1003.001)
When the adversary uses Mimikatz for dumping the Windows credentials from memory we were able to inject other Windows credentials that could be used against some of our deception assets. These credentials could be used against our deception assets, so that we can collect any TTPs and IOCs from the adversary after using these credentials.
If the attacker were in one of our deception assets, we could easily detect this OS credential technique as there is a process injecting code into lsass.exe in order to dump the credentials from the lsass.exe memory. Our DeepSense agent is able to detect any process suspicious activity, and definitely injecting code into another process is one, as shown in the “defender” view of the same attack from our Deception Director.
File and Directory Discovery (T1083)
When the adversary was listing files in C:\ we added some new files and directories that could be attractive for the attacker. In this particular scenario, there was an ‘Instructions’ directory with a TXT file that includes some instructions to connect to one of our deception assets. There are so many options when adding files and directories: PowerShell scripts, TXT files, SSH private keys, RDP files, Office Documents, etc.
Network Share Discovery (T1135)
The adversary discovered a network share and enumerated files and directories of the mounted docshare share. Not only are we going to be able to detect the access to the shared folder, but we also include some interesting files and directories in it.
Remote Services: SMB/Windows Admin Shares (T1021.002)
The adversary also mounted one of the remote shares from a deception asset using the smbclient; they also tried to upload a binary and create a Windows service in order to get a psexec-like shell.
The following screenshot shows how this activity was captured when the adversary accessed the remote share in Sabre.
Our detection when the adversary tried to upload the psexec-like binary to Sabre.
File and Directory Discovery (T1083)
The adversary connected via RDP to the deception asset 10.0.1.12 and enumerated files and directories. File Explorer showed an AD_backup directory with employee and site geographic location data files. Once the adversary connected to the RDP in our deception asset Stamford, we can get a full telemetry of the actions performed by the adversary.
An event with the RDP authentication was also created:
Remote System Discovery (T1018)
The adversary queried for the domain name of 10.0.1.9 and terminal output reported sabre.internal.cloudapp.net, which is a deception asset. Sabre is exposing an RDP and a shared folder presenting the adversary with a path of least resistance for lateral movement to this new device.
Getting full telemetry from the deception assets also means that we can get all the PowerShell commands.
Remote System Discovery (T1018)
The adversary queried for the domain name of 10.0.1.11 and 10.0.1.12 and names were returned for deception assets, ubuntu-20.internal.cloudapp.net and stamford.internal.cloudapp.net, respectively.
The following screenshots show how this activity was reported in the CounterCraft platform:
Unsecured Credentials: Credentials In Files (T1552.001)
The adversary, after discovering an interesting host in a previous step, connected using RDP to 10.0.1.9 and collected a file on the desktop that contained active directory credentials. These AD credentials are valid credentials for a Domain Admin account, in a separate Windows Domain.
As we are getting full telemetry of the adversary actions, an event was generated with all the relevant information.
Data from Information Repositories: Code Repositories (T1213.003)
One of the ActiveLures that was present in all the endpoints was a direct access from Microsoft Edge to the vulnerable Gitlab server. The adversary then opened a browser and discovered it, but they didn’t try to login or exploit it.
An event containing information about the Gitlab access was created, but there was no further interaction with the Gitlab server.
Data from Local System (T1005)
The adversary viewed the command history in powershell and found a deception host name, apart from other interesting ActiveLures.
The corresponding event was also created
System Information Discovery (T1082)
The adversary queried for hostname and received a response for a deception asset.
Unsecured Credentials: Credentials In Files (T1552.001)
When the adversary connected via RDP into Utica as oscar and performed additional exploration, they found a saved RDP configuration pointing to a deception asset, 10.0.1.9 with its credentials.
The adversary activity was also captured
System Information Discovery (T1082)
The adversary queried for domain name and received a response for a decoy doman, DMEVALS-BACKUP. This is the AD domain that has a one-way trust relationship with the production AD, and it was also vulnerable to ZeroLogon.
Once the adversary was interested in the backup AD Domain, they also executed a number of discovery steps in order to gather information about the Windows domain.
Valid Accounts: Domain Accounts (T1078.002)
As part of additional investigations into a deception asset, the adversary used the net utility to add Toby as a user, terminal output reported the user was added successfully on the decoy host.
Summary
Even though the Deception Evaluation only focused on a limited subset of deception capabilities we think that it was a great opportunity to show how deception technology can help improve your detection capabilities by impacting the adversary with deception (the main use case of the evaluation), as well as for improving your collection of targeted threat intelligence (what we would like to see in future evaluations).
Our expectation was that the testing would be a collaborative process allowing all deception vendors who participated to define the testing strategy based on our experience combined with the experience of MITRE in the deception field. The testing process still requires work to achieve this, but we are keen to further our collaboration during future stages.
We hope that in the future the evaluation process will provide an independent analysis, on common ground from a known impartial source, to help identify how best to deploy deception. It should provide a benchmark to measure all deception platforms equally, highlighting strong areas and helping potential consumers identify which is the best tool for their specific deception deployment.
For more examples of how deception has prevented cyber attacks and to learn more about CounterCraft, contact us.
