27 February 2024
Eito Tamura
Introduction
Endpoint security is something we actively test in our detection lab. Recently, I have been exploring Sysmon and its capabilities to better understand what it has to offer. In this blog post, I will walk through the basic implementation of Sysmon, share some tips, provide useful links, and also discuss strategies to ensure the integrity of Sysmon. Additionally, I'll illustrate the real-world application of Sysmon through two examples of actual attacks, showcasing its effectiveness and versatility.
What is Sysmon
Sysmon, short for System Monitor, is a Windows system service and device driver that monitors and logs system activity on a Windows host (there is a Linux version but it's not covered in this post). It is a lightweight monitoring tool and is a part of Windows Sysinternals suite. Sysmon event logs provide a more detailed view of system activity than what users can get out of the Windows events and typical endpoint sensors. This includes information on process creation and termination, network connections, file creation, and more. Sysmon is typically installed alongside other endpoint defence solutions, such as traditional antivirus or Endpoint Detection and Response (EDR), to add an additional layer of security.
EDR vs Sysmon
The primary distinction between EDR and Sysmon lies in their capabilities. EDR is equipped to detect malicious activities and respond proactively by implementing actions like blocking. In contrast, Sysmon serves as telemetry, lacking built-in detection and response features. To generate alerts for specific events, configuring a SIEM is necessary. Certain EDR solutions, such as Microsoft Defense for Endpoint and CrowdStrike, provide event reporting as well as detection and response capabilities enabling users to conduct threat hunting and incident analysis. While there are some overlaps with Sysmon events, leveraging Sysmon can help fill the gaps in telemetry.
Detection lab setup
Before diving into Sysmon implementation, I will cover our detection lab setup. While slightly off-topic, refining Sysmon configuration is ideal with a standalone central log aggregator, separate from your production SIEM. We've opted for the Hunting ELK (HELK) platform, utilising its capabilities for efficient log viewing and analysis at scale. It is an open-source hunting platform developed primarily for research, but due to its flexible design and core components, it can be deployed in larger environments. For detailed information on HELK, you can refer to the official documentation here.
The HELK built-in Sysmon dashboard provides a robust platform for searching and identifying generated events, aiding in the analysis of 'normal' events for the creation of effective filtering rules.
Sysmon installation
Installation of Sysmon is very simple, download the Sysmon zip file from the Sysinternals website. The zip file contains the following files:
There are three executable files. 'Sysmon.exe' contains both x86 and x64 architectures, 'Sysmon64.exe' is for x64 and 'Sysmon64a.exe' is for 64-bit ARM architecture. The command options are well explained on the official website. Some of the key commands are:
Install:
sysmon64 -i <configfile xml> (-accepteula)
Update configuration:
sysmon64 -c <configfile xml>
Load default configuration:
sysmon64 -c --
Uninstall:
sysmon64 -u [force]
Sysmon installation option
Sysmon executable file and the driver can be renamed, a tactic sometimes used to hide Sysmon and add complexity for adversaries attempting to identify security tools in use, although it does not guarantee foolproof security. For example, the below screenshot shows 'Sysmon64.exe' renamed to 'TZMon.exe', then using -d option to rename the driver to 'TZMonDrv'.
The description and display name of the Sysmon service can also be updated from the registry (shown below), as adversaries may search for Sysmon instances using this information.
This also applies to the altitude value for minifilter driver (the minifilter will be discussed later in this blog). The Sysmon minifilter altitude value for Sysmon, 385201, is frequently used by adversaries to identify Sysmon instances. The value can also be updated from the registry. However, it should be approached with caution, as if you use the same value twice in the system, one of the services would not start. The example below demonstrates the successful update of the altitude value in the registry, now changed to 385190. Before selecting a value, it's essential to check the list of allocated altitudes here to ensure it is not already in use by other services.
Sysmon events
At the time of writing, the Sysmon version is v15.2, featuring 29 different events and 1 Sysmon error event. Notable events include process creation, file manipulation, network connections, service state changes, process termination, driver and image loading, registry modifications, file stream creation, and more. Additionally, there are block actions for executable file creation and permanent deletion of executable files. The complete list is available on the Sysmon website, and Black Hills has detailed breakdown for each event here, so I won't be covering each event in this post.
Sysmon configuration/event filtering
The true strength of Sysmon lies in its powerful filtering capability. It can be configured to filter out known non-malicious events to suit your organisation to save valuable storage space, but careful implementation is crucial to avoid security gaps that adversaries could exploit.
The default configuration covers about 60% of its full MITRE ATT&CK matrix coverage, with notable exclusions for network connection and image loading (Event ID 3 and 7) as shown below.
The configuration file is in XML format and you can load the configuration with: sysmon64 -c config.xml. Upon executing the command, the binary blob of the configuration is stored in the registry as shown below. Although the path and SHA256 hash of the configuration file are also stored, Sysmon does not use them during startup. Deleting the configuration file can help prevent adversaries from accessing it and identifying potential security gaps.
You can print the configuration schema using the following command: sysmon64 -s. It's worth noting that the current schema version is 4.90 as shown below. Many Sysmon configuration examples can be found on the internet, but be cautious about using older schema versions, as they might lack configurations for newly added events.
Sysmon configuration
I won't cover the XML configuration structure and filtering options in this post, as they are well-explained on the Sysmon website. Furthermore, credit particularly goes to the following two individuals for creating valuable content and sharing their expertise in the community:
- Olaf Hartgon (FalconForce) - has published many Sysmon related blog posts and sysmon-modular. sysmon-modular is a highly customisable sysmon configuration generator. You can find "How to use sysmon-modular" video here. He also discussed different Sysmon implementation scenarios and how it can complement MDE in this post
- Carlos Perez (TrustedSec) - has Sysmon Community Guide and offers free online Sysmon training on TrustedSec YouTube channel. Also he shares useful Sysmon related tools: VSCode Extension and PSGumshoe
Sysmon extension for VSCode (link)
The VSCode extensions offers a series of snippets for helping in building a Microsoft Sysinternals Sysmon XML configuration. The extension is based on the 4.30 version of the Sysinternals Sysmon schema which is slightly dated. It features automatic closing of element tags for the filter fields.
PSGumshoe (Link)
PSGumshoe, a Windows PowerShell module, is designed for collecting OS and domain artifacts to facilitate live response, hunt, and forensics. It proves particularly useful for quickly identifying individual events to refine the event filter. For example, getting a list of Process Create (Event ID 1) events using the following command: Get-SysmonProcessCreateEvent | out-gridview
Sysmon configuration tips
Always test and verify your Sysmon configuration, especially before deploying it to production or upgrading to a newer version. To customise Sysmon for your organisation, my advise is to begin by installing it on representative endpoints, such as a server and a PC, with a reasonably verbose configuration, such as the 'verbose' or 'super verbose' configuration from sysmon-modular. Direct the output to a non-production log aggregator like ELK stack. Analyse what 'normal' events look like in your organisation. Then, use sysmon-modular to generate a configuration or use the pre-generated balanced configuration as a base and tune the configuration to suit your environment. If possible, simulate some adversary Tactics, Techniques, and Procedures (TTPs) and EDR evasion techniques to observe the triggered events. This will aid in refining the configuration and honing your skills as a defender.
Ensuring integrity of Sysmon
In recent times, various Sysmon bypass methods have been disclosed, raising concerns about the integrity of Sysmon, as seen with tools like SysmonEnte from Code White. Microsoft addressed this concern, and since version 15, Sysmon has implemented Protected Process Light (PPL) to protect its process from process injection-based attacks.
This section will explore online-disclosed methods to bypass Sysmon and list the triggered events. These events should be monitored to ensure Sysmon is not tampered with or blinded.
Process injection-based attacks
A common method in process injection-based attacks involves patching the EtwEventWrite API after hooking the Sysmon process. These types of attacks have been mitigated by the implementation of PPL. Some of the examples include:
Modifying sysmon registry
With administrative privilege, adversaries can modify the Sysmon registry where the configuration binary blob is stored, typically at Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\SysmonDrv\Parameters\Rules. This can be achieved using Windows system tools, such as:
- PowerShell - system command line tool, reference: Killing Sysmon Silently
- regedit - registry management tool
If the Sysmon configuration is updated using the Sysmon executable, it generates a Sysmon event ServiceConfigurationChange (event ID 16). However, this event does not trigger if the change is made directly to the registry. Instead it raises Registry value set event (event ID 13) with TargetObject pointing to the Sysmon Rules:
Bypassing FileBlockExecutable (link)
This article was published by Denis Nagayuk & Francisco Dominguez in 2022 with a Proof of Concept (PoC). The PoC comprises six different bypass methods to create an executable file, despite the block rule, and all of them still function. The PoC copies itself using each bypass and create executable files, i.e. Test[1-6].exe.
To uphold Sysmon's integrity, when configuring FileBlockExecutable, it's advisable to pair it with FileCreate under matched conditions. If this event triggers, raise an alert in your SIEM, signalling a potential attempt to bypass and create an executable
Unload Sysmon driver
When the Sysmon service starts up Sysmon driver registers as a Minifilter driver that is attached to volumes, enabling it to observe all actions taken by Windows APIs. If you want more detailed information about Sysmon and Minifilter drivers, you can refer to here. With administrative privilege, unloading the driver can disable file system-related events, affecting most event categories except for network-related ones. The following tools can be used for this purpose:
- Shhmon - searches for Sysmon driver and unload it using Windows APIs.
- Minifilter driver management utility (fltMC.exe) - is a system-supplied command line utility for common minifilter driver management operations. You can unload the driver using the following command:
fltMC.exe unload SysmonDrv
It generates an event from Microsoft-Windows-FilterManager with event message clearly stating that Sysmon driver is unloaded (System Event ID 1).
Uninstall/rename Sysmon
With administrative privileges, attackers can uninstall Sysmon or rename its executable, effectively stopping the service upon the next restart. For uninstallation, ProcessCreate (event ID 1) with a command line system64.exe -u force and Sysmon service state changed (event ID 4) with state "Stopped" are raised, as shown below:
Renaming the Sysmon executable, such as C:\Windows\Sysmon64.exe, leads to the Sysmon service failing to start, triggering a system error event logged by the Service Control Manager (System event ID 7000).
So which events should I monitor?
To summarise the above section, the following events should be monitored to ensure the integrity of Sysmon:
- Event ID 1: Process creation, command-line containing
sysmon64 -uandfltMC.exe unload SysmonDrv - Event ID 4: Sysmon service state changed
- Event ID 11: FileCreate - monitor executable file creation if you have Event ID 27: FileBlockExecutable enabled
- Event ID 12,13,14: Sysmon RegistryEvents
- Event ID 16: Sysmon Service Configuration Change
- Event ID 255: Config Monitoring Thread, Description: "Failed to send message to update configuration"
- System Event ID 7000: Service Control Manager, Description: "Sysmon service failing to start"
- System Event ID 1: File System Filter 'SysmonDrv' unloaded successfully.
Practical Examples
In this section, I'll simulate real-world attacks in our detection lab by deploying two types of implants that establish C2 connections and perform credential dumping. The target host is equipped with both EDR (one of the leading brands, which I won't disclose) and Sysmon, replicating a typical organisational setup. I've employed the 'default+' Sysmon configuration from sysmon-modular for this demonstration.
Example 1
For this example, I generated a default Havoc C2 payload in EXE format (called it faster.x64.exe) using the Havoc client. I added 500MB of null bytes data to discourage EDR scanning. Despite being an almost out-of-the-box payload, it managed to successfully evade EDR and dumped LSASS using built-in nanodump command as shown below:
Now, let's inspect the Sysmon logs to check whether it could have been detected by Sysmon telemetry with detection rules such as Sigma.
In the presented screenshots, two events at point 1 indicate the generation of events when the executable file was copied and executed. Point 2 showcases the DNS lookup and C2 connection from the faster.x64.exe process. At point 3, we observe the event related to the credential dump.
Additionally, the screenshot below provides a detailed view of the ProcessAccess event at point 3, revealing that faster.x64.exe accessed lsass.exe to perform credential dumping.
Leveraging the identified events, effective detection rules could be formulated for this attack. It depends on your organisation, but potential detection rules could be written for:
- Executable file creation with hash value checks
- An unfamiliar process engaging in DNS lookup and establishing a network connection
- An unknown process accessing the LSASS process
Example 2
For this example, a custom payload named faster2.x64.exe was employed, utilising D/Invoke as an evasion technique. This payload follows a sequence where it downloads shellcode from an S3 bucket, spawns msedge, injects the acquired shellcode into the msedge process, and concludes by creating a batch file to remove the original executable file, faster2.x64.exe. After establishing the C2 connection, a credential dump was performed. This simulated attack successfully evaded EDR detection and dumped credentials, as demonstrated in the Havoc client screenshot below:
Similar to example 1, two events at point 1 indicate the generation of events when the executable file was copied and executed. At point 2, the DNS of the S3 bucket was resolved, and it established a network connection to download the shellcode. A ProcessAccess event was generated at point 3, where faster2.x64.exe accessed the msedge process to inject the shellcode. Points 4 and 6 relate to the creation of a batch file, which gets executed to delete files. A DNS lookup by msedge was observed at point 5. No network connection event was observed, as it was filtered because the connection was made from msedge, which is considered 'normal'. At point 7, we observe the credential dump event.
Additionally, the screenshot below provides a detailed view of the ProcessAccess event at point 7, revealing that msedge accessed lsass.exe to perform credential dumping.
For this example, potential detection rules could be written for:
- Executable file creation with hash value checks
- An unfamiliar process accessing msedge process (potential injection attack)
- msedge process accessing the LSASS process
Summary
Sysmon is a valuable tool, offering telemetry that can be utilised to write detect rules for attacks potentially overlooked by EDR, and it can provide event logs for effective threat hunting and forensic analysis. Frequently underestimated by adversaries who often concentrate on EDR bypass, a successful implementation requires a thorough understanding of your environment, adversary TTPs, and Sysmon capabilities. Having detection rules for the mentioned Sysmon events and triggering alerts in your SIEM is crucial for maintaining its integrity.
Eito Tamura - Principal Consultant