Nice conference in Athens @BsidesAth
Presentation on Sysmon and the pyramid of hell 🙂
Nice conference in Athens @BsidesAth
Presentation on Sysmon and the pyramid of hell 🙂
Sysmon Deployment Presentation
On 28/11/2016 Sysmon v5 was released delivering nice features. In this post I will describe some use cases for the new events. If you are new to Sysmon please start with the presentation from Marc Russinovich at RSA 2016.
Another nice presentation by @c_APT_ure at Botconf https://www.botconf.eu/2016/advanced-incident-detection-and-threat-hunting-using-sysmon-and-splunk/
I have stopped my description of Sysmon events in EventID 9.Before version 5
EventID 10 ProcessAccess was added.
You can use it to monitor access to lsass.exe. The challenge is to filter legitimate processes that generate this event.
In version 5 the 2 highlights are
1.EventID 11 which enables the monitoring of startup folder
2. Event IDs 12,13,14 for registry monitoring. These 3 events have the same descriptor in sysmon.xml (RegistryEvent) so a registry key that is put in this configuration section is monitored for all actions (add,delete,Value set,rename).
There is also EventID 15 which logs the hashes of attachments
As it is expected if in the configuration of sysmon you just include these events without filtering your log management system will be dead :-). Filtering is crucial for deplying sysmon at scale.
Event ID 7 (Image loaded) is the most noisy event. This event should be configured carefully as it generates a HUGE number of events.
On my test system,without filtering, within 5 minutes few thousands of events were generated.
If somebody wants to log only images loaded from user profile directory, clear some noise and also monitor what is loaded on lsass.exe the following section should be added to config.xml :
System Monitor (Sysmon) is a Windows system service and device driver that, once installed on a system, remains resident across system reboots to monitor and log system activity to the Windows event log. It provides detailed information about process creations, network connections, and changes to file creation time [https://goo.gl/1Gh9tt].
Sysmon log contains information which can be analyzed to detect modern attacks that bypass traditional detection tools. Mark Russinovich did a presentation at RSA 2016 about sysmon, its EventIDs and explained how sysmon can be used to detect malware. [https://goo.gl/lvwKMw]
Sysmon Events should be sent to a Log Management System e.g Spunk, Elastic Search for analysis and there are few ways to do this e.g build-in WEF capability of Windows or an agent on endpoint like Splunk Universal Forwarder [https://goo.gl/7zUWN1].
The main challenges in centralizing and analyzing sysmon logs are the handling of the volume and the filtering of the noise. This is very important in big networks (>10.000 hosts) especially when the licensing cost of the log management system is based on indexed volume like in Splunk.
Sysmon logs are a part of endpoints logs that must be analyzed and other sources include specific events from the security log, EMET log and PowerShell version 5 logs. It should be noted that how Sysmon data can be used and which detection rules can be developed depends on other security tools and policies that exist on a given network e.g A correlation rule can be developed to alert for malicious attachments that entered a network and an alarm raised by a network IDS without further information if finally at the endpoint the attachment was opened or not. A malicious attachment can be blocked by AV on Email gateway or on email server or on the endpoint or by user awareness.Full command line of Acrobat and Office executables in sysmon EventID 1 can be used to see if a malicious attachment was finally opened.
This post describe an approach taken to gather and analyze Sysmon logs using Windows Eventlog Forwarding (WEF) and Splunk. The idea is to propose a cost-effective solution in order to implement Sysmon and PowerShell logs analysis using Splunk in large networks and to bring them to the level 3 of the following Endpoint Logs Maturity Model.
|Endpoint Logs Maturity Model|
|Level 1||Sysmon and Powershellv5 are not installed||Valuable information for Incident Response and Detection is lost|
|Level 2||Sysmon and Powershellv5 are installed but logs remain on host||During an incident response valuable info can be found on operational logs if not removed by the attacker|
|Level 3||An output-drive approach is implemented to import into Log Management system only the information (at field level) that is needed for creating valuable alerts. From security log only specific events are centralized||A realistic approach|
|Level 4||All endpoint logs centralized in the Log Management System||Very expensive in large networks(no personal experience with OSS solutions and their hidden costs)|
An output-driven approach was used which mean that only the EventIDs and moreover the fields necessary to build a detection rule was sent to Splunk.
A big Sysmon Operational Log was created on the endpoints which can be used during forensics analysis of the host but filtering is applied both using Sysmon config.xml and the Splunk Heavy Forwarder before sysmon data arrives in Splunk.
Actually in order to conclude to a query result without noise a third step of filtering is applied during search time and this is specific to each network based on the software installed and the windows management practices. After an initial assessment of the volume generated in the first phase of the project the following EventsIDs were collected : 1, 3, 6, 8, 255
Since network connections Event ID 3 is very verbose only connections towards the proxy server that are not coming from standard browsers image are collected (see Appendix 1 for an example of Sysmon configuration file). The new Event ID 9 is also very verbose and its filtering, import into Splunk and analysis hasn’t been done yet. It is generated and stays locally on the endpoint. For an idea of the volume of logs generated see the graph below when a test was done to include EventID 9 into Splunk for 1 day.
– Active Hosts
– Number of Events per day. Around 30 GB storage per month is needed on Splunk Indexer for this volume.
sourcetype=”WinEventLog:Microsoft-Windows-Sysmon/Operational” eventcode=1 parentimage=”C:\\Program Files (x86)\\Microsoft Office\\Office14\\WINWORD.EXE” OR parentimage=”C:\\Program Files (x86)\\Microsoft Office\\Office14\\OUTLOOK.EXE” OR parentimage=”C:\\Program Files (x86)\\Microsoft Office\\Office14\\EXCEL.EXE” OR parentimage=”C:\\Program Files (x86)\\Microsoft Office\\Office14\\POWERPNT.EXE” OR parentimage=”C:\\Program Files (x86)\\Internet Explorer\\iexplore.exe” OR parentimage=”C:\\Program Files\\Internet Explorer\\iexplore.exe” OR parentimage=”\”C:\\Program Files (x86)\\Mozilla Firefox\\firefox.exe\”” OR parentimage=”C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe” AND (image=”C:\\Windows\\SysWOW64\\WindowsPowerShell\\v1.0\\powershell.exe” OR image=”C:\\Windows\\System32\\cmd.exe” OR image=”C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe”)
The event below triggered an alert when a malicious excel attachment was opened and code executed to give a reverse shell to the attacker during a red team exercise.This was the only way to detect a simulation of a targeted attack.
Query for long powershell command:
sourcetype=”WinEventLog:Microsoft-Windows-Sysmon/Operational” eventcode=1 image=”C:\\Windows\\SysWOW64\\WindowsPowerShell\\v1.0\\powershell.exe” OR image=”C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe” | eval c_length=len(commandline) | where c_length>1000
Search for suspicious strings:
sourcetype=”WinEventLog:Microsoft-Windows-Sysmon/Operational” eventcode=1 powershell.exe Invoke* OR IEX OR Download* | table _time, computername, processid, image,commandline, parentprocessid,parentimage,parentcommandline
The following command is an example from a Red Team exercise that was detected by these rules :
Since Powershell is more than powershell.exe  [http://www.adsecurity.org/?p=2604]even if Powershell.exe is blacklisted (not easy since it is normally used for administrative tasks) a more detailed analysis for PowerShell logs is needed. Good references for configuring PowerShell logging from FireEye  [https://goo.gl/zIEV5i]and for analyzing PowerShell logs from Garbon Black and from Australian Department of Defence[http://goo.gl/RrJGbX]
sourcetype=”WinEventLog:Microsoft-Windows-Sysmon/Operational” eventcode=8 targetimage=C:\\Windows\\System32\\svchost.exe | eval ppid=sourceimage+”;”+targetimage |rare ppid
Following the same idea similar alerts have been built to monitor thread injections to browser images and injections coming from sources in usual suspicious directories e.g user profile,programdata etc. All these results were put in a Splunk dashboard for daily review e.g
sourcetype=”WinEventLog:Microsoft-Windows-Sysmon/Operational” eventcode=8 NOT antivirus_image sourceimage=*temp* OR sourceimage=*Temp* OR sourceimage=C:\\ProgramData\\* OR sourceimage=C:\\Users\\* | eval ppid=sourceimage+”;”+targetimage |rare ppid
The result below was used to detect an adware that was installed silently on an IT admin PC without any alert by other systems.
sourcetype=”WinEventLog:Microsoft-Windows-Sysmon/Operational” wmic.exe NOT management_use_1 … NOT management_use_2 NOT management_use_3
Following the same idea a Splunk lookup search was developed to search for normal windows binaries that used by attackers in the various stages of an attack [http://goo.gl/TGD9A4] including very dangerous executables signed by Microsoft like regasm.exe,InstallUtil.exe and mshta.exe [http://goo.gl/reWl6E]
[Update]In the lookup table the regsvr32.exe is included so if its not blacklisted, attempts to bypass Applocker according to http://goo.gl/MMC28m can be detected
Detect suspicious execution of rundll32.exe when the commandline contains path to User Profile and the parent commandline is browser
sourcetype=”WinEventLog:Microsoft-Windows-Sysmon/Operational” eventcode=1 NOT Yammer image=C:\\Windows\\System32\\rundll32.exe commandline=*C:\\Users\\* parentcommandline=”\”C:\\Program Files\\Internet Explorer\\IEXPLORE.EXE\””
This is also common malware behavior to drop a dll on user profile and run it using rundll32.exe. BlackEnergy APT dropper was using this technique [https://goo.gl/MRZsq8]
An Event like the one below worths investigation and there are not many of them on a daily basis.
sourcetype=”WinEventLog:Microsoft-Windows-Sysmon/Operational” eventcode = 1 | rex field=image “(?<filename>[^\\\]+)$” | eval file_length=len(filename) | where file_length < 6 | table image,filename,computername
Example with rundll32.exe
sourcetype=”WinEventLog:Microsoft-Windows-Sysmon/Operational” eventcode = 3 rundll32.exe | timechart count(_raw) by computername
Example result set:
Comment: This query was useful to detect malicious activity when rundll32 was used for https communication with C&C. The green spikes concern the infected workstations used for C&C communication. Rare executions of suspicious windows executables should also be analyzed
In small to medium networks is relatively easy to gather and analyze Sysmon logs without big investments in hardware and software.
Detecting malicious activities that can bypass traditional prevention controls are possible with sysmon logs that can give deep insights into infections and intrusions.
A trade-off analysis that is specific to each network is needed in order to decide which sysmon info will remain on host and which will be forwarded to the Log Management System.
More Splunk rules can be developed based on known normal or abnormal behavior used in forensics as described in [https://goo.gl/1O3OTw],[https://goo.gl/BfNx9n]. Another interesting example for detecting system file manipulation is presented in Detect System File Manipulation with Sysmon [https://goo.gl/7WZVOr]. This alert was useful to detect an insider that has by-passed security controls in order to get local admin privileges by replacing cmd.exe with sethc.exe.
One of the open questions regarding the collection of events from endpoints is the scaling of collector servers in networks with many thousands of endpoints.A pool of WEF collector servers can be created and few thousands of endpoints can be configured to send logs to the pool. This configuration is in place in a network with more than 500.000 hosts!
[Update] Microsoft published a very detailed article about WEF 6 days after this post 🙂 As it is mentioned in the TechNet post [https://goo.gl/WR2NA7] the rule is 10K X 10 K. Medium sized networks(up to 20K hosts) need just 2-3 VMs for WEC servers. Well done Microsoft team!
Rule based approach has its limitations [https://goo.gl/f1rhfh] so machine learning features of Splunk version 6.4 should be tested on Sysmon logs
 Sysinternals Sysmon https://technet.microsoft.com/en-us/sysinternals/sysmon
 Mark Russinovich RSA 2016 Tracking Hackers on Your Network with Sysinternals Sysmon
 Microsoft Event Subscriptions: https://technet.microsoft.com/en- us/library/cc749183.aspx.
 Mark Russinovich RSA 2016 Machine Learning and the Cloud: Disrupting Threat Detection and Prevention
 Carbon Black Detects Locky : https://www.carbonblack.com/2016/03/04/tracking-locky-ransomware-using-carbon-black/
 Windows Commands Abused by Attackers http://blog.jpcert.or.jp/2016/01/windows-commands-abused-by-attackers.html
 Mind The Gap http://www.slideshare.net/infosecsmith/mind-the-gap-troopers-2016
 SANS know Normal Find Evil
 Using Sysmon to Enrich Security Onion’s Host-Level Capabilities
 Detect System File Manipulation with Sysmon
 Detecting Offensive PowerShell Attack Tools http://www.adsecurity.org/?p=2604