System Monitor (Sysmon) is a Windows system service and device driver that, once installed on a system, remains resident across system reboots to monitor and log system activity to the Windows event log. It provides detailed information about process creations, network connections, and changes to file creation time [https://goo.gl/1Gh9tt].
Sysmon log contains information which can be analyzed to detect modern attacks that bypass traditional detection tools. Mark Russinovich did a presentation at RSA 2016 about sysmon, its EventIDs and explained how sysmon can be used to detect malware. [https://goo.gl/lvwKMw]
Sysmon Events should be sent to a Log Management System e.g Spunk, Elastic Search for analysis and there are few ways to do this e.g build-in WEF capability of Windows or an agent on endpoint like Splunk Universal Forwarder [https://goo.gl/7zUWN1].
The main challenges in centralizing and analyzing sysmon logs are the handling of the volume and the filtering of the noise. This is very important in big networks (>10.000 hosts) especially when the licensing cost of the log management system is based on indexed volume like in Splunk.
Sysmon logs are a part of endpoints logs that must be analyzed and other sources include specific events from the security log, EMET log and PowerShell version 5 logs. It should be noted that how Sysmon data can be used and which detection rules can be developed depends on other security tools and policies that exist on a given network e.g A correlation rule can be developed to alert for malicious attachments that entered a network and an alarm raised by a network IDS without further information if finally at the endpoint the attachment was opened or not. A malicious attachment can be blocked by AV on Email gateway or on email server or on the endpoint or by user awareness.Full command line of Acrobat and Office executables in sysmon EventID 1 can be used to see if a malicious attachment was finally opened.
This post describe an approach taken to gather and analyze Sysmon logs using Windows Eventlog Forwarding (WEF) and Splunk. The idea is to propose a cost-effective solution in order to implement Sysmon and PowerShell logs analysis using Splunk in large networks and to bring them to the level 3 of the following Endpoint Logs Maturity Model.
|Endpoint Logs Maturity Model|
|Level 1||Sysmon and Powershellv5 are not installed||Valuable information for Incident Response and Detection is lost|
|Level 2||Sysmon and Powershellv5 are installed but logs remain on host||During an incident response valuable info can be found on operational logs if not removed by the attacker|
|Level 3||An output-drive approach is implemented to import into Log Management system only the information (at field level) that is needed for creating valuable alerts. From security log only specific events are centralized||A realistic approach|
|Level 4||All endpoint logs centralized in the Log Management System||Very expensive in large networks(no personal experience with OSS solutions and their hidden costs)|
2. Management of Volume and Filtering
An output-driven approach was used which mean that only the EventIDs and moreover the fields necessary to build a detection rule was sent to Splunk.
A big Sysmon Operational Log was created on the endpoints which can be used during forensics analysis of the host but filtering is applied both using Sysmon config.xml and the Splunk Heavy Forwarder before sysmon data arrives in Splunk.
Actually in order to conclude to a query result without noise a third step of filtering is applied during search time and this is specific to each network based on the software installed and the windows management practices. After an initial assessment of the volume generated in the first phase of the project the following EventsIDs were collected : 1, 3, 6, 8, 255
Since network connections Event ID 3 is very verbose only connections towards the proxy server that are not coming from standard browsers image are collected (see Appendix 1 for an example of Sysmon configuration file). The new Event ID 9 is also very verbose and its filtering, import into Splunk and analysis hasn’t been done yet. It is generated and stays locally on the endpoint. For an idea of the volume of logs generated see the graph below when a test was done to include EventID 9 into Splunk for 1 day.
2.1 Graphs with the volume generated
– Active Hosts
– Number of Events per day. Around 30 GB storage per month is needed on Splunk Indexer for this volume.
3.Playbook of Sysmon detection rules
- Monitor browser and office processes that have cmd or Powershell as child process on workstations. [Update]Other suspicious executables like cscript.exe, wscript.exe,rundll32.exe etc should be added to the rule below
sourcetype=”WinEventLog:Microsoft-Windows-Sysmon/Operational” eventcode=1 parentimage=”C:\\Program Files (x86)\\Microsoft Office\\Office14\\WINWORD.EXE” OR parentimage=”C:\\Program Files (x86)\\Microsoft Office\\Office14\\OUTLOOK.EXE” OR parentimage=”C:\\Program Files (x86)\\Microsoft Office\\Office14\\EXCEL.EXE” OR parentimage=”C:\\Program Files (x86)\\Microsoft Office\\Office14\\POWERPNT.EXE” OR parentimage=”C:\\Program Files (x86)\\Internet Explorer\\iexplore.exe” OR parentimage=”C:\\Program Files\\Internet Explorer\\iexplore.exe” OR parentimage=”\”C:\\Program Files (x86)\\Mozilla Firefox\\firefox.exe\”” OR parentimage=”C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe” AND (image=”C:\\Windows\\SysWOW64\\WindowsPowerShell\\v1.0\\powershell.exe” OR image=”C:\\Windows\\System32\\cmd.exe” OR image=”C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe”)
The event below triggered an alert when a malicious excel attachment was opened and code executed to give a reverse shell to the attacker during a red team exercise.This was the only way to detect a simulation of a targeted attack.
- Its very common in modern malware to exploit a vulnerability in Office macros or Flash player and start a command prompt. Microsoft discovered a 0-day based in Flash based on this behaviour [https://goo.gl/lvwKMw].
- Next Generation Endpoint Protection Products are using this behaviour :
- Carbon Black detects Locky  [https://goo.gl/LRwHPf]
- Detect long PowerShell commands that most probably will include obfuscated malicious code by length of command. A similar query searches for the presence of Invoke-Expression or IEX or Download strings
Query for long powershell command:
sourcetype=”WinEventLog:Microsoft-Windows-Sysmon/Operational” eventcode=1 image=”C:\\Windows\\SysWOW64\\WindowsPowerShell\\v1.0\\powershell.exe” OR image=”C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe” | eval c_length=len(commandline) | where c_length>1000
Search for suspicious strings:
sourcetype=”WinEventLog:Microsoft-Windows-Sysmon/Operational” eventcode=1 powershell.exe Invoke* OR IEX OR Download* | table _time, computername, processid, image,commandline, parentprocessid,parentimage,parentcommandline
The following command is an example from a Red Team exercise that was detected by these rules :
Since Powershell is more than powershell.exe  [http://www.adsecurity.org/?p=2604]even if Powershell.exe is blacklisted (not easy since it is normally used for administrative tasks) a more detailed analysis for PowerShell logs is needed. Good references for configuring PowerShell logging from FireEye  [https://goo.gl/zIEV5i]and for analyzing PowerShell logs from Garbon Black and from Australian Department of Defence[http://goo.gl/RrJGbX]
- Detect rare thread injections to svchost
sourcetype=”WinEventLog:Microsoft-Windows-Sysmon/Operational” eventcode=8 targetimage=C:\\Windows\\System32\\svchost.exe | eval ppid=sourceimage+”;”+targetimage |rare ppid
Following the same idea similar alerts have been built to monitor thread injections to browser images and injections coming from sources in usual suspicious directories e.g user profile,programdata etc. All these results were put in a Splunk dashboard for daily review e.g
sourcetype=”WinEventLog:Microsoft-Windows-Sysmon/Operational” eventcode=8 NOT antivirus_image sourceimage=*temp* OR sourceimage=*Temp* OR sourceimage=C:\\ProgramData\\* OR sourceimage=C:\\Users\\* | eval ppid=sourceimage+”;”+targetimage |rare ppid
The result below was used to detect an adware that was installed silently on an IT admin PC without any alert by other systems.
- Monitor the execution of Windows Management Instrumentation Command-line on endpoints (WMIC.exe)
sourcetype=”WinEventLog:Microsoft-Windows-Sysmon/Operational” wmic.exe NOT management_use_1 … NOT management_use_2 NOT management_use_3
- Query gives an overview of Windows Management Instrumentation Command-line. Since wmi is used for windows management, filtering of normal commands needed in order from few thousands events to arrive to a manageable number of suspicious events. The numbers in the figures below refer to a period of 1 month that used to tune the query.
Following the same idea a Splunk lookup search was developed to search for normal windows binaries that used by attackers in the various stages of an attack [http://goo.gl/TGD9A4] including very dangerous executables signed by Microsoft like regasm.exe,InstallUtil.exe and mshta.exe [http://goo.gl/reWl6E]
[Update]In the lookup table the regsvr32.exe is included so if its not blacklisted, attempts to bypass Applocker according to http://goo.gl/MMC28m can be detected
Detect suspicious execution of rundll32.exe when the commandline contains path to User Profile and the parent commandline is browser
sourcetype=”WinEventLog:Microsoft-Windows-Sysmon/Operational” eventcode=1 NOT Yammer image=C:\\Windows\\System32\\rundll32.exe commandline=*C:\\Users\\* parentcommandline=”\”C:\\Program Files\\Internet Explorer\\IEXPLORE.EXE\””
This is also common malware behavior to drop a dll on user profile and run it using rundll32.exe. BlackEnergy APT dropper was using this technique [https://goo.gl/MRZsq8]
An Event like the one below worths investigation and there are not many of them on a daily basis.
- Detect Malware based on the fact that it often uses short names(few letters)
sourcetype=”WinEventLog:Microsoft-Windows-Sysmon/Operational” eventcode = 1 | rex field=image “(?<filename>[^\\\]+)$” | eval file_length=len(filename) | where file_length < 6 | table image,filename,computername
- Analyze the EventID 3 for non-browsers executables with abnormal number of connections to Internet
Example with rundll32.exe
sourcetype=”WinEventLog:Microsoft-Windows-Sysmon/Operational” eventcode = 3 rundll32.exe | timechart count(_raw) by computername
Example result set:
Comment: This query was useful to detect malicious activity when rundll32 was used for https communication with C&C. The green spikes concern the infected workstations used for C&C communication. Rare executions of suspicious windows executables should also be analyzed
In small to medium networks is relatively easy to gather and analyze Sysmon logs without big investments in hardware and software.
Detecting malicious activities that can bypass traditional prevention controls are possible with sysmon logs that can give deep insights into infections and intrusions.
A trade-off analysis that is specific to each network is needed in order to decide which sysmon info will remain on host and which will be forwarded to the Log Management System.
More Splunk rules can be developed based on known normal or abnormal behavior used in forensics as described in [https://goo.gl/1O3OTw],[https://goo.gl/BfNx9n]. Another interesting example for detecting system file manipulation is presented in Detect System File Manipulation with Sysmon [https://goo.gl/7WZVOr]. This alert was useful to detect an insider that has by-passed security controls in order to get local admin privileges by replacing cmd.exe with sethc.exe.
One of the open questions regarding the collection of events from endpoints is the scaling of collector servers in networks with many thousands of endpoints.A pool of WEF collector servers can be created and few thousands of endpoints can be configured to send logs to the pool. This configuration is in place in a network with more than 500.000 hosts!
[Update] Microsoft published a very detailed article about WEF 6 days after this post 🙂 As it is mentioned in the TechNet post [https://goo.gl/WR2NA7] the rule is 10K X 10 K. Medium sized networks(up to 20K hosts) need just 2-3 VMs for WEC servers. Well done Microsoft team!
Rule based approach has its limitations [https://goo.gl/f1rhfh] so machine learning features of Splunk version 6.4 should be tested on Sysmon logs
APPENDIX 1 Sysmon configuration File for version 3.21
<!– Log all drivers except if the signature contains Microsoft or Windows –>
<!– Log non browser connections to proxy and remove connections to internal web servers –>
<DestinationHostname condition=”contains”>*internal domain servers </DestinationHostname>
<DestinationPort condition=”less than”>proxy port</DestinationPort>
<DestinationPort condition=”more than”>proxy port</DestinationPort>
 Sysinternals Sysmon https://technet.microsoft.com/en-us/sysinternals/sysmon
 Mark Russinovich RSA 2016 Tracking Hackers on Your Network with Sysinternals Sysmon
 Microsoft Event Subscriptions: https://technet.microsoft.com/en- us/library/cc749183.aspx.
 Mark Russinovich RSA 2016 Machine Learning and the Cloud: Disrupting Threat Detection and Prevention
 Carbon Black Detects Locky : https://www.carbonblack.com/2016/03/04/tracking-locky-ransomware-using-carbon-black/
 Windows Commands Abused by Attackers http://blog.jpcert.or.jp/2016/01/windows-commands-abused-by-attackers.html
 Mind The Gap http://www.slideshare.net/infosecsmith/mind-the-gap-troopers-2016
 SANS know Normal Find Evil
 Using Sysmon to Enrich Security Onion’s Host-Level Capabilities
 Detect System File Manipulation with Sysmon
 Detecting Offensive PowerShell Attack Tools http://www.adsecurity.org/?p=2604