Sysmon logs at scale analyzed with Splunk


1. Introduction

System Monitor (Sysmon) is a Windows system service and device driver that, once installed on a system, remains resident across system reboots to monitor and log system activity to the Windows event log. It provides detailed information about process creations, network connections, and changes to file creation time[1] [].
Sysmon log contains information which can be analyzed to detect modern attacks that bypass traditional detection tools. Mark Russinovich did a presentation at RSA 2016 about sysmon, its EventIDs and explained how sysmon can be used to detect malware.[2]  []

Sysmon Events should be sent to a Log Management System e.g Spunk, Elastic Search for analysis and there are few ways to do this e.g build-in WEF capability of Windows or an agent on endpoint like Splunk Universal Forwarder[3] [].

The main challenges in centralizing and analyzing sysmon logs are the handling of the volume and the filtering of the noise. This is very important in big networks (>10.000 hosts) especially when the licensing cost of the log management system is based on indexed volume like in Splunk.

Sysmon logs are a part of endpoints logs that must be analyzed and other sources include specific events from the security log, EMET log and PowerShell version 5 logs. It should be noted that how Sysmon data can be used and which detection rules can be developed depends on other security tools and policies that exist on a given network e.g A correlation rule can be developed to alert for malicious attachments that entered a network and an alarm raised by a network IDS without further information if finally at the endpoint the attachment was opened or not. A malicious attachment can be blocked by AV on Email gateway or  on email server or on  the endpoint or by user awareness.Full command line of Acrobat and Office executables in sysmon EventID 1 can be used to see if a malicious attachment was finally opened.

This post describe an approach taken to gather and analyze Sysmon logs  using Windows Eventlog Forwarding (WEF) and Splunk. The idea is to propose a cost-effective solution in order to implement Sysmon and PowerShell logs analysis using Splunk in large networks  and to bring them to the level 3 of the following Endpoint Logs Maturity Model.

Endpoint Logs Maturity Model
Level 1 Sysmon and Powershellv5 are not installed Valuable information for Incident Response and Detection is lost
Level 2 Sysmon and Powershellv5 are installed but logs remain on host During an incident response valuable info can be found on operational logs if not removed by the attacker
Level 3 An output-drive approach is implemented to import into Log Management system only the information (at field level) that is needed for creating valuable alerts. From security log only specific events are centralized A realistic approach
Level 4 All endpoint logs centralized in the  Log Management System Very expensive in large networks(no personal experience with  OSS solutions and their hidden costs)

2. Management of Volume and Filtering

An output-driven approach was used which mean that only the EventIDs and moreover the fields necessary to build a detection rule was sent to Splunk.

A big Sysmon Operational Log was created on the endpoints which can be used during forensics analysis of the host but filtering is applied both using Sysmon config.xml and the Splunk Heavy Forwarder before sysmon data arrives in Splunk.

Actually in order to conclude to a query result without noise a third step of filtering is applied during search time and this is specific to each network based on the software installed and the windows management practices. After an initial assessment of the volume generated in the first phase of the project the following EventsIDs were collected : 1, 3, 6, 8, 255

Since network connections Event ID 3 is very verbose only connections towards the proxy server that are not coming from standard browsers image are collected (see Appendix 1 for an example of Sysmon configuration file). The new Event ID 9 is also very verbose and its filtering, import into Splunk and analysis hasn’t been done yet. It is generated and stays locally on the endpoint. For an idea of the volume of logs generated see the graph below when a test was done to include EventID 9 into Splunk for 1 day.

2.1 Graphs with the volume generated

– Active Hosts

– Number of Events per day. Around 30 GB storage per month is needed on Splunk Indexer for this volume.


3.Playbook of Sysmon detection rules

Query No1:


  • Monitor browser and office  processes that have cmd or Powershell as child process on workstations. [Update]Other suspicious executables like  cscript.exe, wscript.exe,rundll32.exe etc should be added to the rule below


sourcetype=”WinEventLog:Microsoft-Windows-Sysmon/Operational” eventcode=1   parentimage=”C:\\Program Files (x86)\\Microsoft Office\\Office14\\WINWORD.EXE” OR parentimage=”C:\\Program Files (x86)\\Microsoft Office\\Office14\\OUTLOOK.EXE” OR parentimage=”C:\\Program Files (x86)\\Microsoft Office\\Office14\\EXCEL.EXE” OR parentimage=”C:\\Program Files (x86)\\Microsoft Office\\Office14\\POWERPNT.EXE” OR parentimage=”C:\\Program Files (x86)\\Internet Explorer\\iexplore.exe” OR parentimage=”C:\\Program Files\\Internet Explorer\\iexplore.exe” OR parentimage=”\”C:\\Program Files (x86)\\Mozilla Firefox\\firefox.exe\”” OR parentimage=”C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe” AND (image=”C:\\Windows\\SysWOW64\\WindowsPowerShell\\v1.0\\powershell.exe” OR image=”C:\\Windows\\System32\\cmd.exe” OR image=”C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe”)

The event below triggered an alert when a malicious excel attachment was opened and code executed to give a reverse shell to the attacker during a red team exercise.This was the only way to detect a simulation of a targeted attack.



  • Its very common in modern malware to exploit a vulnerability in Office macros or Flash player and start a command prompt. Microsoft discovered a 0-day based in Flash based on this behaviour[2] [].
  • Next Generation Endpoint Protection Products are using this behaviour :

 Query No2:


  • Detect long PowerShell commands that most probably will include obfuscated malicious code by length of command. A similar query searches for the presence of Invoke-Expression or IEX or Download strings

Query for long powershell command:

sourcetype=”WinEventLog:Microsoft-Windows-Sysmon/Operational” eventcode=1   image=”C:\\Windows\\SysWOW64\\WindowsPowerShell\\v1.0\\powershell.exe” OR   image=”C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe” | eval   c_length=len(commandline) | where c_length>1000

Search for suspicious strings:

sourcetype=”WinEventLog:Microsoft-Windows-Sysmon/Operational” eventcode=1   powershell.exe Invoke* OR IEX   OR Download* | table _time, computername, processid, image,commandline, parentprocessid,parentimage,parentcommandline

The following command is an example from a Red Team exercise that was detected by these rules :


Comment :

Since Powershell is more than powershell.exe [13] []even if Powershell.exe is blacklisted (not easy since it is normally used for administrative tasks) a more detailed analysis for PowerShell logs is needed. Good references for configuring PowerShell logging from FireEye [14] []and for analyzing PowerShell logs from Garbon Black[15] and from Australian Department of Defence[]

Query No3:


  • Detect rare thread injections to svchost

Query :

sourcetype=”WinEventLog:Microsoft-Windows-Sysmon/Operational” eventcode=8 targetimage=C:\\Windows\\System32\\svchost.exe | eval ppid=sourceimage+”;”+targetimage |rare ppid


Following the same idea similar alerts have been built to monitor thread injections to browser images and injections coming from sources in usual suspicious directories e.g user profile,programdata etc. All these results were put in a Splunk dashboard for daily review e.g

sourcetype=”WinEventLog:Microsoft-Windows-Sysmon/Operational” eventcode=8 NOT antivirus_image sourceimage=*temp* OR sourceimage=*Temp* OR sourceimage=C:\\ProgramData\\* OR sourceimage=C:\\Users\\* | eval ppid=sourceimage+”;”+targetimage |rare ppid

The result below was used to detect an adware that was installed silently on an IT admin PC without any alert by other systems.


 Query No4:


  • Monitor the execution  of Windows Management Instrumentation Command-line  on endpoints (WMIC.exe)


sourcetype=”WinEventLog:Microsoft-Windows-Sysmon/Operational” wmic.exe NOT management_use_1 … NOT management_use_2 NOT management_use_3

Query description:

  • Query gives an overview of Windows Management Instrumentation Command-line. Since wmi is used for windows management, filtering of normal commands needed in order from few thousands events to arrive to a manageable number of suspicious events. The numbers in the figures below refer  to a period of 1 month  that used to tune the query.



Following the same idea a Splunk lookup search was developed to search for normal windows binaries that used by attackers in the various stages of an attack [7][] including very dangerous executables signed by Microsoft like regasm.exe,InstallUtil.exe and mshta.exe [8][]

[Update]In the lookup table the regsvr32.exe is included so if its not blacklisted, attempts to bypass Applocker according to can be detected

Query No5:


Detect suspicious execution of rundll32.exe when the commandline contains path to User Profile and the parent commandline is browser


sourcetype=”WinEventLog:Microsoft-Windows-Sysmon/Operational” eventcode=1 NOT Yammer image=C:\\Windows\\System32\\rundll32.exe commandline=*C:\\Users\\* parentcommandline=”\”C:\\Program Files\\Internet Explorer\\IEXPLORE.EXE\””


This is also common malware behavior to drop a dll on user profile and run it using rundll32.exe. BlackEnergy APT dropper was using this technique [9][]

An Event like the one below worths investigation and there are not many of them on a daily basis.


Query No6:


  • Detect Malware based on the fact that it often uses short names(few letters)
  • Query:

sourcetype=”WinEventLog:Microsoft-Windows-Sysmon/Operational” eventcode = 1   | rex field=image “(?<filename>[^\\\]+)$” | eval file_length=len(filename) | where file_length < 6 | table image,filename,computername

Query No7:


  • Analyze the EventID 3 for non-browsers executables with abnormal number of connections to Internet

Example with rundll32.exe

  • Query:

sourcetype=”WinEventLog:Microsoft-Windows-Sysmon/Operational” eventcode = 3 rundll32.exe | timechart count(_raw) by computername

Example result set:


Comment: This query was useful to detect   malicious activity when rundll32 was used for https communication with C&C. The green spikes concern the infected workstations used for C&C communication. Rare executions of suspicious windows executables should also be analyzed


 In small to medium networks is relatively easy to gather and analyze Sysmon logs without big investments in hardware and software.

Detecting malicious activities that can bypass traditional prevention controls are possible with sysmon logs that can give deep insights into infections and intrusions.

A trade-off analysis that is specific to each network is needed in order to decide which sysmon info will remain on host and which will be forwarded to the Log Management System.

More Splunk rules can be developed based on known normal or abnormal behavior used in forensics as described in [10][],[11][]. Another interesting example for detecting system file manipulation is presented in  Detect System File Manipulation with Sysmon [12][]. This alert was useful to detect an insider that has by-passed security controls in order to get local admin privileges by replacing cmd.exe with sethc.exe.

One of the open questions regarding the collection of events from endpoints is the scaling of collector servers in networks with many thousands of endpoints.A pool of WEF collector servers can be created and few thousands of endpoints can be configured to send logs to the pool. This configuration is in place in a network with more than  500.000 hosts!

[Update] Microsoft published a very detailed article about WEF  6  days after this post 🙂 As it is mentioned in the TechNet post [] the rule is  10K  X 10 K.  Medium sized networks(up to 20K hosts) need just 2-3 VMs for WEC servers. Well done Microsoft team!

Rule based approach has its limitations[5] [] so machine learning features of Splunk version 6.4  should be tested on Sysmon logs

APPENDIX 1   Sysmon configuration File for version  3.21

<Sysmon schemaversion=”2.01″>
<!– Log all drivers except if the signature contains Microsoft or Windows –>
<DriverLoad onmatch=”include”>
<Signature condition=”contains”>microsoft</Signature>
<Signature condition=”contains”>windows</Signature>
<CreateRemoteThread onmatch=”exclude”/>
<FileCreateTime onmatch=”include”/>
<ImageLoad onmatch=”include”/>
<!– Log non browser connections to proxy and remove connections to internal web servers –>
<NetworkConnect onmatch=”exclude”>
<Image condition=”contains”>chrome.exe</Image>
<Image condition=”contains”>iexplore.exe</Image>
<Image condition=”contains”>firefox.exe</Image>
<Image condition=”contains”>OUTLOOK.EXE</Image>
<Image condition=”contains”>Skype.exe</Image>
<Image condition=”contains”>lync.exe</Image>
<DestinationHostname condition=”contains”>*internal domain servers </DestinationHostname>
<DestinationPort condition=”less than”>proxy port</DestinationPort>
<DestinationPort condition=”more than”>proxy port</DestinationPort>
<ProcessCreate onmatch=”exclude”/>
<ProcessTerminate onmatch=”include”/>


[1] Sysinternals Sysmon

[2] Mark Russinovich RSA 2016 Tracking Hackers on Your Network with Sysinternals Sysmon


[4] Microsoft Event Subscriptions: us/library/cc749183.aspx.

[5] Mark Russinovich RSA 2016     Machine Learning and the Cloud: Disrupting Threat Detection and Prevention

[6] Carbon Black Detects Locky :

[7] Windows Commands Abused by Attackers

[8] Mind The Gap

[9] Black Energy APT

[10] SANS know Normal Find Evil

[11] Using Sysmon to Enrich Security Onion’s Host-Level Capabilities

[12] Detect System File Manipulation with Sysmon

[13] Detecting Offensive PowerShell Attack Tools




Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s