← All posts

Unit-testing the Splunk Processing Language

In my previous post Githubify the SOC, I declared my undying love for continuous integration and deployment capabilities applied to Detection Engineering. Now, let’s put the theory into practice! And maybe the best in class to inspire us is Microsoft Azure Sentinel.

Have you seen the Github Checks in place for adding rules into Azure/Azure-Sentinel alongside their Azure Sentinel Pipeline? 😍

Those are some serious CI/CD practices! How could we apply the same thing for our on-prem deployment?

Goals

In a “classic” software shop, developers rely on two levels of testing:

  1. Unit-tests, usually achieved in a few seconds. Coupled with basic tests for immediate feedback (similarly to the checks done by your IDE: syntax checks, undefined functions, etc.)
  2. Integration tests for more thorough scenarios, taking a few minutes to complete

In the context of Detection Engineering (i.e. Writing detection rules for a SIEM):

  1. Unit-tests would be checking that:
  • The syntax is valid
  • We are not using fields that do not exist
  • The styling guideline is respected
  • There is no performance trap
  1. While integration tests would check that:
  • We are correctly alerting for a True Positive
  • False-Positives are under control
  • We are not adding a hit to the platform performance
  • We are paying attention to delayed events

Today, this post will address only the unit-testing part, applied to Splunk.

Step 1, parsing Splunk’s Search Processing Language

Splunk’s Search Processing Language (aka SPL) is a very powerful and expressive query language. It is a very pleasant language to use, especially when you come from Elastic Query DSL.

The language is very permissive; almost everything is optional: field separators, usage of quotes, no types (string or integer, nobody cares), positions of arguments, etc.

Initially, in early 2020, I expected to have a lot of robust SPL parsers available as open source but… nope (well, there is a caveat here; follow me).

I found two projects:

splparser

Here I am trying salspaugh/splparser, developed in 2013, and to add confidence, its README states up-front “It is capable of parsing 66 of the most common approximately 132 SPL commands”.

Of course, its Python distribution is broken (quite expected for a project with its last commit 4 years ago) but I quickly fixed it. I ran it on our dataset of queries and it failed on the first query because of one unsupported function. I had 0️⃣ knowledge of PLY/LEX/YACC and adding an SPL command looked abyssal to me.

🤬

I rage-quit, thinking it was a 💩 project, and moved on. BIG MISTAKE retrospectively but 🤷‍♂️ sorry @salspaugh for having doubted you. More on that later.

splunk_antlr_spl

splunk_antlr_spl implements SPL using ANTLR4; it was clearly incomplete but the experience of modifying the ANTLR4 language was soooo nice that I could quickly hack it for my needs and kept trying while reading The Definitive ANTLR 4 Reference.

Eventually, I had to butcher most of the code to support enough SPL commands to parse our complete dataset. This fork lives in https://github.com/airbus-cert/splunk_antlr_spl

It works mostly fine, but it has one big problem: it is unbearably slow. This is not surprising as I am a total n00b in ANTLR4 (or even in the parsing field).

For example, parsing 338 rules takes 20 minutes. (Update: While I was writing those lines, and because I could not accept releasing such a crappy tool, I optimized my ANTLR4 syntax to make it faster.)

So it was not an option to have such slow tests in our CI/CD. This post is also an opportunity for me to do some kind of introspection and see if it was worth doing it; I was curious to see what commands were missing from salspaugh/splparser for it to be used on our dataset and… Only three tiny commands are missing 😢

On the other hand, the learning curve of ANTLR4 is so smooth that I had my first version in less than 5 days, and I wonder how long it would have taken me to learn Lex, Yacc, its PLY integration, and the time to implement these 3 commands and create a PR to salspaugh/splparser. 🤷

Plan B

When there is no perfect solution satisfying all constraints, it is time to work around this with hackish solutions.

And the grossest, but quickest, way to do some basic checks is to use regular expressions all the way around. As an example, I shared an example of our setup on Twitter:

As a takeaway, here is an extract from our code base: test_statically_spl.py

This Twitter discussion re-ignited my desire to level up our SPL parsing and I recently discovered a new project, kotlaluk/spl-parser.

This time, the project relies on an official Splunk feature, splunk btool can generate the search and datatypes BNF, no need to reinvent the wheel in fact!. Such an epiphany!

Thanks to Lukáš’s finding, there may be a way to leverage Splunk’s EBNF… 🤔

Step 2: Now what?

Once we have a parsing engine, we need to take a step back and write a high-level library that will abstract the details of the low-level parsing and expose only the “big picture”.

Stay tuned.