This post got me thinking about a recent conversation I had with the CISO of a financial company. He commented on how quickly his team was able to instantiate a big data project with open source tools. He was of the view that such power could not be matched by IT security vendors who, in his opinion, charged too much money for demonstrably poorer performance.

The runaway success of the ELK stack has the DIY crowd energized. Why pay security vendors for specialist solutions when a “big data” project that we already have going on, based on this same stack, can work so much better, the thinking goes. And it’s free, of course.

What we know from 10+ years of rooting around in the security world is that solving the platform problem gets you about a quarter of the way to the security outcome. After that comes detection content, and then the skills to work the data plus the process discipline. Put another way, “Getting data into the data lake, easy. Getting value out of the data in the lake, not so much.”

In 2017, it is easier than ever to spin up an instance of ELK on premises or in the cloud and presume that success is at hand just because the platform is now available. Try using generic tools to solve the security problem and you will soon discover why security vendors have spent so much time writing rules and why service providers spend so much effort on process/procedure and recruitment/training.