Randy Franklin Smith compares methods for detecting malicious activity from logs including monitoring for high impact changes, setting up tripwires and anomalous changes in activity levels.

Security standards and auditors make much of reviewing logs for malicious activity. I am frequently asked what event signatures are indicative of intrusions: “What are the top Event IDs for intrusion detection?” Ah, if it was only as easy as the movies make it, where the protagonist furiously defends the network while a computer voice stridently calls out “Intruder! Intruder!”

In the real world, a system under attack takes immediate preventive steps instead of simply logging the intrusion. For example: an outsider attacking the system to brute force an account password through repeated logon attempts will find the account locked out by the system. But what about an authentic user accessing authorized data? If a disgruntled employee copies sensitive data to a flash drive, it looks like any other day of normal usage to the operating system. If the system is logging anything at all, there will just be an audit trail of successful access attempts.

It’s not impossible to use logs to detect some intrusions and malicious activity but the first and most important return on investment from logging is an audit trail that can be used to detect high impact changes in security stance, investigate incidents, perform impact analysis, take action against offenders, and to learn how to prevent repeat attacks in the future. Moreover, a properly deployed audit system serves as an effective deterrent control against insider abuse. All of this can be accomplished by enabling logging on monitored systems and implementing a log management solution with high integrity deployment characteristics.

But to get more than change detection and audit trail usage from your logs, you must go upstream from the logging function and get the cooperation from system administrators to configure and operate systems in such a way that certain foreseeable types of malicious activity will recognizable to your log management solution.

Take the security log integrity in Windows: it provides strong protection against tampering with the security log. Unless the operating system is down and you have physical access to the system, you have to be an administrator to erase the security log. (It’s very difficult even with administrator authority to modify event records in the security log – you need a program like the old WinZapper.) Windows logs a specific event ID (517 on Win2003 and 1102 on Win2008) whenever the log is cleared, so it makes sense to generate an alert whenever security logs are cleared as a deterrent to rogue administrators trying to cover their tracks or intruders doing the same. But if your administrators are in the habit of clearing the log when diagnosing system problems, your alert rule will generate false positives. Therefore, in addition to reactive log management you must proactively get administrators to comply with a policy never to clear security logs and allow old events to be deleted as records are naturally purged.

While the previous case demonstrated an example of procedural measures implemented upstream from the log management process, detecting the spread of file-based malware requires you to work with system administrators to set up system objects ahead of time for the purpose of helping the log management solution distinguish between normal file access and malicious activity. Here’s the premise: you want to detect malware that has made it through your preventive controls like antivirus. Most malware either spreads by injecting itself into files of affected file types (i.e. Word documents or image files) or looks for files that it can send back to its nefarious operator.

So, we set up the equivalent of a trip wire for unsuspecting malware or outside intruders to trigger. (In this example I’ll stick with the Windows and Active Directory environment but the concepts would be applicable to other platforms.) To do this, we work with various system administrators to create “honeypot” folders on servers throughout the network. In these folders we put a collection of files with all the common file types including Office documents, image files (e.g. JPG, GIF), PDFs and other files that have been targeted by malware. Ideally we name these honeypot folders with just one or a few easily recognizable folder names. We grant everyone access to these folders and enable file system auditing so no matter what user is the victim, the malware will have access to the folder in order to trigger the tripwire. If we want to detect malware that is spreading itself or simply corrupting data, we limit file system auditing to WriteData and AppendData. On the other hand if we also hope to detect malware that is stealing data we would also enable auditing of ReadData. Then back at our log management solution we would enable alert rules when file system audit events (event ID 560 on Windows 2003 and 4663 on Windows 2008) arrive which identify one of our honeypot folders as having activity. To prevent false positives some training and reminders may be necessary for end-users so that they understand these folders (especially those on file server shares) have nothing useful for them and are there for “system” purposes.

These two examples demonstrate you can take a foreseeable type of malicious activity and implement procedures or system objects to make it possible to detect activity that would normally be ignored because it blends with everyday, legitimate usage. The downside is that cooperation is required from people who are upstream to the log management solution. Depending on the capabilities of your log management solution, you may be able to discover malicious activity without implementing upstream tripwires using a final method: anomalous changes in activity levels.

Some types of malicious activity cause a spike in audit events that can be detected if your log management solution has sufficient real-time analysis capabilities. My favorite example is the disgruntled insider who intends to post thousands of documents to a site like Wikileaks. The user likely already has access to the sensitive documents in question so there may be little or no failed access attempts to alert you to what is happening. Instead, file system auditing, if enabled, will generate successful read attempt events for each file accessed. If the disgruntled employee has been access a handful of documents daily, the sudden access of multiple files will trigger a feature-rich log management solution that keeps a rolling average of normal Read events on sensitive folders for each user and detect a spike in activity. At that point, there is sufficient cause to generate and alert and devote some security staff time to determining the cause. With anomalous activity level detection, some thresholds can be generalized and baked into the log management solution but others would need at least some amount of customization by the organization such defining folders or servers that are considered sensitive.

As you can see, detecting intrusions and abuse with logs is possible but it requires a variety of analysis techniques, the cooperation of system administrators upstream from your log management system, and your log management solution’s ability to analyze large amounts of data to detect sudden changes in activity levels.