Standardizing System Event Formats

I agree whole heartedly with the views and ideas expressed by Richard Mackey in his report for DarkReading and InformationWeek.  In most organizations that are capable, event monitoring exists as a set of “disjointed data streams” that the security team is responsible for reviewing.  There is little if any coordination of monitoring activities between departments, platforms or applications, never mind across competitive organizations.  This is polar opposite to the way that the criminals are operating today, and believe me, you and your competitors are in the same boat, facing the same sharks.

As Richard states, the complexity of security itself remains its greatest challenge. Even when companies collect detailed streams of data, there is little correlation of events across systems and platforms, and very little chance that those responsible for detecting attacks will recognize them before, or as, an incident transpires.  The deluge of data is often to great to distill actionable intelligence from in a timely manner.  We need what the bad guys have: a growing, coordinated, intelligence network.

Standardization is required for:

  • The kinds of log information being captured.
  • The mechanisms used to consolidate events.
  • The methods used to analyze and report the output.

If only we could adopt standards and practices to use event-correlation expertise and tools that are available more effectively.  Unfortunately, IDS, firewalls, and other systems collect different data from environment to environment and can’t be used easily for effective comparison or correlation.

Richard also mentions MITRE’s Common Event Expression (CEE) project and its elegant layered architecture that allows event-producing and event-processing systems communication flexibility.  The project’s logging recommendations provide a standard for encouraging a minimum common set of events to be made available from all systems.

Open Security Intelligence proposes standardizing the interfaces and protocols used to organize and manipulate event data.  OSI establishes SQL as the language to express queries against event data stores, and ODBC/JDBC as the standard interface for programs to gain access to the events collected across the enterprise.

To defend against well-coordinated attackers, companies need to cooperate and learn from each others’ experiences.  Sharing event data that may have led to compromises would help those involved to recognize the precursors and indicators of attacks, intrusion traffic patterns, and system and application events that, if correlated, would prompt investigations.

Thank you to InformationWeek, DarkReading, and to Richard for putting together and sharing a great report.  Add them to your reading list, as together they make a fantastic and timely set of information resources.