A few years ago, a major debate ensued between the residents in my town and its police department. The city had just announced that police officers would not be dispatched in response to burglar alarms without verification that the alarms weren’t set off accidentally due to homeowner error. Instead, police would adopt a verified response system where officers would show up only when an adverse security incident could be verified by the alarm company, the homeowner, or a neighbor. The city’s police chief argued that 7,000 burglar alarms went off in the city in 2004—and only 66 were legitimate alerts. The police department was clearly dealing with alert fatigue (both physical and financial).
While this decision and its merits continue to be a topic of animated discussion my city’s government, alert fatigue in the case of IT security brings with it a whole host of issues. So, what causes alert fatigue in IT security and how can enterprises minimize its effects?
Security alerts and signal-to-noise ratio
Many enterprises have organically grown their toolkit of data center security products over time. The layers of security—from perimeter firewalls, IDS/IPS systems, anti-virus software, and endpoint security, to APT detection and SIEM solutions—generate a large number of system alerts. According to a recent article in CSO online, the average enterprise in North America deals with 10,000 security alerts on any given day. This number can be as high as 150,000 in the noisiest networks.
Often, security and operations teams find themselves overwhelmed by the number of alerts generated by different systems. In a large enterprise, entire full-time equivalents of security and risk personnel can be kept busy analyzing and responding to these alerts. The problem is that not all alerts are created equal and it is easy to look past the serious ones when confronted by a barrage from so many systems.
Not all alerts are created equal.
Crying wolf engenders complacency when the majority of notifications turn out to be non-events. For example, alert fatigue has been a major factor in some of the recent high-profile breaches and pernicious cyberattacks.
Reducing noise through security precision
Security systems should take a minimalist approach to alerts—issuing fewer, but accurate, alerts and making them count. Otherwise they run the risk of garnering nonchalance from the very teams that need to react to the alerts.
When security is precise, it can reduce the amount of “alert noise.” Most data center security solutions lack the context of the very entities they need to protect. In today’s dynamic and distributed computing environments, the context (i.e., properties, relationships, and operating environment) of application workloads changes with newly deployed applications, auto scaling of workloads, application migrations, and server decommissions. This makes it difficult for static, network-centric solutions to keep up and to generate valid or precise alerts that are actionable.
Without context, network security alerts received from systems are likely to be best guesses. Adaptive security uses context to continuously compute security policies in response to infrastructure and application changes. With adaptive security, both the specification and enforcement of security are decoupled from the underlying infrastructure. Context data is collected from individual application workloads starting from their inception and throughout their life cycle. Security policies are then dynamically computed using context and rules specifying permitted interactions between applications. The policies are then applied in real time at a granular level to individual application workloads, locking down interactions to only permitted flows. Security is built in to applications from right from the start and the enforcement of security follows a zero-trust model where communications not specified are simply not allowed. This approach reduces the number of alerts since policies are explicit and the criteria for policy violations are black and white.
Without context, network security alerts received from systems are likely to be best guesses.
Alert fatigue is a real problem for security teams. In terms of real costs, a recent study by the Ponemon institute says that false alerts cost organizations $1.27 million annually. It is time that enterprises took a hard look at their security architecture not only to protect their assets but also to get an accurate view of the protection that is actually delivered.