Little Known Features of Illumio ASP – Log Export to Amazon S3 Buckets

In this quick series, the Illumio product management team will highlight the lesser known (but no less powerful) features of Illumio ASP.

Amazon Simple Storage Service (“S3”) is an easy to use, cost-efficient, scalable data storage service that can be used to store and retrieve any type of data from anywhere on the Internet. Although it has many uses, it is primarily used for backup and recovery, disaster recovery, data archives, and cloud storage.

Typically, an organization creates an S3 bucket, which is similar to an internet-accessible file folder. On this S3 bucket, S3 access control policies can be applied to allow one organization to write data and other organizations to read data from the shared storage location. S3 buckets can be owned by one organization and be written/read by another organization. Additionally, long-lived, infrequently used data can be stored cheaply.

In addition to a web interface, S3 also provides an API for integration with other web services.

Vendors write integrations that can read/write S3 data. Illumio Secure Cloud, like other SaaS vendors, leverages Amazon S3 to write (deliver) logs to customers. Customers read (access) this data by connecting the S3 bucket to their SIEM or log analysis tools.

Commonly, customers create their own S3 bucket and provide their bucket name and account ID to Illumio. To make it easier for customers to set this up, we published a knowledge base article that includes a CloudFormation template. By loading this template into AWS, our customers can create the S3 buckets and apply the necessary Identity and Access Management (IAM) policies in a few easy steps.

Alternatively, customers can request Illumio to create and host the S3 bucket on their behalf and simply access the data from their side. (Current customers: see this documentation for the CloudFormation template and additional details.)

Once the S3 bucket is set up, Illumio’s SaaS Operations team will configure the provided account ID and bucket name to enable the delivery of logs. We will also create a couple of sub-folders in that S3 bucket for different types of data. Logs are batch delivered within 10 minutes of successful setup, and log data is batched by Illumio and written every 10 minutes.

Illumio Secure Cloud can provide two types of logs via Amazon S3: traffic flow summaries and audit events. Traffic flow summaries are records showing application-to-application communication in your data center, i.e., east-west traffic. Audit events are records of every change made on Illumio. These audit events include not only the traditional who/what/when/where data, but also notifications and the actual resource changes.

Both of these log types are structured messages in JSON format. Extensive documentation is available here.

SIEM vendors like Splunk and IBM QRadar provide pre-built integrations that seamlessly allow their products to utilize generic storage provided by S3.

• Splunk provides the Splunk Add-on for AWS.
• QRadar provides a log source type of Amazon AWS CloudTrail, which can be used as a gateway log source to pass data to other log sources.

We’ll be back with another edition of our “Little Known Features” soon, but in the meantime, message our product team at [email protected] for more information!