Organization trails with AWS Cloudtrail
During cloud security assessments one of the findings that often comes back is missing the ability to query the AWS audit logs, or get alerted on certain events that could indicate suspicious activity.
AWS Cloudtrail is the audit logs service, which captures all API calls within an AWS account. Cloudtrail is enabled by default, and without any configurations you will have audit logs for 90 days. This makes many believe you have it automatically out of the box when you create a new AWS (member) account. As you will find out the hard way during a security incident, relying on that standard 90 days and the GUI of AWS Cloudtrail is not enought to dig deep into the cloudtrail logs to get to the root cause of a breach or find out the prime question of who did what, when.
Most smaller organizations don’t have the need, nor the resources for a full blown SIEM solution. However basic notifications should be configured to get notified for events that should not happen or at least be investigated.
This blog post is part of a series that will describe one way to configure Cloudtrail so you have these audit logs in a centralized place where you can query them, and get notified when a suspicious event is happening in one of your AWS accounts.
AWS account setup
When we look at a very basic AWS account setup we have the following accounts.
Management account: Used to configure your initial IAM configuration (e.g. Identity Center), your organization Service Control Policies (SCP) and is the place where you have your consolidated billing. This is also the account where you configure Cloudtrail so it will be automatically deployed in each AWS region and account within your AWS Organization. This account should not have any resources and is only used to login and assume an IAM role within a member AWS account.
Development account: The account where development happens and can be experimented in. Granting engineers more permissive access rights in this account is often desired to promote experimentation within an organization. (Pay good attention to the cloud bill in these type of accounts).
Production account: This is the account where your production resources are running and is more restricted than the development account in terms of permissions and access. It’s also the account that is most interesting in terms of audit logs.
Log Archive account: A centralized account that stores the audit logs (cloudtrail and/or other internally used SaaS services), and should not have standard access. When someone accesses this account, an alert should be triggered.
But talk is cheap, let’s implement this!
Terraforming our implementation
Let start by creating a Cloudtrail terraform module that creates the organization trail for us. This will deploy a Cloudtrail in every account and in every region so you don’t miss anything. We chose to use KMS for encrypting the Cloudtrail logs and enable log insights which gives us more information if there is an increase for example in API calls.
Cloudtrail also offers the possibility to log data events (e.g. s3:GetObject, s3:PutObject, dynamodb:GetItem, etc), however these can become quite expensive when enabled considering the vast amount of calls that can happen to tables or buckets. Check with your compliance team to see if these need to be enabled or not. For now we don’t enable them.
# Location: example/modules/cloudtrail
# variables.tf
variable "cloudtrail_name" {
description = "Name of the organization trail"
type = string
}
variable "bucket_name" {
description = "Name of the S3 bucket where cloudtrail logs will be stored"
type = string
}
variable "kms_key_id" {
description = "KMS Key id that will encrypt the logs"
type = string
}
# main.tf
resource "aws_cloudtrail" "cloudtrail" {
name = var.cloudtrail_name
s3_bucket_name = var.bucket_name
include_global_service_events = true
enable_logging = true
is_organization_trail = true
is_multi_region_trail = true
kms_key_id = var.kms_key_id
event_selector {
read_write_type = "All"
include_management_events = true
}
insight_selector {
insight_type = "ApiCallRateInsight"
}
}
Okay, this module creates the cloudtrail as an organization trail and also multi region. The module takes some input parameters. One of them being the S3 bucket where Cloudtrail needs to send the logs. Let’s create that module next.
# Location: example/modules/cloudtrail-bucket
# variables.tf
variable "cloudtrail_name" {
description = "Name of the organization CloudTrail that will write to this bucket"
type = string
}
variable "kms_key_arn" {
description = "ARN of the KMS key used for encrypting the cloudtrail logs"
type = string
}
variable "log_retention_days" {
description = "Amount of days the cloudtrail logs need to be kept"
type = number
default = 365
}
# main.tf
resource "aws_s3_bucket" "cloudtrail_logs" {
bucket = var.cloudtrail_name
}
resource "aws_s3_bucket_public_access_block" "cloudtrail_logs" {
bucket = aws_s3_bucket.cloudtrail_logs.id
block_public_acls = true
block_public_policy = true
ignore_public_acls = true
restrict_public_buckets = true
}
# Optional - leave out if you want to save costs
resource "aws_s3_bucket_versioning" "cloudtrail_logs" {
bucket = aws_s3_bucket.cloudtrail_logs.id
versioning_configuration {
status = "Enabled"
}
}
resource "aws_s3_bucket_server_side_encryption_configuration" "cloudtrail_logs" {
bucket = aws_s3_bucket.cloudtrail_logs.id
rule {
apply_server_side_encryption_by_default {
sse_algorithm = "aws:kms"
kms_master_key_id = var.kms_key_arn
}
bucket_key_enabled = true
}
}
resource "aws_s3_bucket_lifecycle_configuration" "cloudtrail_logs" {
bucket = aws_s3_bucket.cloudtrail_logs.id
rule {
id = "cloudtrail_log_retention"
status = "Enabled"
# Apply rule to all objects in the bucket
filter {}
expiration {
days = var.log_retention_days
}
noncurrent_version_expiration {
noncurrent_days = 30
}
abort_incomplete_multipart_upload {
days_after_initiation = 7
}
}
}
resource "aws_s3_bucket_policy" "cloudtrail_logs" {
bucket = aws_s3_bucket.cloudtrail_logs.id
policy = jsonencode({
Version = "2012–10–17"
Statement = [
{
Sid = "AWSCloudTrailAclCheck"
Effect = "Allow"
Principal = {
Service = "cloudtrail.amazonaws.com"
}
Action = "s3:GetBucketAcl"
Resource = aws_s3_bucket.cloudtrail_logs.arn
},
{
Sid = "AWSCloudTrailWriteOrganization"
Effect = "Allow"
Principal = {
Service = "cloudtrail.amazonaws.com"}
Action = "s3:PutObject"
Resource = "${aws_s3_bucket.cloudtrail_logs.arn}/*"
},
{
Sid = "DenyInsecureConnections"
Effect = "Deny"
Principal = "*"
Action = "s3:*"
Resource = [
aws_s3_bucket.cloudtrail_logs.arn,
"${aws_s3_bucket.cloudtrail_logs.arn}/*"
]
Condition = {
Bool = {
"aws:SecureTransport" = "false"
}
}
}
]
})
}
# output.tf
output "bucket_name" {
value = aws_s3_bucket.cloudtrail_logs.bucket
}
output "bucket_arn" {
value = aws_s3_bucket.cloudtrail_logs.arn
}
Here we create the S3 bucket, we set all access to private and encrypt the logs in the bucket with a KMS key, automatically clean up the logs after a given period to stay compliant, which is often 1 year.
As mentioned in the code you can choose to not turn on versioning. This will be cheaper but in case you delete one of the logs by accident you won’t have a backup version anymore. It’s a trade off you need to decide for your organization. Finally we give the Cloudtrail service access to the bucket to put objects (gziped cloudtrail logs) in the bucket.
We are almost there, there is one module we need to make it all work. We chose KMS to encrypt our audit logs so we need to make a quick module for that, which we can use as input for the other two modules.
# Location: example/modules/kms-key
# variables.tf
variable "kms_key_alias" {
description = "Alias for the KMS key for easier recognition, will be prefixed with alias/"
type = string
}
variable "description" {
description = "Description of the KMS key"
type = string
}
variable "deletion_window_in_days" {
description = "Amount of days the key gets fully deleted after initializing the deletion of the key"
type = number
default = 30
}
variable "enable_key_rotation" {
description = "Enable automatic key rotation for the Key done by AWS"
type = bool
default = true
}
variable "key_policy" {
description = "IAM Key policy in json format"
}
# main.tf
resource "aws_kms_key" "kms_key" {
description = var.description
enable_key_rotation = var.enable_key_rotation
deletion_window_in_days = var.deletion_window_in_days
policy = jsonencode(var.key_policy)
}
resource "aws_kms_alias" "kms_key" {
name = "alias/${var.kms_key_alias}"
target_key_id = aws_kms_key.kms_key.key_id
}
# output.tf
output "kms_key_arn" {
value = aws_kms_alias.kms_key.arn
}
output "kms_key_id" {
value = aws_kms_alias.kms_key.id
}
Within this kms-key terraform module we create a KMS key that has an alias for easy referencing, and that is scheduled to be rotated to not exhaust the key and produce ciphertext with subtle patterns.
Now that we created the modules it’s time to actually use them. Let’s start in the Log archive account and create the S3 bucket.
# Location example/environments/log-archive/
# data.tf
data "aws_caller_identity" "current" {}
data "aws_region" "current" {}
# cloudtrail.tf
locals {
service_name = "<<COMPANY_NAME>>-cloudtrail-logs-<<RANDOM_X_CHARACTERS>>"
}
module "kms_key" {
source = "./modules/kms-key"
kms_key_alias = local.service_name
description = "Key used for encrypting the cloudtrail logs"
key_policy = {
Version = "2012–10–17"
Id = "DefaultKeyPolicy"
Statement = [
{
Action = "kms:*"
Effect = "Allow"
Principal = {
AWS = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:root"
}
Resource = "*"
},
{
Sid = "Allow CloudTrail to encrypt logs"
Effect = "Allow"
Principal = {
Service = ["cloudtrail.amazonaws.com"]
}
Action = [
"kms:Decrypt",
"kms:GenerateDataKey",
"kms:DescribeKey"
]
Resource = "*"
}
]
}
}
module "cloudtrail_bucket" {
source = "./modules/cloudtrail-bucket"
cloudtrail_name = local.service_name
kms_key_arn = module.kms_key.kms_key_arn
}
We are creating the KMS key here and use that as an input variable for the bucket module. This will create the bucket and encrypt all objects with the KMS key. Everyone with access to the Log Archive account can use the KMS key, hence the caution of alerting on when this account is accessed.
Now for the last step, to actually create the organization trail in all accounts, lets create the terraform code for the management account.
# Location: example/environments/management
# main.tf
locals {
service_name = "<<COMPANY_NAME>>-cloudtrail-logs-<<RANDOM_X_CHARACTERS>>"
}
module "cloudtrail" {
source = "./modules/cloudtrail"
cloudtrail_name = local.service_name
bucket_name = local.service_name
kms_key_id = "arn:aws:kms:<<REGION>>:<<ACCOUNT_ID>>:key/<<KEY_ID>>"
}
This will create the organization trail within your AWS organization and will enable it in every region and account that is part of the AWS organization in this moment. The moment a new AWS account is created or joins the organization, it will automatically create a cloudtrail in that account as well in every region that is enabled.
It will send all logs every 5 minutes to the S3 bucket specified as a GZIP file in the following structure:
S3 bucket/
└── AWSLogs/
└── organization_id/
├── account_id1/
│ ├── CloudTrail-Insight/
│ │ └── region/
│ │ └── year/
│ │ └── month/
│ │ └── day/
│ │ └── files.json.gz
│ └── CloudTrail/
│ └── region/
│ └── year/
│ └── month/
│ └── day/
│ └── files.json.gz
└── account_id2/
You can now run a terraform plan and **terraform apply **to actually deploy the needed resources, and will give you a centralized place where your Cloudtrail logs will reside. You can then query, forward them to an external SIEM solution or can be notified upon via the **s3:CreateObject **event.
In the next part of this series we will describe how to setup Athena in order to query these logs and present initial queries to investigate what is happening in your account(s).
If you made it all the way to the end, thanks for reading and happy querying your Cloudtrail logs. Stay secure!