Problem: Malicious attacker get privileged access to your AWS account and destroying your production infrastructure in a matter of seconds.
In case of cloud based infrastructures you can't rely on classic SIEM solutions - by the time your SIEM will detect attack, your infrastructure will be gone. We need a "near real time" way to detect and mitigate the attack.
Attack scenario: using compromised aws api key (no MFA) and CLI/SDK to perform destructive actions.
Attack detection and mitigation strategy:
All destructive actions start with EC2 instance termination. To prevent such scenario, you should always have "TerminationProtection" feature enabled on your production instances. Based on this, attacker must disable termination protection before demolishing your environment. For the PoC, I will use disabling "TerminationProtection" event as a attack detector (sure thing, real attack detection is a way more complicated process).
Starting point: AWS api key with admin policy attached, all production instances protected using AWS Termination Protection.
Possible solutions and attack mitigation delays:
SIEM:
CloudWatch Logs and alarms:
CloudWatch Subscriptions:
CloudWatch Events:
Implementation:
Design:Based on the implementation scenarios shown above and their performance (tested during PoC) - the fastest way is to leverage AWS CloudWatch events and trigger a lambda function.
Speaking AWS technical language we need:
- choose what type of AWS event we are looking for. Based on the our attack detection strategy, we are looking for ec2 call: modify-instance-attribute. Using exactly this API call you can enable/disable TerminationProtection. So let's look for "AWS API Call Events" type of the CloudWatch events.
- Create: event rule : "match incoming events and route them to one or more targets for processing" in our case target is a Lambda function.
- Create all required policies and Lambda function role in IAM.
- Build the Lambda function itself.
Setting-up event rule:
I was using following event pattern inside the CloudWatch Event rule:
{ "detail-type": [ "AWS API Call via CloudTrail" ], "detail": { "eventSource": [ "ec2.amazonaws.com" ], "eventName": [ "ModifyInstanceAttribute" ] } }
Getting sample event:
To start writing our event-detection-mitigation lambda function we need to get example of the AWS events for the API call we are monitoring.
We can achieve this with the following simple Lambda function:
import json
def lambda_handler(event, context):
print event
or, if you need nice formatted json to test you lambda function offline:
import json
def lambda_handler(event, context):
print json.dumps(event, indent=4, sort_keys=False)
You will find output of your lambda function (result of the print statement) in appropriate (naming as your lambda function) CloudWatch Log Stream
Challenges with event format:
During first tests, I've found that Amazon AWS not following any JSON contract (defined format) even for the same 1 API call. Making the same API call in the 3 different ways produced 3 different event formats:
Disabling TerminationProtection from GUI with MFA:
{u'account': u'150905', u'region': u'eu-west-1', u'detail': {u'eventVersion': u'1.05', u'eventID': u'b3c4d3b4-353e-44bf-8973-37abccd085b5', u'eventTime': u'2016-10-14T17:31:37Z', u'requestParameters': {u'instanceId': u'i-d8916d57', u'disableApiTermination': {u'value': False}}, u'eventType': u'AwsApiCall', u'responseElements': {u'_return': True}, u'awsRegion': u'eu-west-1', u'eventName': u'ModifyInstanceAttribute', u'userIdentity': {u'userName': u'ihork', u'principalId': u'AIDAI3UNW', u'accessKeyId': u'ASIAIN2', u'invokedBy': u'signin.amazonaws.com', u'sessionContext': {u'attributes': {u'creationDate': u'2016-10-14T16:48:44Z', u'mfaAuthenticated': u'true'}}, u'type': u'IAMUser', u'arn': u'arn:aws:iam::150905:user/igor', u'accountId': u'150905'}, u'eventSource': u'ec2.amazonaws.com', u'requestID': u'e7b585e-af38-49d0-88a8-979ef5052f', u'userAgent': u'signin.amazonaws.com', u'sourceIPAddress': u'174.231.5.2'}, u'detail-type': u'AWS API Call via CloudTrail', u'source': u'aws.ec2', u'version': u'0', u'time': u'2016-10-14T17:31:37Z', u'id': u'55084ea-e4bc-45e6-a7a6-0c8e7d16b32', u'resources': []}
Disabling TerminationProtection from aws cli (no MFA):
command:
$ aws ec2 modify-instance-attribute --no-disable-api-termination --instance-id i-378579b8
event:
{u'account': u'150905', u'region': u'eu-west-1', u'detail': {u'eventVersion': u'1.05', u'eventID': u'f8ae9323-91b0-4100-b27b-dce348641a5c', u'eventTime': u'2016-10-14T17:31:46Z', u'requestParameters': {u'instanceId': u'i-d8916d57', u'disableApiTermination': {u'value': True}}, u'eventType': u'AwsApiCall', u'responseElements': {u'_return': True}, u'awsRegion': u'eu-west-1', u'eventName': u'ModifyInstanceAttribute', u'userIdentity': {u'userName': u'ihork', u'principalId': u'AIDAI3UNW', u'accessKeyId': u'ASIAIN2', u'invokedBy': u'signin.amazonaws.com', u'sessionContext': {u'attributes': {u'creationDate': u'2016-10-14T16:48:44Z', u'mfaAuthenticated': u'true'}}, u'type': u'IAMUser', u'arn': u'arn:aws:iam::150905:user/igor', u'accountId': u'150905'}, u'eventSource': u'ec2.amazonaws.com', u'requestID': u'cd889e-039e-4f8f-bfe9-4d293012335', u'userAgent': u'signin.amazonaws.com', u'sourceIPAddress': u'174.231.5.2'}, u'detail-type': u'AWS API Call via CloudTrail', u'source': u'aws.ec2', u'version': u'0', u'time': u'2016-10-14T17:31:46Z', u'id': u'cd32fc6-39ae-4237-b46d-62d237d4d89', u'resources': []}
Disabling TerminationProtection from aws cli. 2nd variant
command:
$ aws ec2 modify-instance-attribute --attribute disableApiTermination --value false --instance-id i-199a6696
event:
{u'account': u'150905', u'region': u'eu-west-1', u'detail': {u'eventVersion': u'1.05', u'eventID': u'75fe4852-d3d9-4c9c-a702-7f025e0c4c50', u'eventTime': u'2016-10-14T17:29:24Z', u'requestParameters': {u'instanceId': u'i-d8916d57', u'attribute': u'disableApiTermination', u'value': u'false'}, u'eventType': u'AwsApiCall', u'responseElements': {u'_return': True}, u'awsRegion': u'eu-west-1', u'eventName': u'ModifyInstanceAttribute', u'userIdentity': {u'userName': u'ihork', u'principalId': u'AIDAI3U7LBIY', u'accessKeyId': u'AKIAJL7', u'type': u'IAMUser', u'arn': u'arn:aws:iam::150905:user/igor', u'accountId': u'150905'}, u'eventSource': u'ec2.amazonaws.com', u'requestID': u'd3c46-2def-4450-b3c1-4827d9f78', u'userAgent': u'aws-cli/1.10.45 Python/2.7.11 Linux/4.7.3-100.fc23.x86_64 botocore/1.4.60', u'sourceIPAddress': u'174.231.5.2'}, u'detail-type': u'AWS API Call via CloudTrail', u'source': u'aws.ec2', u'version': u'0', u'time': u'2016-10-14T17:29:24Z', u'id': u'9fb88a9e-025b-4859-9b98-6180cd14a9b', u'resources': []}
Take a precise look on:
'sessionContext': {u'attributes': {u'creationDate': u'2016-10-14T16:48:44Z', u'mfaAuthenticated': u'true'}} - not expect this part of JSON if you are not using Mfa
u'disableApiTermination': {u'value': True}} and 'attribute': u'disableApiTermination', u'value': u'false'} same API call done using different AWS CLI options, but serving the same purpose produce 2 different events
How we can disable a user in AWS ?
You just can't disable user in AWS. You can delete it, but you need to remove it from the groups first. It takes times, lines of code and API calls. Solution? - attach inline user policy with explicit deny (will override all allows) for all the actions you need to block.
Lambda function:
Here my PoC lambda function: really "dirty" and serving only one simple use case:
def lambda_handler(event, context):
print event
# analyzing event
if event['detail']['requestParameters'].get('disableApiTermination')!= None:
protection_status = event['detail']['requestParameters']['disableApiTermination']['value']
UserName = event['detail']['userIdentity']['userName']
UserID = event['detail']['userIdentity']['principalId']
if event['detail']['userIdentity'].get('sessionContext') != None:
mfa = event['detail']['userIdentity']['sessionContext']['attributes']['mfaAuthenticated']
else:
mfa = "false"
print protection_status, UserName, UserID, mfa
# disabling user using inline user policy if no MFA being used
if mfa != "true" and not protection_status:
iam = boto3.resource('iam')
user_policy = iam.UserPolicy(UserName,'disable_user')
response = user_policy.put(PolicyDocument='{ "Version": "2012-10-17", "Statement": [{"Sid": "Disableuser01","Effect": "Deny","Action": ["ec2:StopInstances", "ec2:TerminateInstances"],"Resource": ["*"]}]}')
print response
How near this "near real time" events:
My test showed about 40 second delay. IMHO too much for the "near real time". I'm looking for the potential bottleneck and delays that may caused by Lambda function itself on Event type I used.
Conclusions:
- not a "near real time" to react fast and mitigate attack without additional protective measures.
- could work if you able to detect attack 40 second earlier
- could reduce overall damages
- definitely very very promising if reaction delay will be less (let's say 5-10 sec).
event:
{u'account': u'150905', u'region': u'eu-west-1', u'detail': {u'eventVersion': u'1.05', u'eventID': u'75fe4852-d3d9-4c9c-a702-7f025e0c4c50', u'eventTime': u'2016-10-14T17:29:24Z', u'requestParameters': {u'instanceId': u'i-d8916d57', u'attribute': u'disableApiTermination', u'value': u'false'}, u'eventType': u'AwsApiCall', u'responseElements': {u'_return': True}, u'awsRegion': u'eu-west-1', u'eventName': u'ModifyInstanceAttribute', u'userIdentity': {u'userName': u'ihork', u'principalId': u'AIDAI3U7LBIY', u'accessKeyId': u'AKIAJL7', u'type': u'IAMUser', u'arn': u'arn:aws:iam::150905:user/igor', u'accountId': u'150905'}, u'eventSource': u'ec2.amazonaws.com', u'requestID': u'd3c46-2def-4450-b3c1-4827d9f78', u'userAgent': u'aws-cli/1.10.45 Python/2.7.11 Linux/4.7.3-100.fc23.x86_64 botocore/1.4.60', u'sourceIPAddress': u'174.231.5.2'}, u'detail-type': u'AWS API Call via CloudTrail', u'source': u'aws.ec2', u'version': u'0', u'time': u'2016-10-14T17:29:24Z', u'id': u'9fb88a9e-025b-4859-9b98-6180cd14a9b', u'resources': []}
Take a precise look on:
'sessionContext': {u'attributes': {u'creationDate': u'2016-10-14T16:48:44Z', u'mfaAuthenticated': u'true'}} - not expect this part of JSON if you are not using Mfa
u'disableApiTermination': {u'value': True}} and 'attribute': u'disableApiTermination', u'value': u'false'} same API call done using different AWS CLI options, but serving the same purpose produce 2 different events
How we can disable a user in AWS ?
You just can't disable user in AWS. You can delete it, but you need to remove it from the groups first. It takes times, lines of code and API calls. Solution? - attach inline user policy with explicit deny (will override all allows) for all the actions you need to block.
Lambda function:
Here my PoC lambda function: really "dirty" and serving only one simple use case:
def lambda_handler(event, context):
print event
# analyzing event
if event['detail']['requestParameters'].get('disableApiTermination')!= None:
protection_status = event['detail']['requestParameters']['disableApiTermination']['value']
UserName = event['detail']['userIdentity']['userName']
UserID = event['detail']['userIdentity']['principalId']
if event['detail']['userIdentity'].get('sessionContext') != None:
mfa = event['detail']['userIdentity']['sessionContext']['attributes']['mfaAuthenticated']
else:
mfa = "false"
print protection_status, UserName, UserID, mfa
# disabling user using inline user policy if no MFA being used
if mfa != "true" and not protection_status:
iam = boto3.resource('iam')
user_policy = iam.UserPolicy(UserName,'disable_user')
response = user_policy.put(PolicyDocument='{ "Version": "2012-10-17", "Statement": [{"Sid": "Disableuser01","Effect": "Deny","Action": ["ec2:StopInstances", "ec2:TerminateInstances"],"Resource": ["*"]}]}')
print response
How near this "near real time" events:
My test showed about 40 second delay. IMHO too much for the "near real time". I'm looking for the potential bottleneck and delays that may caused by Lambda function itself on Event type I used.
Conclusions:
- not a "near real time" to react fast and mitigate attack without additional protective measures.
- could work if you able to detect attack 40 second earlier
- could reduce overall damages
- definitely very very promising if reaction delay will be less (let's say 5-10 sec).
Update:
Fill free to pull from GitHub AWS CloudFormation template for the PoC above.
To deploy you need:
1. selfdefence.infosec.vpc.json - template itself.
2. selfdefence_infosec.py - Lambda function. You will need to Zip it and upload to the s3 bucket with versioning enabled.
3. Edit template (selfdefence.infosec.vpc.json) and specify: S3 bucket name in format you.bucket.name.env.cloudform (where env - is your environment name: prod, test, staging, etc) and S3 version for selfdefence_infosec.zip file.
4. upload template to the same s3 bucket.
5. Create a stack using this template end specify corresponding environment name at the creation time.
Enjoy!
