Using MFA with AWS CLI

It quite obvious nowadays that you must use MFA if it's available.
Enabling MFA for your user account in AWS IAM will automatically enforce it for the AWS Web UI login. 

But what about AWS CLI, your code using AWS SDK and 3d party SDK based tools?
In this case, to leverage MFA you need to enforce it using "Condition" statement for the IAM policy assigned to you user as it described in following AWS manual:
https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-role.html

In nutshell something like this:

Enforce MFA for the assume role:
{ "Version": "2012-10-17", "Statement": [ { "Sid": "", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::123456789012:user/anika" }, "Action": "sts:AssumeRole", "Condition": { "Bool": { "aws:multifactorAuthPresent": true } } } ] }

Add MFA to you AWS CLI profile:

[profile role-with-mfa] region = us-west-2 role_arn= arn:aws:iam::128716708097:role/cli-role source_profile = cli-user mfa_serial = arn:aws:iam::128716708097:mfa/cli-user

Simple? Not exactly - here some tricky things that not covered by AWS documentation (at least I was not able to find).

1. AWS documentation a bit misleading: in the AIM statement and documentation user name  mentioned is Anika but all CLI configs are pointing to the non existing cli profile "cli-user"

2. AWS CLI MFA configuration will work ONLY when you are assuming Role. Yep, if you have one simple account, few users  and groups (as many small companies do) you can't leverage this functionality without some small trick(item 3)

3. You can still leverage MFA with CLI using role:

• Strip all access from the user you are using to login, except "assume role", or alternatively,  enforce the MFA for all the actions using condition from the above.  

Note:
If you will strip all permissions you will need to assume role even if you are using WEB UI.
If you use alternative approach and enforce MFA for all API actions you can keep using WEB UI without assuming role the same way as you was doing before.

• Create a role (exampe: MyOrganizationAccountAccessRole) to assume in the same account with MFA enforced and all required access rights. If you have more than one account - create this role in other accounts as well with the same MFA enforcement condition.
• Create extra profile my-account-mfa (in addition to the main account profile my_account ) for the accessing the same account (my-account) using this role: 

[profile my-account-mfa]

role_arn = arn:aws:iam:: 123456789:role/MyOrganizationAccountAccessRole

source_profile = my_account

mfa_serial = arn:aws:iam:: 123456789:mfa/it-security@ca

[profile my_account]

output = json

region = us-east-1

mfa_serial = arn:aws:iam::123456789:mfa/it-security@ca

[profile my_second_account]

ole_arn = arn:aws:iam:: 987654321:role/MyOrganizationAccountAccessRole

source_profile = my_account

mfa_serial = arn:aws:iam:: 123456789:mfa/it-security@ca

Note : all profile reference  my_account profile as a source


If needed create an extra profile(my_second_account) for any other account you need to access using the role.

Use  profile  my-account-mfa for you CLI access to the main account or for any tools. You will see MFA request and after providing MFA everything will work like a charm.

Enjoy and stay secure!

Awesome list of Native AWS logging capabilities

While looking on centralized logging  capabilities of AWS and going trough bunch of documentation, I noticed lack of the one "big table" where I can find all AWS native logging capabilities per each service and up-to-date service coverage for AWS cloudwatch logs service.
Building I big table is not really version control friendly, so please welcome:

Awesome list of Native AWS logging capabilities:
https://github.com/IhorKravchuk/it-security/blob/master/AWS_logging.md

While I was building this list, some service have already changed their capabilities causing some information in the list being out-of-sync.
 I'll try my best to regularly review existing services and keep adding new one, but if you find mistake or would like to contribute feel free to contact me or create a PR.

Using Terraform to create project and users required in GCP and GSuite

This article is more like quick HOWTO/QuickNote page to start using Terraform with GCP, grant required permission, connect Terraform to GSuite and create users and projects using Terraform.

Connect Terraform to GCP:

1. Download and install Google Cloud SDK: https://cloud.google.com/sdk/install

2. Initialize SDK gcloud init 
 This process will launch browser-based authorization flow  https://cloud.google.com/sdk/docs/initializing

3. Use browser to create  project, service account and download credentials: https://cloud.google.com/sdk/docs/authorizing Note: You need to have GCP billing account and payment method configured first. You can use cli as well:

gcloud projects list
gcloud beta billing accounts list
gcloud beta billing projects link infosec-gcp --billing-account 01122-74525-1222
gcloud config list
gcloud iam service-accounts create infosec-terraform --display-name "Infosec Terraform admin account"
gcloud iam service-accounts keys create ~/.config/gcloud/infosec-terraform-admin.json --iam-account infosec-terraform@infosec-gcp.iam.gserviceaccount.com

4. Give appropriate permissions to the Terraform:
get you organization id
gcloud organizations list

Enable iam api (yes you need to enable each api set you are planning to use with GCP, they are disabled by default) you can check what services are enabled using gcloud services list --available
gcloud services enable iam.googleapis.com

Check existing IAM policies in you org:
gcloud organizations get-iam-policy ORGANIZATION_ID

Grant all required permissions(example):
gcloud organizations add-iam-policy-binding ORGANIZATION_ID --member serviceAccount:infosec-terraform@infosec-gcp.iam.gserviceaccount.com --role roles/resourcemanager.projectCreator

gcloud organizations add-iam-policy-binding ORGANIZATION_ID --member serviceAccount:infosec-terraform@infosec-gcp.iam.gserviceaccount.com --role roles/billing.user

gcloud organizations add-iam-policy-binding ORGANIZATION_ID --member serviceAccount:infosec-terraform@infosec-gcp.iam.gserviceaccount.com --role roles/owner

5. Start using terraform from my example to create project and grant access to it.

The only missing part is actually users.
Connecting Terraform to GSuite:

Why do we need GSuite at all? GCP does not provide you any built-in identity and rely on the user identities from Gmail, GSuite or Google Cloud Identity (+ service accounts)

As AWS user I do really love to have user/groups management and infra/project creation using the same automation tool. Unfortunately, user/GSuite functionality is not provided by GCP Terraform provider. Luckily, there is pretty nice open-sourced Terraform provider for GSuite writtend by DeviaVir: https://github.com/DeviaVir/terraform-provider-gsuite
At the moment when I tested it, some group membership functionality was still lacking idempotency, but  using the way from my example everything started to work like a charm.

So the code finally:
https://github.com/IhorKravchuk/it-security/tree/master/GCP


PS.
Way more details and examples are in the articles below :
https://medium.com/@josephbleroy/using-terraform-with-google-cloud-platform-part-1-n-6b5e4074c059
https://cloud.google.com/community/tutorials/managing-gcp-projects-with-terraform
https://cloud.google.com/community/tutorials/getting-started-on-gcp-with-terraform

AWS Route53 DNS records backup/change using aws cli

Challenge:
 you need to change a lot of DNS records inside the AWS Route53 hosted zone. In prod...
 Let's  skip the obvious question why these DNS records are not managed as Infra-as-aCode..
Sure thing, you need to backup all these record prior to change for rollback purpose.

Solution: 
1. create a list of the dns names to change

cat multisitest.it-security.ca.list 

test1.it-security.ca.

test2.it-security.ca.

test3.it-security.ca.

2. get zone id from AWS cli:

aws route53 list-hosted-zones

3. Normally aws route53 list-resource-record-sets --hosted-zone-id Z1YS
will give you JSON, but unfortunately it's not useful for quick restore due to the format difference from the change-resource-record-sets.json file you need to have to change/restore records.

4. With a quick and quite dirty bash we can get better formatted JSON:

while read site; do echo '{ "Action": "UPSERT","ResourceRecordSet":';  aws route53 list-resource-record-sets --hosted-zone-id Z1YS --query "ResourceRecordSets[?Name == '$site']" --profile it-sec | jq .[] ; echo "},"; done < multisitest.it-security.ca.list > multisitest.it-security.ca.back.json

This file has almost everything needed to build change-batch file for the aws cli: https://docs.aws.amazon.com/cli/latest/reference/route53/change-resource-record-sets.html
Almost.. We need to add

{

  "Comment": "Point some Test TLS1.2 enviroments to the Incapsula",

  "Changes": [

in the beginning of the change set, and
remove "," and add

  ]

}

 att the end.

5. Now you have Route53 DNS records backed up and  ready to restore.
Next step is to create a copy of you backup file and modify it to reflect changes you need to make.

6. Final step: apply your changes:

aws route53 change-resource-record-sets --hosted-zone-id Z1YS  --change-batch file://multisitest.it-security.ca.json --profile it-sec

7. And, in case of disaster, use the same command to roll it back quickly specifying backup file:

aws route53 change-resource-record-sets --hosted-zone-id Z1YS  --change-batch file://multisitest.it-security.ca.back.json --profile it-sec

Secure your AWS account using Terrafrom and CloudFormation

This is very updated version of the blog post: http://blog.it-security.ca/2016/11/secure-your-aws-account-using.html

As I mention before:
The very first thing you need to do while building your AWS infrastructure is to enable and configure all AWS account level security features such as: CloudTrail, CloudConfig, CloudWatch, IAM, etc.

Time flies when you're having fun and flies even faster in the infosec world. My templates become outdated and now I'm presenting an updated version of the AWS security automation with following new features:

1. integrated with Terraform (use terraform templates in the folder tf)
2. creates prerequisites for Splunk integration (User, key, SNS, and SQS)
3. configures cross-account access (for multiaccount organizations, adding ITOrganizationAccountAccessRole with MFA enforced)
4. implements Section 3 (Monitoring) of the CIS Amazon Web Services Foundations benchmark.
5. configures CloudTrail according to the new best practices (KMS encryption, validation etc)
6. configures basic set of the CloudConfig rules to monitor best practices

First, my security framework now consists of two main parts: cf (CloudFormation) and tf (Terraform) with Terraform template as a bootstrapper of the  whole deployment.

You can use  Terraform, you can use CloudFormation, but why both ?
Terraform is very quickly evolves, has cross-cloud support and implements some missing in CloudFormation features (like account level password policy configuration, etc); CloudFormation is native for AWS, well supported, and, most important, AWS provides a lot of best practices and solutions in the form of the CloudFormation templates.

Using both (tf and cf) gives me (and you) ability to reuse solutions, suggested and provided by AWS, without rewriting the code, have flexibility and power of terraform and one single interface for whole cloud automation.
No more bucket pre-creation or specific sequence of the CloudFormation deployment - just terraform apply. It will take care of all CloudFormation prerequisites, version control and template updates.
But,  if you wish, at current state you can use only my CloudFormation templates - cf still does all heavy lifting.

The main trick of the Terraform - CloudFormation integration was to tell terrafrom when CloudFormation template is updated to ensure that terraform will trigger cf stack update.
I achieved this using S3 bucket with version control enabled and always updating (just setting template version) security.global.yaml.

This code takes care of Terraform and CloudFormation integration:
# creating Security cloudforation stack

resource "aws_cloudformation_stack" "Security" {
  name = "Security"
  depends_on = ["aws_s3_bucket_object.iam_global", "aws_s3_bucket_object.cloudtrailalarms_global", "aws_s3_bucket_object.awsconfig_global", "aws_s3_bucket_object.cloudtrail_global", "aws_s3_bucket_object.security_global"]
  parameters {
    AccountNickname = "${var.enviroment_name}",
    CompanyName = "${var.company_name}",
    MasterAccount = "${var.master_account}"
  }
  template_url = "https://s3.amazonaws.com/${aws_s3_bucket.CFbucket.bucket}/${var.security_global}?versionId=${aws_s3_bucket_object.security_global.version_id}"
  capabilities = [ "CAPABILITY_NAMED_IAM" ]
  tags { "owner" = "infosec"}
}

And finally deployment steps are:

1. Get code from my git repo:  https://github.com/IhorKravchuk/it-security
2. Switch to tf folder and update terraform.tfvars specifying: your AWS profile name (configured for aws cli using aws configure --profile profile_name); name for the environment (prod, test, dev ..) ; company(or division) name; region and AWS master account ID.
3. terraform init to get aws provider downloaded by terraform
4. terraform plan
5. terraform apply

S3 buckets audit: check bucket's public access level, etc .. updated with authorised audit support

I previous post: S3 buckets audit: check bucket existence, public access level, etc - without having access to target AWS account I described and released tool to audit s3 buckets even without access to the AWS account these buckets belong to.

But what about if I have access to the bucket's account or I would like to audit all buckets in my AWS account?

These features have been addressed in the new release of the s3 audit tool:

https://github.com/IhorKravchuk/it-security/tree/master/scripts.aws

$python aws_test_bucket.py --profile prod-read --bucket bucket2test

$python aws_test_bucket.py --profile prod-read --file aws

$python aws_test_bucket.py --profile prod-read --file buckets.list

  -P AWS_PROFILE, --profile=AWS_PROFILE
                        Please specify AWS CLI profile
  -B BUCKET, --bucket=BUCKET
                        Please provide bucket name
  -F FILE, --file=FILE  Optional: file with buckets list to check or aws to check all buckets in your account

Note:

--profile=AWS_PROFILE - yours AWS access profile (from aws cli). This profile  might or might not have access to the audited bucket (we need this just to become Authenticated User from AWS point of view ).

If  AWS_PROFILE allows authorised access to the bucket being audited - tool will fetch bucket's ACLs, Policies and S3 Static Web setting and perform authorised audit.

If AWS_PROFILE does not allow authorised access - tool will work in pentester mode

You can specify:

•  one bucket to check using --bucket option
•  file with list of buckets(one bucket name per line) using --file option
•  all buckets in your AWS account (accessible using AWS_PROFILE) using --file=aws option

Based on the your AWS profile limitations tool will provide you:

• indirect scan results (AWS_profile have no API access to the bucket being audited)
• validated scan results based on you s3 buckets settings like ACL, bucket policy and s3 website config. (AWS_profile have API access to the bucket being audited )

Enjoy and stay secured.

PS. Currently tool does not support bucket check for Frankfurt region (AWS Signature Version 4). Working on it.

S3 buckets audit: check bucket existence, public access level, etc - without having access to target AWS account

      Currently, publicly accessible buckets become a big deal and root cause of many recent data leaks.
All of these events even drive Amazon AWS to proactively send out emails to the customers who has such s3 configurations. Let's become a bit more proactive as well and audit s3 buckets

        First, let's take look why bucket might become publicly available:
- Configured for public access intentionally (S3 static web hosting or just public resource) or by mistake
- Configured for the access of the Authenticated Users  (option, misinterpreted by many as users from your account, which is wrong, it's any AWS authenticated user from any account)
     
         Auditing AWS account you have full access to is quite easy - just list the buckets and check theirs ACL, users and bucket policies via aws cli or web gui.

         What about cases when you:
- have many accounts and buckets (will take forever to audit manually)
- do not have enough permissions in the target AWS account to check bucket access
- you do not have permissions at all in this account (pentester mode)

To address everything above I've created small tool to do all dirty job for you (updated to v2):
https://github.com/IhorKravchuk/it-security/tree/master/scripts.aws


$python aws_test_bucket.py --profile prod-read --bucket test.bcuket

  -P AWS_PROFILE, --profile=AWS_PROFILE
                        Please specify AWS CLI profile
  -B BUCKET, --bucket=BUCKET
                        Please provide bucket name
  -F FILE, --file=FILE  Optional: file with buckets list to check

Note: --profile=AWS_PROFILE - any your AWS access profile (from aws cli). This profile HAS to NOT have access to the audited bucket (we need this just to become Authenticated User from AWS point of view )

You can specify one bucket to check using --bucket option or file with list of buckets(one bucket name per line) using --file option

Based on the bucket access status tool will provide you following responses:

Bucket: test.bucktet - The specified bucket does not exist

Bucket: test.bucktet -  Bucket exists, but Access Denied

Bucket: test.bucktet -  Found index.html, most probably S3 static web hosting is enabled

Bucket: test.bucktet - Bucket exists, publicly available and no S3 static web hosting, most probably misconfigured! 

Enjoy!

PS. More over, you can create list of the buckets(even using some DNS/name alterations and permutations) to test in the file and loop through it checking each.

Stay secure.

MediaWiki as a static website and content sharing

Using wiki for knowledge management in a teams or individually is easy and often is an obvious choice.

       Challenges appear when you need to share information stored in the wiki. 

Challenges are: hardening MediaWiki installation for public access and partially sharing wiki content.

If your main goal is to just to publish content, you can extract wiki pages as a static html pages using relatively simple wget one-liner. After extracting, you can publish your wiki using AWS  S3 static web hosting.

To share only part of the information available in the wiki you can leverage Categoies and restrict user access to specified categories using special extension. Afterwards, you can use this user restricted access to grab wiki content.
Another simple way is to use Category special wiki page as a starting point for crawler to grab pages related to the specific category, let's say Public category.
The code is way shorter than all description above:

# get the wiki content
wget --recursive --level=1 --page-requisites --html-extension --no-directories --convert-links --no-parent -R "*Special*" -R "*action=*" -R "*printable=*"  -R "*oldid=*" -R "*title=Talk:*" -R "*limit=*" "http://mywikiprivate:80/wiki/index.php/Category:Public"
# replace sensitive by the link to the stub page
sed -i -E 's/http:\/\/mywikiprivate[^"]*/http:\/\/wiki.your_website.ca\/404.html/g' *.html 
# remove sensitive file
rm Category\:Public.1.html
# rename Public category pages to be an a list of published pages
mv Category:Public.html Public.html 
# sync content to AWS
aws s3 sync ./ s3://you_bucket/

Result of such script running along with some public notes from my wiki could be found here:
http://wiki.it-security.ca

Disclaimer: current wiki publication contains only small part of the information available and will be updated on almost daily basis to add more content cleared for publishing. Main purpose of this wiki is to keep technical notes and references in the structured way. Some of them are obvious, outdated or incomplete.

Goal of the establishing public publishing process is to keep wiki information up-do-date  and have ability to publish small useful notes which does not fit blog format and style.

Secure your AWS account using CloudFormation

      The very first thing you need to do while building your AWS infrastructure is to enable and configure all AWS account level security features such as: CloudTrail, CloudConfig, CloudWatch, IAM, etc..

       To do this, you can use mine Amazon AWS Account level security checklist and how-to or any other source.

        To avoid manual steps and to be align with SecuityAsCode concept, I use set of CloudFormation templates, simplified version of which I would like to share: 

GitHub: https://github.com/IhorKravchuk/it-security

Global Security stack template structure:

security.global.json - parent template for all nested templates to link them together and control dependency between nested stacks.

cloudtrail.clobal.json - nested template for Global configuration of the CloudTrail:

• creates S3 bucket for the logs
• creates CloudTrail-related IAM roles and policies
• creates CloudLog LogGroup
• enables CloudTrail on default region including global events and multi-region feature
• creates SNS ans SQS configuration for easy integration with Splunk AWS app.

cloudtrailalarms.global.json - nested template for Global CloudWatch Logs alarms and security metrics creation. Uses FilterMap to create different security-related filters for ClouTrail LogGroup, corresponding metrics and notifications for suspicious or dangerous events. You can customise filter per environment basis.

Predefined filters are:

• rds-change: RDS related changes
• iam-change; IAM changes
• srt-instance: Start, Reboot, Terminate instance
• large-instance: launching large instances
• massive-operations: massive operations- more 10 in 5 min 
• massive-terminations: massive terminations- more 10 in 5 min 
• detach-force-ebs: force detachment of the EBS volume from the instance
• change-critical-ebs: any changes related to the critical EBS volumes
• change-secgroup: any changes related to the security group
• create-delete-secgroup: creation and deletion of the security group 
• secgroup-instance: attaching security group to the instance
• route-change: routing  changes
• create-delete-vpc: creation and deletion of a VPC
• netacl-change: changes at Network ACL
• cloudtrail-change: changes in the CloudTrail configuration
• cloudformation-change: changes related to the CloudFormation
• root-access: any root access events
• unauthorised: failed and unauthorised operations
• igw-change: Internet Gateway related changes
• vpc-flow-logs: Delete or Create VPC flow logs
• critical-instance: any operation on the critical instances
• eip-change: Elastic IP changes
• net-access: Any access outside of predefined known IP ranges

4 preconfigured notification topics : 

• InfosecEmailTopic, 
• DevOpsEmailTopic
• InfosecSMSTopic
• DevOpsSMSTopic

awsconfig.global.json - nested template for Global AWS Config Service configuration.

• creates S3 bucket for the config dumps
• creates AWS Config-related IAM roles and policies
• creates AWS config delivery channel and schedule config dumps (hourly)
• creates and enables AWS config recorder 
• creates SNS ans SQS configuration for easy integration with Splunk AWS app.

cloudwatchsubs.global.json - nested template for configuring AWS CloudWatch Subscription Filter to extract and analyse most severe CloudTrail events using custom Lambda function:

• creates Lambda function and all requred roles and permissions
• creates Subscription filter as a compilation of the filters from the FilterMap
    

Currently uses following filters and aggregate them to one due to the AWS CloudWatch Logs subscription limitation (only one filter supported):

• critical-instance
• iam-change
• srt-instance
• cloudtrail-change
• root-access
• net-access
• detach-force-ebs
• unauthorised

iam.global.json - nested template for IAM Global configuration: 

• creates Infosec Team IAM Group and managed policy
• creates DevOps Team IAM Group and managed policy
• creates DBA Team IAM Group and managed policy
• creates Self Service Policy  for users to manage API keys and MFA
• creates  ProtectProdEnviroment to protect production environment from destructive actions 
• creates EnforceMFAPolicy to enforce MFA for sensitive operations
• creates EnforceAccessFromOfficePolicy to restrict some operation to office source IPs
• creates DomainJoin role and all required policy to perform automated domain join
• creates SaltMasterPolicy and Role for Configuration Management Tool (in this case - Salt)
• creates SQLDataBaseInstancePolicy and Instance profile example policy
• creates SIEM system example policy
• creates VPC flow log role
• creates and manages SIEM user and API keys
• creates and manages SMTP user (for the AWS SES service ) and API keys

cloudwatchsubs_kinesis.global.json - PoC template (not linked as nested to the security.global.json)  for configuring AWS CloudWatch Subscription Filter to send most severe CloudTrail events to AWS Kinesis stream using subscription filter similar to the cloudwatchsubs.global.json 

Supported features:

Environments and regions: Stack supports unlimited amount of environments with 4 environments predefined (staging, dev, prod, and dr) and use 1 account and 1 region per environment concept to reduce blast radius (if account become compromised)

AWS services used by stack: CloudTrail, AWS Config, CloudWatch, CloudWatch Logs and Events, IAM,  Lambda, Kinesis.

To deploy:

1. Create bucket using following naming convention: com.ChangeMe.EnviromentName.cloudform, replacing ChangeMe and EnviromentName with your value to make it look like this:            com.it-security.prod.cloudform
2. Enable bucket versioning 
3. in the templates  security.global.json and cloudwatchsubs.global.json replace "ChangeMe" with name used in the bucket creation.
4. In the template cloudtrailalarms.global.json modify SNS endpoints for email notification infosec@ChangeMe.com and devops@ChangeMe.com; Add endpoints with mobile phone numbers for SMS notification to appropriate SNS topics if needed.
5. Modify iam.global.json template to adrress you SQL DataBase bucket location (com-ChangeMe-", {"Ref": "Environment"} , "-sqldb/)  and modify any permission if need according to your organisation structure, roles, responsibilities and services.
6. Modify FilterMap in cloudtrailalarms.global.json and cloudwatchsubs.global.json templates make filters work for your infrastructure (Critical Instance IDs, Critical Volume IDs, you ofiice IP range, you NAT gateways, etc) 
7. Zip example Lambda function LogCritical_lambda_security_global.py like LogCritical_lambda_security_global.zip
8. Upload this function into S3 bucket created at step 1 and copy object version (GUI- show version -object properties ) and insert into cloudwatchsubs.global.json template into "LogCriticalLambdaCodeVer" mapping at the appropriate environment (prod, staging ..)
9. Modify "regions" Environments mapping in the iam.global.json, and cloudwatchsubs.global.json templates to specify correct AWS region you are using for the deployment.
10. Upload all *.global.json templates into S3 bucket created at step 1. 
11. Create new CloudFormation stack  using parent security template security.global.json and your bucket name (Example: ttps://s3.amazonaws.com/com.it-security.prod.cloudform/security.global.json ),  call it "Security" and specify environment name you going to deploy.
12. Done!

Self-Defending Cloud PoC or Amazon CloudWatch Events usage

Problem:  Malicious attacker get privileged access to your AWS account and destroying your production infrastructure in a matter of seconds.

In case of cloud based infrastructures you can't rely on classic SIEM solutions - by the time your SIEM will detect attack, your infrastructure will be gone. We need a "near real time" way to detect and mitigate the attack.

Attack scenario: using compromised aws api key (no MFA) and CLI/SDK to perform destructive actions.

Attack detection and mitigation strategy:  
All destructive actions start with EC2 instance termination. To prevent such scenario, you should always have "TerminationProtection" feature enabled on your production instances. Based on this, attacker must disable termination protection before demolishing your environment. For the PoC, I will use disabling "TerminationProtection" event as a attack detector (sure thing, real attack detection is a way more complicated process).

Starting point: AWS api key with admin policy attached, all production instances protected using AWS Termination Protection.

Possible solutions and attack mitigation delays:

SIEM:

CloudWatch Logs and alarms:

CloudWatch Subscriptions:

CloudWatch Events:

Implementation: 

Design:
Based on the implementation scenarios shown above and their performance (tested during PoC)  - the fastest way is to leverage AWS CloudWatch events and trigger a lambda function.

Speaking AWS technical language we need:

1. choose what type of AWS event we are looking for. Based on the our attack detection strategy, we are looking for ec2 call: modify-instance-attribute. Using exactly this API call you can enable/disable TerminationProtection. So let's look for "AWS API Call Events" type of the CloudWatch events.
2. Create: event rule : "match incoming events and route them to one or more targets for processing" in our case target is a Lambda function.
3. Create all required policies and Lambda function role in IAM.
4. Build the Lambda function itself.

Setting-up event rule:

I was using following event pattern inside the CloudWatch Event rule:
{ "detail-type": [ "AWS API Call via CloudTrail" ], "detail": { "eventSource": [ "ec2.amazonaws.com" ], "eventName": [ "ModifyInstanceAttribute" ] } }


Getting sample event:

To start writing our event-detection-mitigation lambda function we need to get example of the AWS events for the API call we are monitoring.
We can achieve this with the following simple Lambda function:

import json

def lambda_handler(event, context):
    print event


or, if you need nice formatted json to test you lambda function offline:

import json

def lambda_handler(event, context):
    print json.dumps(event, indent=4, sort_keys=False)

You will find output of your lambda function (result of the print statement) in appropriate (naming as your lambda function) CloudWatch Log Stream


Challenges with event format:

During first tests, I've found that Amazon AWS not following any JSON contract (defined format)  even for the same 1 API call. Making the same API call in the 3 different ways produced 3 different event formats:

Disabling TerminationProtection from GUI with MFA:

{u'account': u'150905', u'region': u'eu-west-1', u'detail': {u'eventVersion': u'1.05', u'eventID': u'b3c4d3b4-353e-44bf-8973-37abccd085b5', u'eventTime': u'2016-10-14T17:31:37Z', u'requestParameters': {u'instanceId': u'i-d8916d57', u'disableApiTermination': {u'value': False}}, u'eventType': u'AwsApiCall', u'responseElements': {u'_return': True}, u'awsRegion': u'eu-west-1', u'eventName': u'ModifyInstanceAttribute', u'userIdentity': {u'userName': u'ihork', u'principalId': u'AIDAI3UNW', u'accessKeyId': u'ASIAIN2', u'invokedBy': u'signin.amazonaws.com', u'sessionContext': {u'attributes': {u'creationDate': u'2016-10-14T16:48:44Z', u'mfaAuthenticated': u'true'}}, u'type': u'IAMUser', u'arn': u'arn:aws:iam::150905:user/igor', u'accountId': u'150905'}, u'eventSource': u'ec2.amazonaws.com', u'requestID': u'e7b585e-af38-49d0-88a8-979ef5052f', u'userAgent': u'signin.amazonaws.com', u'sourceIPAddress': u'174.231.5.2'}, u'detail-type': u'AWS API Call via CloudTrail', u'source': u'aws.ec2', u'version': u'0', u'time': u'2016-10-14T17:31:37Z', u'id': u'55084ea-e4bc-45e6-a7a6-0c8e7d16b32', u'resources': []}

Disabling TerminationProtection from aws cli (no MFA):

command:
$ aws ec2 modify-instance-attribute --no-disable-api-termination --instance-id i-378579b8 

event:
{u'account': u'150905', u'region': u'eu-west-1', u'detail': {u'eventVersion': u'1.05', u'eventID': u'f8ae9323-91b0-4100-b27b-dce348641a5c', u'eventTime': u'2016-10-14T17:31:46Z', u'requestParameters': {u'instanceId': u'i-d8916d57', u'disableApiTermination': {u'value': True}}, u'eventType': u'AwsApiCall', u'responseElements': {u'_return': True}, u'awsRegion': u'eu-west-1', u'eventName': u'ModifyInstanceAttribute', u'userIdentity': {u'userName': u'ihork', u'principalId': u'AIDAI3UNW', u'accessKeyId': u'ASIAIN2', u'invokedBy': u'signin.amazonaws.com', u'sessionContext': {u'attributes': {u'creationDate': u'2016-10-14T16:48:44Z', u'mfaAuthenticated': u'true'}}, u'type': u'IAMUser', u'arn': u'arn:aws:iam::150905:user/igor', u'accountId': u'150905'}, u'eventSource': u'ec2.amazonaws.com', u'requestID': u'cd889e-039e-4f8f-bfe9-4d293012335', u'userAgent': u'signin.amazonaws.com', u'sourceIPAddress': u'174.231.5.2'}, u'detail-type': u'AWS API Call via CloudTrail', u'source': u'aws.ec2', u'version': u'0', u'time': u'2016-10-14T17:31:46Z', u'id': u'cd32fc6-39ae-4237-b46d-62d237d4d89', u'resources': []}

Disabling TerminationProtection from aws cli. 2nd variant

command:

$ aws ec2 modify-instance-attribute --attribute disableApiTermination --value false --instance-id i-199a6696 

event:
{u'account': u'150905', u'region': u'eu-west-1', u'detail': {u'eventVersion': u'1.05', u'eventID': u'75fe4852-d3d9-4c9c-a702-7f025e0c4c50', u'eventTime': u'2016-10-14T17:29:24Z', u'requestParameters': {u'instanceId': u'i-d8916d57', u'attribute': u'disableApiTermination', u'value': u'false'}, u'eventType': u'AwsApiCall', u'responseElements': {u'_return': True}, u'awsRegion': u'eu-west-1', u'eventName': u'ModifyInstanceAttribute', u'userIdentity': {u'userName': u'ihork', u'principalId': u'AIDAI3U7LBIY', u'accessKeyId': u'AKIAJL7', u'type': u'IAMUser', u'arn': u'arn:aws:iam::150905:user/igor', u'accountId': u'150905'}, u'eventSource': u'ec2.amazonaws.com', u'requestID': u'd3c46-2def-4450-b3c1-4827d9f78', u'userAgent': u'aws-cli/1.10.45 Python/2.7.11 Linux/4.7.3-100.fc23.x86_64 botocore/1.4.60', u'sourceIPAddress': u'174.231.5.2'}, u'detail-type': u'AWS API Call via CloudTrail', u'source': u'aws.ec2', u'version': u'0', u'time': u'2016-10-14T17:29:24Z', u'id': u'9fb88a9e-025b-4859-9b98-6180cd14a9b', u'resources': []}


Take a precise look on:

'sessionContext': {u'attributes': {u'creationDate': u'2016-10-14T16:48:44Z', u'mfaAuthenticated': u'true'}} - not expect this part of JSON if you are not using Mfa

 u'disableApiTermination': {u'value': True}} and 'attribute': u'disableApiTermination', u'value': u'false'} same API call done using different AWS CLI options, but serving the same purpose produce 2 different events

How we can disable a user in AWS ?

You just can't disable user in AWS. You can delete it, but you need to remove it from the groups first. It takes times, lines of code and API calls. Solution? - attach inline user policy with explicit deny (will override all allows) for all the actions you need to block.

Lambda function:

Here my PoC lambda function: really "dirty" and serving only one simple use case:

def lambda_handler(event, context):
    print event
# analyzing event
    if event['detail']['requestParameters'].get('disableApiTermination')!= None:
        protection_status = event['detail']['requestParameters']['disableApiTermination']['value']
        UserName = event['detail']['userIdentity']['userName']
        UserID = event['detail']['userIdentity']['principalId']
        if event['detail']['userIdentity'].get('sessionContext') != None:
            mfa = event['detail']['userIdentity']['sessionContext']['attributes']['mfaAuthenticated']
        else:
            mfa = "false"
        print protection_status, UserName, UserID, mfa
# disabling user using inline user policy if no MFA being used
        if mfa != "true" and not protection_status:
            iam = boto3.resource('iam')
            user_policy = iam.UserPolicy(UserName,'disable_user')
            response = user_policy.put(PolicyDocument='{ "Version": "2012-10-17", "Statement": [{"Sid": "Disableuser01","Effect": "Deny","Action": ["ec2:StopInstances", "ec2:TerminateInstances"],"Resource": ["*"]}]}')
            print response



How near  this "near real time" events:
My test showed about 40 second delay. IMHO too much for the "near real time". I'm looking for the potential bottleneck and delays that may caused by Lambda function itself on Event type I used. 


Conclusions:
- not a "near real time" to react fast and mitigate attack without additional protective measures.
- could work if you able to detect attack 40 second earlier
- could reduce overall damages
- definitely very very promising if reaction delay will be less (let's say 5-10 sec).

Update:

Fill free to pull from GitHub AWS CloudFormation template for the  PoC above.

To deploy you need: 

1. selfdefence.infosec.vpc.json - template itself.

2. selfdefence_infosec.py - Lambda function. You will need to Zip it and upload to the s3 bucket with versioning enabled.

3. Edit template (selfdefence.infosec.vpc.json) and specify: S3 bucket name in format you.bucket.name.env.cloudform (where env - is your environment name: prod, test, staging, etc) and S3 version for  selfdefence_infosec.zip file. 

4. upload template to the same s3 bucket.

5. Create a stack using this template end specify corresponding environment name at the creation time.

Enjoy! 

S3 bucket policies for sensitive security logs storage

Inspired by this AWS  blog post : How to Restrict Amazon S3 Bucket Access to a Specific IAM Role

Goal:
Build a storage for sensitive security logs using S3 bucket.

Restrictions:  

• EC2 instances could only upload logs. 
• Infosec team could only download logs and (just for this particular case) delete them with MFA .
•  All other user must not have any access despite whatever mentioned in their IAM policies.

 Solution:
custom bucket policy


        "PolicyDocument": {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Sid": "OnlyForInfosecEyes",
              "Effect": "Deny",
              "Principal":"*",
              "Action": ["s3:GetObject*", "s3:Delete*", "s3:PutObjectAcl", "s3:PutObjectVersionAcl"],
              "Resource": "s3-top-secret-bucket/*",
              "Condition": {
                "StringNotLike": {
                  "aws:userId":  "InfosecGroupUserIDs"
                }
              }
            },
            {
              "Sid": "OnlyServerAllowToPut",
              "Effect": "Deny",
              "Principal":"*",
              "Action": ["s3:PutObject"],
              "Resource": "s3-top-secret-bucket/*",
              "Condition": {
                "StringNotLike": {
                  "aws:userId":  "SeverIAMRoleID:*"
                }
              }
            },
            {
              "Sid": "EnforceEncryption",
              "Effect": "Deny",
              "Principal":"*",
              "Action": ["s3:PutObject"],
              "Resource": "s3-top-secret-bucket/*",
              "Condition": {
                "Null": {
                  "s3:x-amz-server-side-encryption": "true"
                }
              }
            },
            {
              "Sid": "EnforceMFADelete",
              "Effect": "Deny",
              "Principal":"*",
              "Action": ["s3:Delete*"],
              "Resource": "s3-top-secret-bucket/*",
              "Condition": {
                "Null": {
                  "aws:MultiFactorAuthAge": true
                }
              }
            }
          ]
        }

Where:

InfosecGroupUserIDs - list of IAM infosec users' IDs (aws iam get-user -–user-name USER-NAME)

SeverIAMRoleID:* - ID of the IAM role used by the your EC2 server instances with ":*" added to cover all instances in this role (aws iam get-role -–role-name ROLE-NAME.)

AWS EC2 status check alarms using python and boto3

Important part of security that we (infosec guys) often delegate :-)  to the Operation teams(NOC) is Availability.
       For the IaaS service provider (Amazon AWS) is responsible for Infrastructure availability, but we must design all layers above ( Availability Zones, VPCs, Networks, Instances and LB ) for high availability or at least fault tolerance. One of the most important step in this process is actually detection IaS failure.
From AWS:
"With instance status monitoring, you can quickly determine whether Amazon EC2 has detected any problems that might prevent your instances from running applications. Amazon EC2 performs automated checks on every running EC2 instance to identify hardware and software issues. You can view the results of these status checks to identify specific and detectable problems."

Below simple python script that will help you to configure status check alarms for all you running instances:

#!/usr/bin/python

import boto3
import pprint

boto3.setup_default_session(profile_name='staging', region_name='eu-west-1')
ec2 = boto3.resource('ec2')
cloudwatch=boto3.resource('cloudwatch')

# Getting all running instances
instance_iterator = ec2.instances.all()
for instance in instance_iterator:
    instance_name = "unnamed"
    for tag in instance.tags:
        if tag['Key'] == "Name":
            instance_name = tag['Value']
    print instance_name, instance.id
    if instance.state["Name"] == "running" :
        metric = cloudwatch.Metric("AWS/EC2", "StatusCheckFailed")
        response = metric.put_alarm(
        AlarmName = instance.id + "/" + instance_name + "-status-alarm",
        AlarmDescription = 'status check for %s %s' % (instance.id, instance_name),
        ActionsEnabled = True,
        OKActions = ["arn:aws:sns:eu-west-1:your_account_id:YOUR_SNS-EmailSMS-Notification"],
        AlarmActions = ["arn:aws:sns:eu-west-1:your_account_id:YOUR_SNS-EmailSMS-Notification"],
        Statistic = "Maximum",
        Dimensions = [{'Name': 'InstanceId', 'Value': instance.id}],
        Period = 60,
        EvaluationPeriods = 2,
        Threshold = 1.0,
        ComparisonOperator = "GreaterThanOrEqualToThreshold"
        )
        pprint.pprint(response)

Amazon AWS Account level security checklist and how-to

Disclaimer :-):
There are bunch of Amazon AWS security checklists and recommendations online. Definitely the best one is https://d0.awsstatic.com/whitepapers/compliance/AWS_CIS_Foundations_Benchmark.pdf 
I'm not trying to reinvent the wheel, but integrate and summarize lessons I learned and advices given to me by other AWS experts.

This checklist starts from the moment when you begin AWS account creation.

1.  Create dedicated email address for AWS account registration. This email will become you root account login name, so, please, do not use your daily used or published online email
2. Enable MFA  (Multi Factor Authentication) on the root account. Details: http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_mfa.html
3. Remove or DO NOT create any API key associated with root account. API keys has no MFA - anyone who has root API keys gets full  access to you account. Unintentional leaking of the API key quite common security incident.
4. Copy/bookmark/save IAM sign-in url. You will need to access you AWS Web GUI.
5. Create IAM user with  AdministratorAccess policy attached. It will be your new  "root" like account.
6. Create other IAM users required. Minimize their permission using built-in AWS managed policies like: PowerUserAccess; ReadOnlyAccess; AmazonEC2FullAccess , etc
7. Enable MFA on all users created.  Details: http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_mfa.html
8. Enforce strict password policy. Details: http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_passwords_account-policy.html
9. Generate API keys for users who needs it. For "high-power" user make this keys inactive. They will activate keys through MFA protected AWS Web GUI only when it needed.
10. Do not use API keys in applications running inside AWS. Use IAM roles instead. Details: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
11. Enable and configure CloudTrail  for all regions  + s3 bucket for the CloudTrailLogs.  Details: http://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-create-a-trail-using-the-console-first-time.html
12. Send CloudTrails Events to the CloudWatch Logs. Details: http://docs.aws.amazon.com/awscloudtrail/latest/userguide/send-cloudtrail-events-to-cloudwatch-logs.html
13. Configure monitoring of the CloudTrail Log Files using Amazon CloudWatch Logs metric filters and alarms. Details: http://docs.aws.amazon.com/awscloudtrail/latest/userguide/monitor-cloudtrail-log-files-with-cloudwatch-logs.html
14. Configure near-real time Log data processing using Subscriptions or/and using lambda function.  Details: http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/Subscriptions.html
15. Using #13 and 14 configure notification for suspicions events
16. Enable AWS Config Service to get AWS configuration snapshots and change notifications. Details: http://docs.aws.amazon.com/config/latest/developerguide/gs-console.html
17. Enable and configure AWS VPC flow logs to get visibility on network level. Details: http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/flow-logs.html
18. Enforce server side encryption on your S3 buckets: Details: http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html
19. Enable encryption on you EBS volumes: Details: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSEncryption.html

Almost all steps covered above could and must be automated. I already published and will publish more automation examples in this blog.

Check your resulted account security status:

• Check Amazon AWS Trusted advisor 
• Run AWS Credentials Report 
• Run awesome AWS audit Scout2 tool : https://isecpartners.github.io/Scout2/

And do this periodically. 

Checklists and Best Practices:

AWS CIS Foundations Benchmark (must read document)
https://d0.awsstatic.com/whitepapers/compliance/AWS_CIS_Foundations_Benchmark.pdf

AWS Auditing Security Checklist
https://d0.awsstatic.com/whitepapers/compliance/AWS_Auditing_Security_Checklist.pdf

PS. I would like to thank Liem aka Pimpon  for advices in preparing this checklist.

AWS "one-liners": Configure AWS password policy in one shot

"As soon as you have passwords you need a password policy" - © captain obvious

Limitations:
AWS allows you to have only one password policy for whole AWS account.

You can configure it using web GUI or, if you prefer to have all your infrastructure and security as code, using boto and python:

#!/usr/bin/python

import boto3
import pprint

boto3.setup_default_session(profile_name='staging')
iam=boto3.resource('iam')
account_password_policy = iam.AccountPasswordPolicy()
response = account_password_policy.update(
    MinimumPasswordLength=12,
    RequireSymbols=True,
    RequireNumbers=True,
    RequireUppercaseCharacters=True,
    RequireLowercaseCharacters=True,
    AllowUsersToChangePassword=True,
    MaxPasswordAge=90,
    PasswordReusePrevention=12,
    HardExpiry=False
)

pprint.pprint(response)

You can find more details about particular password policy parameters here:

http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_passwords_account-policy.html

Scary false positive or story about Best practice to secure your root AWS account

What the best practice of securing AWS root account? - Not using it at all!

Let's clean it up first:

1. remove any API key associated with root account
2.  reset root password and change email assoicated
3. enable MFA (or deactivate previous and create new) on the root account.

Start using IAM:

1. Copy/bookmark/save IAM sign-in url 
2. create required users including one with AdministratorAccess policy attached. 
3. Enable MFA on all users created

Secure root account:

1. Print you root account credentials.
2. Log in using printed credentials to ensure that it works.
3. Put in tamper evident envelope
4. Add some signatures, stamps or voodoo on envelope. 
5. Hide it in SafeBox
6. Use it only in case of emergency :-)  

Now let's add some monitoring just in case:

1. Enable and configure CloudTrail + bucket for Logs
2. Configure CloudWatchLogs (CloudWatch) to process CloudTrail logs
3. Add metric filters to detect root-user related events
4. Set-up alarm and notifications (SNS) for the metrics

For root users  CloudWatchLog metric filter looks like:

Filter Name:

Security-CloudWarchAlarms-RootAccessMetricFilte

Filter Pattern:

{$.userIdentity.type = "Root"}

I did everything mentioned above and was ,let's say, "surprised" to get months after notification saying "Root log-in  detected" . Checked CloudTrail looking for  the root user - nothing....Hmm.. Start looking into CloudTrailLogs content  for the detailed row events and found this:

"eventVersion": "1.02", "userIdentity": { "type": "Root", "principalId": "577343344455", "arn": "arn:aws:iam::577343344455:root", "accountId": "5577343344455", "userName": "my_company", "invokedBy": "support.amazonaws.com" }, "eventTime": "2016-03-22T19:22:23Z", "eventSource": "iam.amazonaws.com", "eventName": "GetAccountSummary", "awsRegion": "us-east-1", "sourceIPAddress": "support.amazonaws.com", "userAgent": "support.amazonaws.com", "requestParameters": null, "responseElements": null, "requestID": "675d-fxx3-1x5-9xxd-4768xxx17", "eventID": "b9xxxxfcaf-3xx7-4xxd-a220-exxxx8", "eventType": "AwsApiCall",  "recipientAccountId": "577343344455"

Dear AWS support - you got me :-))