S3 bucket policies for sensitive security logs storage

Inspired by this AWS  blog post : How to Restrict Amazon S3 Bucket Access to a Specific IAM Role

Goal:
Build a storage for sensitive security logs using S3 bucket.

Restrictions:  

• EC2 instances could only upload logs. 
• Infosec team could only download logs and (just for this particular case) delete them with MFA .
•  All other user must not have any access despite whatever mentioned in their IAM policies.

 Solution:
custom bucket policy


        "PolicyDocument": {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Sid": "OnlyForInfosecEyes",
              "Effect": "Deny",
              "Principal":"*",
              "Action": ["s3:GetObject*", "s3:Delete*", "s3:PutObjectAcl", "s3:PutObjectVersionAcl"],
              "Resource": "s3-top-secret-bucket/*",
              "Condition": {
                "StringNotLike": {
                  "aws:userId":  "InfosecGroupUserIDs"
                }
              }
            },
            {
              "Sid": "OnlyServerAllowToPut",
              "Effect": "Deny",
              "Principal":"*",
              "Action": ["s3:PutObject"],
              "Resource": "s3-top-secret-bucket/*",
              "Condition": {
                "StringNotLike": {
                  "aws:userId":  "SeverIAMRoleID:*"
                }
              }
            },
            {
              "Sid": "EnforceEncryption",
              "Effect": "Deny",
              "Principal":"*",
              "Action": ["s3:PutObject"],
              "Resource": "s3-top-secret-bucket/*",
              "Condition": {
                "Null": {
                  "s3:x-amz-server-side-encryption": "true"
                }
              }
            },
            {
              "Sid": "EnforceMFADelete",
              "Effect": "Deny",
              "Principal":"*",
              "Action": ["s3:Delete*"],
              "Resource": "s3-top-secret-bucket/*",
              "Condition": {
                "Null": {
                  "aws:MultiFactorAuthAge": true
                }
              }
            }
          ]
        }

Where:

InfosecGroupUserIDs - list of IAM infosec users' IDs (aws iam get-user -–user-name USER-NAME)

SeverIAMRoleID:* - ID of the IAM role used by the your EC2 server instances with ":*" added to cover all instances in this role (aws iam get-role -–role-name ROLE-NAME.)

AWS EC2 status check alarms using python and boto3

Important part of security that we (infosec guys) often delegate :-)  to the Operation teams(NOC) is Availability.
       For the IaaS service provider (Amazon AWS) is responsible for Infrastructure availability, but we must design all layers above ( Availability Zones, VPCs, Networks, Instances and LB ) for high availability or at least fault tolerance. One of the most important step in this process is actually detection IaS failure.
From AWS:
"With instance status monitoring, you can quickly determine whether Amazon EC2 has detected any problems that might prevent your instances from running applications. Amazon EC2 performs automated checks on every running EC2 instance to identify hardware and software issues. You can view the results of these status checks to identify specific and detectable problems."

Below simple python script that will help you to configure status check alarms for all you running instances:

#!/usr/bin/python

import boto3
import pprint

boto3.setup_default_session(profile_name='staging', region_name='eu-west-1')
ec2 = boto3.resource('ec2')
cloudwatch=boto3.resource('cloudwatch')

# Getting all running instances
instance_iterator = ec2.instances.all()
for instance in instance_iterator:
    instance_name = "unnamed"
    for tag in instance.tags:
        if tag['Key'] == "Name":
            instance_name = tag['Value']
    print instance_name, instance.id
    if instance.state["Name"] == "running" :
        metric = cloudwatch.Metric("AWS/EC2", "StatusCheckFailed")
        response = metric.put_alarm(
        AlarmName = instance.id + "/" + instance_name + "-status-alarm",
        AlarmDescription = 'status check for %s %s' % (instance.id, instance_name),
        ActionsEnabled = True,
        OKActions = ["arn:aws:sns:eu-west-1:your_account_id:YOUR_SNS-EmailSMS-Notification"],
        AlarmActions = ["arn:aws:sns:eu-west-1:your_account_id:YOUR_SNS-EmailSMS-Notification"],
        Statistic = "Maximum",
        Dimensions = [{'Name': 'InstanceId', 'Value': instance.id}],
        Period = 60,
        EvaluationPeriods = 2,
        Threshold = 1.0,
        ComparisonOperator = "GreaterThanOrEqualToThreshold"
        )
        pprint.pprint(response)

AWS s3 bucket encryption audit (Updated)

Tool, mentioned in my previous blog post article got some new functionality:

https://github.com/IhorKravchuk/it-security/blob/master/s3_enc_check.py

1. batch mode.

$ python s3_enc_check.py --bucket com-company-prod-data-backup --profile prod-read

will check the bucket mentioned and give you the option to save to file or print report on screen

$ s3_enc_check.py --bucket com-company-prod-data-backup --profile prod-read  --file test_results.txt 

will check the bucket mentioned and save report to the file. Very useful for the large buckets with thousands of objects

2. Interactive mode.

run tool, specifying just AWS profile name, and it will scan your account for s3 bucket available and let you choose one for detailed audit.

$ python s3_enc_check.py --profile staging

3. Ability to check if encryption is enforced on the bucket level using AWS bucket policy.

Whatever way you start the tool, it will verify if bucket/buckets has s3 server side encryption enforced:

Amazon AWS Account level security checklist and how-to

Disclaimer :-):
There are bunch of Amazon AWS security checklists and recommendations online. Definitely the best one is https://d0.awsstatic.com/whitepapers/compliance/AWS_CIS_Foundations_Benchmark.pdf 
I'm not trying to reinvent the wheel, but integrate and summarize lessons I learned and advices given to me by other AWS experts.

This checklist starts from the moment when you begin AWS account creation.

1.  Create dedicated email address for AWS account registration. This email will become you root account login name, so, please, do not use your daily used or published online email
2. Enable MFA  (Multi Factor Authentication) on the root account. Details: http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_mfa.html
3. Remove or DO NOT create any API key associated with root account. API keys has no MFA - anyone who has root API keys gets full  access to you account. Unintentional leaking of the API key quite common security incident.
4. Copy/bookmark/save IAM sign-in url. You will need to access you AWS Web GUI.
5. Create IAM user with  AdministratorAccess policy attached. It will be your new  "root" like account.
6. Create other IAM users required. Minimize their permission using built-in AWS managed policies like: PowerUserAccess; ReadOnlyAccess; AmazonEC2FullAccess , etc
7. Enable MFA on all users created.  Details: http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_mfa.html
8. Enforce strict password policy. Details: http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_passwords_account-policy.html
9. Generate API keys for users who needs it. For "high-power" user make this keys inactive. They will activate keys through MFA protected AWS Web GUI only when it needed.
10. Do not use API keys in applications running inside AWS. Use IAM roles instead. Details: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
11. Enable and configure CloudTrail  for all regions  + s3 bucket for the CloudTrailLogs.  Details: http://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-create-a-trail-using-the-console-first-time.html
12. Send CloudTrails Events to the CloudWatch Logs. Details: http://docs.aws.amazon.com/awscloudtrail/latest/userguide/send-cloudtrail-events-to-cloudwatch-logs.html
13. Configure monitoring of the CloudTrail Log Files using Amazon CloudWatch Logs metric filters and alarms. Details: http://docs.aws.amazon.com/awscloudtrail/latest/userguide/monitor-cloudtrail-log-files-with-cloudwatch-logs.html
14. Configure near-real time Log data processing using Subscriptions or/and using lambda function.  Details: http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/Subscriptions.html
15. Using #13 and 14 configure notification for suspicions events
16. Enable AWS Config Service to get AWS configuration snapshots and change notifications. Details: http://docs.aws.amazon.com/config/latest/developerguide/gs-console.html
17. Enable and configure AWS VPC flow logs to get visibility on network level. Details: http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/flow-logs.html
18. Enforce server side encryption on your S3 buckets: Details: http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html
19. Enable encryption on you EBS volumes: Details: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSEncryption.html

Almost all steps covered above could and must be automated. I already published and will publish more automation examples in this blog.

Check your resulted account security status:

• Check Amazon AWS Trusted advisor 
• Run AWS Credentials Report 
• Run awesome AWS audit Scout2 tool : https://isecpartners.github.io/Scout2/

And do this periodically. 

Checklists and Best Practices:

AWS CIS Foundations Benchmark (must read document)
https://d0.awsstatic.com/whitepapers/compliance/AWS_CIS_Foundations_Benchmark.pdf

AWS Auditing Security Checklist
https://d0.awsstatic.com/whitepapers/compliance/AWS_Auditing_Security_Checklist.pdf

PS. I would like to thank Liem aka Pimpon  for advices in preparing this checklist.

AWS "one-liners": Configure AWS password policy in one shot

"As soon as you have passwords you need a password policy" - © captain obvious

Limitations:
AWS allows you to have only one password policy for whole AWS account.

You can configure it using web GUI or, if you prefer to have all your infrastructure and security as code, using boto and python:

#!/usr/bin/python

import boto3
import pprint

boto3.setup_default_session(profile_name='staging')
iam=boto3.resource('iam')
account_password_policy = iam.AccountPasswordPolicy()
response = account_password_policy.update(
    MinimumPasswordLength=12,
    RequireSymbols=True,
    RequireNumbers=True,
    RequireUppercaseCharacters=True,
    RequireLowercaseCharacters=True,
    AllowUsersToChangePassword=True,
    MaxPasswordAge=90,
    PasswordReusePrevention=12,
    HardExpiry=False
)

pprint.pprint(response)

You can find more details about particular password policy parameters here:

http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_passwords_account-policy.html

Scary false positive or story about Best practice to secure your root AWS account

What the best practice of securing AWS root account? - Not using it at all!

Let's clean it up first:

1. remove any API key associated with root account
2.  reset root password and change email assoicated
3. enable MFA (or deactivate previous and create new) on the root account.

Start using IAM:

1. Copy/bookmark/save IAM sign-in url 
2. create required users including one with AdministratorAccess policy attached. 
3. Enable MFA on all users created

Secure root account:

1. Print you root account credentials.
2. Log in using printed credentials to ensure that it works.
3. Put in tamper evident envelope
4. Add some signatures, stamps or voodoo on envelope. 
5. Hide it in SafeBox
6. Use it only in case of emergency :-)  

Now let's add some monitoring just in case:

1. Enable and configure CloudTrail + bucket for Logs
2. Configure CloudWatchLogs (CloudWatch) to process CloudTrail logs
3. Add metric filters to detect root-user related events
4. Set-up alarm and notifications (SNS) for the metrics

For root users  CloudWatchLog metric filter looks like:

Filter Name:

Security-CloudWarchAlarms-RootAccessMetricFilte

Filter Pattern:

{$.userIdentity.type = "Root"}

I did everything mentioned above and was ,let's say, "surprised" to get months after notification saying "Root log-in  detected" . Checked CloudTrail looking for  the root user - nothing....Hmm.. Start looking into CloudTrailLogs content  for the detailed row events and found this:

"eventVersion": "1.02", "userIdentity": { "type": "Root", "principalId": "577343344455", "arn": "arn:aws:iam::577343344455:root", "accountId": "5577343344455", "userName": "my_company", "invokedBy": "support.amazonaws.com" }, "eventTime": "2016-03-22T19:22:23Z", "eventSource": "iam.amazonaws.com", "eventName": "GetAccountSummary", "awsRegion": "us-east-1", "sourceIPAddress": "support.amazonaws.com", "userAgent": "support.amazonaws.com", "requestParameters": null, "responseElements": null, "requestID": "675d-fxx3-1x5-9xxd-4768xxx17", "eventID": "b9xxxxfcaf-3xx7-4xxd-a220-exxxx8", "eventType": "AwsApiCall",  "recipientAccountId": "577343344455"

Dear AWS support - you got me :-))

AWS s3 bucket encryption audit

Storing sensitive information at AWS S3?- it's a must to encrypt your data at rest.
How?

• do it yourself (client side encryption) and transfer to S3 already encrypted
• ask AWS to do it for you (server side encryption). In this case you have 2 options: S3 managed encryption keys or KMS-managed encryption keys.

If you create a new bucket for sensitive data NEVER create it without AWS bucket  policy enforcing encryption: encryption is object level attribute at S3 and user specify (technically request) encryption during upload process. Policy will block all uploads if encryption not requested. Simple and Easy.. Except:

      You have existing S3 bucket with data uploaded before you enable this policy, you have mixed (encrypted and non encrypted objects) or just doing security audit. In this case you need to scan the bucket to find unencrypted objects. How? quite easy using  few python lines bellow:


import boto3
import pprint
import sys
boto3.setup_default_session(profile_name='prod')
s3 = boto3.resource('s3')
if len(sys.argv) < 2:
   print "Missing bucket name"
   sys.exit
bucket = s3.Bucket(sys.argv[1])
for obj in bucket.objects.all():
   key = s3.Object(bucket.name, obj.key)
   if key.server_side_encryption is None:
       print "Not encrypted object found:", key

Nice, Yep, But it will take almost forever to scan bucket that contains thousand or tens of thousand of objects. In this it would be nice to have some counters, progress bar, ETA , summary, etc.. So, vuala:

https://github.com/IhorKravchuk/it-security/blob/master/s3_enc_check.py


Small program providing all these features mentioned. Feel free to use it or request reasonable changes/modifications.