Using AWS Config, Lambda and Splunk to build detective controls for AWS.

The key AWS document that helps clients to succeed in Cloud Adoption -  AWS Cloud Adoption Framework, in the Security Perspective section defines Detective Controls as following:

• Detective Control provides guidance to help identify potential security incidents within your AWS environment.

AWS Well Architect Framework adds  to that:

•  You can use detective controls to identify a potential security threat or incident. They are an essential part of governance frameworks and can be used to support a quality process, a legal or compliance obligation, and for threat identification and response efforts. 
• In AWS, you can implement detective controls by processing logs, events, and monitoring that allows for auditing, automated analysis, and alarming

Below I will describe implementations of the AWS Detective controls using native AWS and 3d party services :

• AWS Config and Config Rules (Managed and Custom)
• Lambda 
• CloudWatch Events a.k.a EventBridge 
• Firehose
• Splunk Cloud

Our Goal is: Build a set of the automated detective controls for the multi-account distributed AWS environment, along with automatic remediation, compliance dashboards, a single pane of glass for security events, and notifications. 

Let's start with collecting our requirements:

• All components of the solution must be represented as a code 
• Serverless Application Model
• Maximum usage of the native AWS services 
• 3d party components should be pluggable
• fully distributed architecture with no critical central components.
• Event-driven architecture
• Near-real time event processing and ingestion
• Resource whitelisting (via ARN or/and resource tag) support

Major components:

• AWS Config Service
• Managed (by AWS) config rules 
• Custom (Lambdas, created by customer) config rules
• EventBridge event rule
• Processing Lambda
• Firehose (delivery2Splunk)
• S3 Buckets
• IAM Roles
• DynamoDB tables
• Splunk with HEC configured
• AWS Security Hub Service 

Event flow and processing: 

1. AWS Config rule evaluation starts due to: 
• AWS resource in the scope of the config rule being created/modified/deleted
• new config rule has been deployed
• schedule-driven rule evaluation started
• on-demand evaluation of the rules has been triggered via Web UI, API or CLI
2. Rule evaluation completed and resource compliance status is changed to COMPLIANT | NON_COMPLIANT | NOT_APPLICABLE 
3. Event  ComplianceChangeNotification generated: it generates when the compliance type of a resource that AWS Config evaluates has changed
4. EventBridge event rule will invoke processing Lambda.
5. Processing Lambda will:
• extract all required fields from the EventBridge event.
• create a data structure that suitable for the Splunk ingestion via Splink HEC
• add custom Splunk fields that could be defined at index time with AWS account metadata: AWS Account ID, Account Name,  AWS organization, environment, customer, etc
• Enrich this data structure with information from the central config rule metadata DynamoDB table: Rule severity, Description, associated compliance framework name and section, etc. 
• To fetch information from the DynamodB, processing Lambda will assume a role in the "Security/Log Archive" account that will grant only "Read" access to the required DynamoDB tables.
• Call AWS config service to retrieve additional information about AWS resource, which compliance status has been changed, such as recourse name, ARN, all available Tags, etc
• Enrich existing event data structure with information obtained from the config.
• Fetch resource whitelisting status (whitelisted or not, the reason for whitelisting, whitelisted by whom and when) from the central DynamoDB table of the whitelisted resource using resource particular resource Tag and /or resource ARN.
• Enrich existing event data structure with whitelisting information.
• Send the event, enriched on previous steps, to the Kinesis Data Firehose configured with Splunk HEC as destination and central S3 bucket as backup storage.
• Build a new data structure that corresponds to the AWS Security Finding Format (ASFF)
• Adjust event severity based on the whitelisting status
• Fetch additional security context from different AWS services that might affect security event severity and incorporate it into the ASFF data structure.
• Send ASFF event to the Security Hub
6. Kinesis Data Firehose delivers the event to the Splunk HTTP Event Collector (HEC) endpoint for the indexing.
7. SecurityHub in the administrator(master) account will serve as a single pane of glass for the Global Security team along with Splunk.
8. Security Hub in the member account could be useful for the customer(the account owner) to monitor and address security findings in his account.  

Highlights of the solution:

• event-driven via EventBridge
• uses compliance status change as a trigger to process event
• distributed (AWS account and region) processing Lambda
• processing leverages AWS Config Service to extract additional information about the resource itself (ARN, all Tags, resource name, etc)
• verifies the whitelisting status of the resource  (by ARN or dedicated Tag) using ReadOnly access to secure centralized DB
• obtains additional details about the config rule, that triggered resource compliance status change, from the centralized DynamoDB table: AWS config rule enable/disable status,  severity, event routing information(2Splunk, 2SecurityHub, 2PagerDuty), rule details, etc.
• enriches data event with information obtained from the whitelisting and config rules metadata tables
• automatically adjust event severity based on the whitelisting status
• send the enriched event to Splunk via Firehose
• send the enriched event to the Security Hub using AWS Security Finding Format (ASFF)
• architecture could be extended to accommodate auto-remediation flow
• could serve as an integration point for any 3d party logging/alerting or ticketing tool.  

Aws basic account-level hardening

Cloudformation template to enable basic AWS account level security: Cloudtrail, AWS Config, Cloudwatch Alarms on security events, etc.

The very first thing you need to do while building your AWS infrastructure is to enable and configure all AWS account-level security features such as CloudTrail, CloudConfig, CloudWatch, IAM, etc. To do this, you can use my Amazon AWS Account level security checklist and how-to or any other source. To avoid manual steps and to be aligned with the SecuityAsCode concept, I suggest using a set of the CloudFormation templates which will provide the following functionality:

• configures CloudTrail according to the new best practices (KMS encryption, validation, etc)
• configures AWS Config service and creates a basic set of the CloudConfig rules to monitor best practices
• implements Section 3 (Monitoring) of the CIS Amazon Web Services Foundations Benchmark.

Launch template now: 

Global Security stack template structure:

security.global.yaml - parent template for all nested templates to link them together and control dependency between nested stacks.

cloudtrail.clobal.yaml - nested template for Global configuration of the CloudTrail

awsconfig.global.yaml - nested template for Global AWS Config Service configuration and config rules.

cloudwatchalarms.global.yaml - nested template for Global CloudWatch Logs alarms and security metrics creation. Uses FilterMap to create different security-related filters for ClouTrail LogGroup, corresponding metrics, and notifications for suspicious or dangerous events. You can customize a filter on a per-environment basis.

Input parameters:

• CFtemplateBucketURL: URL of the CloudFromation templates to use (Normally in the s3 bucket). This parameter will be prepopulated with value: https://s3.amazonaws.com/secureincloud.ca/aws/

• Bucket4Logs : Name of the new bucket that will be created to collect cloudtrail and config logs

• LogRetentionDays : Amount of days to store the logs in S3 bucket. Default 365 or 1 year

• AWSAccountName : AWS Account nickname(purpose). User-Friendly name(purpose) of your AWS account. Will be used in the names of the CloudWatch Alarms.

• InfosecEmail : Email of the infosec team to send security-related alerts from the CloudWatch Alerts

• DevOpsEmail : Email of the DevOps team to send operations-related alerts from the CloudWatch Alerts

AWS Managed Config Rules deployed by template:

• iam-user-no-policies-check Description: Checks that none of your IAM users have policies attached. IAM users must inherit permissions from IAM groups or roles
• root-account-mfa-enabled Description: Checks whether the root user of your AWS account requires multi-factor authentication for console sign-in.
• s3-bucket-public-read-prohibited Description: Checks that your S3 buckets do not allow public read access. If an S3 bucket policy or bucket ACL allows public read access, the bucket is noncompliant.
• s3-bucket-public-write-prohibited Description: Checks that your S3 buckets do not allow public write access. If an S3 bucket policy or bucket ACL allows public write access, the bucket is noncompliant.
• restricted-ssh Description: Checks whether security groups that are in use disallow unrestricted incoming SSH traffic.
• iam-password-policy Description: Checks whether the account password policy for IAM users meets the specified requirements.

AWS CIS Checks covered by the template (implemented via CloudWatch Alert mechanism ):

• AWS CIS 3.01 Ensure a log metric filter and alarm exist for unauthorized API calls
• AWS CIS 3.02 Ensure a log metric filter and alarm exist for Management Console sign-in without MFA
• AWS CIS 3.3 Ensure a log metric filter and alarm exist for usage of Root account
• AWS CIS 3.4 Ensure a log metric filter and alarm exist for IAM policy changes
• AWS CIS 3.5 Ensure a log metric filter and alarm exist for CloudTrail configuration changes
• AWS CIS 3.6 Ensure a log metric filter and alarm exist for AWS Management Console authentication failures
• AWS CIS 3.7 Ensure a log metric filter and alarm exist for disabling or scheduled deletion of customer-created CMKs
• AWS CIS 3.8 Ensure a log metric filter and alarm exist for S3 bucket policy changes
• AWS CIS 3.9 Ensure a log metric filter and alarm exist for AWS Config configuration changes
• AWS CIS 3.10 Ensure a log metric filter and alarm exist for security group changes
• AWS CIS 3.11 Ensure a log metric filter and alarm exist for changes to Network Access Control Lists (NACL)
• AWS CIS 3.12 Ensure a log metric filter and alarm exist for changes to network gateways
• AWS CIS 3.13 Ensure a log metric filter and alarm exist for route table changes
• AWS CIS 3.14 Ensure a log metric filter and alarm exist for VPC changes

Custom checks implemented via CloudWatch Alert mechanism:

• Custom: Alarms when a large number of sensitive (Start. Stop, Terminate, Reboot Instance) operations are performed in the short time period
• Custom: Alarms when a large number of Instances are being terminated
• Custom: Alarms when a volume is a force detached from an Instance
• Custom: Alarms when VPC traffic flow is created or deleted

Some important notes:

Many Cloud Security professionals might suggest that using CloudWatchAlarm as security detective control is a bit outdated. AWS has way more robust native mechanisms now like 10th of managed AWS  Config Rules, CIS, PCI, and AWS best practices standards (and associated checks) for the Security Hub. 

In addition, you can leverage nice 3d part tools like Splunk or Sumologic to have way more sophisticated detective controls.

This is true... But each of these mechanisms has significant extra costs associated. AWS Security becomes quite expensive when you leverage AWS Config Rules or Security Hub at scale. Cloudwatch Alarm on the contrary is quite cheap. 

More over CloudWatch Alarm-based security controls rely on the most robust, reliable, and fundamental AWS services and should be used as a 3d layer of your security defense to protect you in case of failure of 3d part or even more complex native AWS security mechanisms. 

Feel free to extend this list with your custom checks as per examples provided in the template and below:

```
rds-change:
  all: '{$.eventName = CopyDB* || $.eventName = CreateDB* || $.eventName = DeleteDB*}'

srt-instance:
  all: '{($.eventName = StopInstances || $.eventName = TerminateInstances || $.eventName
    = RebootInstances)}'

large-instance:
  all: >-
    { (($.eventName = RunInstances) || ($.eventName = StartInstances)) && (($.requestParameters.instanceType
    = *.2xlarge) || ($.requestParameters.instanceType = *.4xlarge) || ($.requestParameters.instanceType
    = *.8xlarge) || ($.requestParameters.instanceType = *.10xlarge)) }

change-critical-ebs:
  prod: >-
    {($.eventName = DetachVolume || $.eventName = AttachVolume || $.eventName
    = CreateVolume || $.eventName = DeleteVolume || $.eventName = EnableVolumeIO
    || $.eventName = ImportVolume || $.eventName = ModifyVolumeAttribute) && ($.requestParameters.volumeId
    = vol-youvol1ID || $.requestParameters.volumeId = vol-youvol2ID)}

create-delete-secgroup:
  all: >-
    {$.eventName = CreateSecurityGroup || $.eventName = CreateCacheSecurityGroup
    || $.eventName = CreateClusterSecurityGroup || $.eventName = CreateDBSecurityGroup
    || $.eventName = DeleteSecurityGroup || $.eventName = DeleteCacheSecurityGroup
    || $.eventName = DeleteClusterSecurityGroup ||  $.eventName = DeleteDBSecurityGroup}

secgroup-instance:
  all: '{$.eventName = ModifyInstanceAttribute && $.requestParameters.groupSet.items[0].groupId
    = * }'

cloudformation-change:
  all: '{$.eventSource = cloudformation.amazonaws.com && ($.eventName != Validate*
    && $.eventName != Describe* && $.eventName != List* && $.eventName != Get*)}'

critical-instance:
  prod: >-
    {$.requestParameters.instanceId = i-instance1ID || $.requestParameters.instanceId
    = i-instance2ID || $.requestParameters.instanceId = i-instance3ID || $.requestParameters.instanceId
    = i-instance4ID || $.requestParameters.instanceId = i-instance5ID || $.requestParameters.instanceId
    = i-instance6ID|| $.requestParameters.instanceId = i-instance7ID}

eip-change:
  all: '{$.eventName = AssociateAddress || $.eventName = DisassociateAddress ||
    $.eventName = MoveAddressToVpc || $.eventName = ReleaseAddress }'

net-access
all: >-
    {$.sourceIPAddress != 111.222.3* && $.sourceIPAddress != 111.222.4* && $.sourceIPAddress
    != cloud* && $.sourceIPAddress != AWS* && $.sourceIPAddress != 11.22.33.00
    && $.sourceIPAddress != 11.22.33.01 }

```

AWS IAM 101/201 and security notes.

Let's start from basic: What's is AWS IAM?

AWS Identity and Access Management (IAM) is a web service that helps you securely control access to AWS resources. You use IAM to control who is authenticated (signed in) and authorized (has permissions) to use resources.

What does exactly IAM provide:

• Shared access to your AWS account
• Granular permissions
• Secure access to AWS resources for applications that run on Amazon EC2
• Multi-factor authentication (MFA)
• Identity federation
• Identity information for assurance
• PCI DSS Compliance (debatable, IMHO) 
• Integrated with many AWS services
• Eventually Consistent
• Free to use

Do all AWS services work with IAM? 

Not exactly: Here the list:

IAM currently supports the following authorization models:

• Role-based access control (RBAC). RBAC defines permissions based on a person's job function, known outside of AWS as a role. 
• Attribute-based access control (ABAC) is an authorization strategy that defines permissions based on attributes. In AWS, these attributes are called tags. Tags can be attached to IAM principals (users or roles) and to AWS resources. 

AWS IAM principals:

- User

- Roles

- Groups 

Deployment time security audit using CloudFormation custom resource.

To prevent deployment of the potentially sensitive resources or infrastructure into the AWS account that might not meet current organizational security standard we can use AWS CloudFormation custom resource to perform quick security audit (or kind of sanity check) of the cloud account before processing with deployment.

Why we need this if we can scan/audit account as a part of the CI/CD pipeline? For the cases when deployments are performed manually or to have CI/CD independent "portable" CloudFormation template that has all security checks built-in and not bolt-on.

How it will look like:

1. To you normal CloudFormation template you will add a custom resource.
2. This custom resource it technically speaking a Lambda function that created and called during CloudFormation stack deployment.
3. This Lambda function will perform quick (to meet CloudFormation deployment timeouts restrictions) security audit of the account where template going to be deployed
4. As result of this audit Lambda will return status that will be interpreted by CloudFormation as a resource creation outcome.
5. If AWS environment pass security check - deployment of other resources in you stack continue as usual
6. If AWS environment fail security check - stack deployment will interrupted and rolled back as a result of the custom resource failure. 

I will publish example on such functionality on my Github shorty and will update this post with more details. 

Nmap.me new version - now with vulnerability scan

Now service:

• Does full port and vulnerability scan of the caller's IP
• Immediately perform and return tcp scan results. Starts vulnerability scan in background.
• Visit nmap.me again in about 30min(depends on load), and results of the vulnerability scan (port, service, vulnerability, CVE) for you IP will be displayed. 
• You can use, console friendly endpoint : scan.nmap.me , or
•  use a human readable website: nmap.me

Scan results examples:

Good  scan results:

Not so good scan  results:

As previously service is built using AWS native capabilities, serverless and containerized approach and design to be extremely scalable.

Quick FAQ:

What's new? Now it does full vulnerability scan of the IP.

What it does? Scan your external IP for open TCP ports and known vulnerabilities.

How to use? Simply do _curl scan.nmap.me_ from your console/terminal or open this webite in browser. You will get TCP scan results immediately and you will need to visit same page in an hour to get vulnerability scan results.

What it's scanning for?: Uses nmap NSE script to perform scan for known vulnerabilities. Based on https://github.com/vulnersCom/nmap-vulners

How fast? Whole TCP scan takes about a few second and results are immediately shown. After this vuln scan is starting. Depending of the backend load it might take about an hour to get scanned. After this simply visit the same page again to get vulnerability scan results. Scan results for each requester IP are cached for 1 hour (TCP scan) and 24 hours (Vuln scan) to reduce load and prevent abuse.

Awesome list of Native AWS logging capabilities

While looking on centralized logging  capabilities of AWS and going trough bunch of documentation, I noticed lack of the one "big table" where I can find all AWS native logging capabilities per each service and up-to-date service coverage for AWS cloudwatch logs service.
Building I big table is not really version control friendly, so please welcome:

Awesome list of Native AWS logging capabilities:
https://github.com/IhorKravchuk/it-security/blob/master/AWS_logging.md

While I was building this list, some service have already changed their capabilities causing some information in the list being out-of-sync.
 I'll try my best to regularly review existing services and keep adding new one, but if you find mistake or would like to contribute feel free to contact me or create a PR.

Revamping AWS APIs' security review and SCP policy generation process.

AWS Cloud provides endless amount of the capabilities and services. Unleashing all this power on the without proper security review process is extremely risky.
Each service and quite often even each api call should be reviewed and evaluated according to the organizational security standards  and compliance requirements. Yes, but.. curently AWS has about 170 services and endless amount of APIs. AWS constantly evolves, introduce new services, APIs and modifying existing.
One of the biggest challenge for me was finding a way to automatically fetch up-to-date annotated  list of the services and api provided by AWS. Luckily, Matt Weagle suggested to use AWS GO SDK as a source of truth. This SDK provides well documented lists of the AWS APIs (docs-2.json)

I crafted small python program that builds/updates following yaml files (one per each service) using json files as a source:

guardduty:

  description: Assess, monitor, manage, and remediate security issues across your

    AWS infrastructure, applications, and data.

  links: [http://guardduty.docs.here]

  security_risk: Cloud IDS

  Allowed_on

  - Prod_en

  Denied_on:

  - none

AcceptInvitation:

  description: Accepts the invitation to be monitored by a master GuardDuty account.

  links: [https://awsdocs.com]

  security_risk: should be allowed only from trusted accounts

  Allowed_on:

  - none

  Denied_on:

  - none

ArchiveFindings:

  description: Archives Amazon GuardDuty findings specified by the list of finding

    IDs.

  links: []

  security_risk: Not defined

  Allowed_on:

  - none

  Denied_on:

  - none

Structure of this file is quite self-explanatory and simplifies security review(still manual process) of the AWS APIs. During security review,  you specify which services/api are enabled/disabled and on which environments by adding environment name to the Allowed_on  and Denied_on lists. Files are stored in the git repo.

After the review, using these files as a source of truth, I (actually another python program) generate an SCP (Service Control Policy) for AWS Organization's accounts, IAM policies and permission boundaries (it depends on the case.)
Due to the very strict SCP size restrictions , generating this policy using automation allows you:

• aggregate APIs using wildcards to reduce SCP size
• validate API wildcards preventing unintentional service exposure/blockage
• perform cross check for the API to avoid whitelisting/blacklisting conflicts
• re-generate/validate SCP if AWS introduces new API calls/services

Everything mentioned above is valid not only for the SCP, but for the IAM policy/permission boundaries generation process.

This automated approach opens another possibility - automated compliance validation for AWS: using the same yaml files as a source of truth ,  perform API calls to the AWS to ensure that these calls will fail. This step could be done after deployment (to validate deployment) or on a regular basis(audit).

PS. Unfortunately code of the tools can't be open-sourced as of now.

- 

Secure your AWS account using Terrafrom and CloudFormation

This is very updated version of the blog post: http://blog.it-security.ca/2016/11/secure-your-aws-account-using.html

As I mention before:
The very first thing you need to do while building your AWS infrastructure is to enable and configure all AWS account level security features such as: CloudTrail, CloudConfig, CloudWatch, IAM, etc.

Time flies when you're having fun and flies even faster in the infosec world. My templates become outdated and now I'm presenting an updated version of the AWS security automation with following new features:

1. integrated with Terraform (use terraform templates in the folder tf)
2. creates prerequisites for Splunk integration (User, key, SNS, and SQS)
3. configures cross-account access (for multiaccount organizations, adding ITOrganizationAccountAccessRole with MFA enforced)
4. implements Section 3 (Monitoring) of the CIS Amazon Web Services Foundations benchmark.
5. configures CloudTrail according to the new best practices (KMS encryption, validation etc)
6. configures basic set of the CloudConfig rules to monitor best practices

First, my security framework now consists of two main parts: cf (CloudFormation) and tf (Terraform) with Terraform template as a bootstrapper of the  whole deployment.

You can use  Terraform, you can use CloudFormation, but why both ?
Terraform is very quickly evolves, has cross-cloud support and implements some missing in CloudFormation features (like account level password policy configuration, etc); CloudFormation is native for AWS, well supported, and, most important, AWS provides a lot of best practices and solutions in the form of the CloudFormation templates.

Using both (tf and cf) gives me (and you) ability to reuse solutions, suggested and provided by AWS, without rewriting the code, have flexibility and power of terraform and one single interface for whole cloud automation.
No more bucket pre-creation or specific sequence of the CloudFormation deployment - just terraform apply. It will take care of all CloudFormation prerequisites, version control and template updates.
But,  if you wish, at current state you can use only my CloudFormation templates - cf still does all heavy lifting.

The main trick of the Terraform - CloudFormation integration was to tell terrafrom when CloudFormation template is updated to ensure that terraform will trigger cf stack update.
I achieved this using S3 bucket with version control enabled and always updating (just setting template version) security.global.yaml.

This code takes care of Terraform and CloudFormation integration:
# creating Security cloudforation stack

resource "aws_cloudformation_stack" "Security" {
  name = "Security"
  depends_on = ["aws_s3_bucket_object.iam_global", "aws_s3_bucket_object.cloudtrailalarms_global", "aws_s3_bucket_object.awsconfig_global", "aws_s3_bucket_object.cloudtrail_global", "aws_s3_bucket_object.security_global"]
  parameters {
    AccountNickname = "${var.enviroment_name}",
    CompanyName = "${var.company_name}",
    MasterAccount = "${var.master_account}"
  }
  template_url = "https://s3.amazonaws.com/${aws_s3_bucket.CFbucket.bucket}/${var.security_global}?versionId=${aws_s3_bucket_object.security_global.version_id}"
  capabilities = [ "CAPABILITY_NAMED_IAM" ]
  tags { "owner" = "infosec"}
}

And finally deployment steps are:

1. Get code from my git repo:  https://github.com/IhorKravchuk/it-security
2. Switch to tf folder and update terraform.tfvars specifying: your AWS profile name (configured for aws cli using aws configure --profile profile_name); name for the environment (prod, test, dev ..) ; company(or division) name; region and AWS master account ID.
3. terraform init to get aws provider downloaded by terraform
4. terraform plan
5. terraform apply

Secure your AWS account using CloudFormation

      The very first thing you need to do while building your AWS infrastructure is to enable and configure all AWS account level security features such as: CloudTrail, CloudConfig, CloudWatch, IAM, etc..

       To do this, you can use mine Amazon AWS Account level security checklist and how-to or any other source.

        To avoid manual steps and to be align with SecuityAsCode concept, I use set of CloudFormation templates, simplified version of which I would like to share: 

GitHub: https://github.com/IhorKravchuk/it-security

Global Security stack template structure:

security.global.json - parent template for all nested templates to link them together and control dependency between nested stacks.

cloudtrail.clobal.json - nested template for Global configuration of the CloudTrail:

• creates S3 bucket for the logs
• creates CloudTrail-related IAM roles and policies
• creates CloudLog LogGroup
• enables CloudTrail on default region including global events and multi-region feature
• creates SNS ans SQS configuration for easy integration with Splunk AWS app.

cloudtrailalarms.global.json - nested template for Global CloudWatch Logs alarms and security metrics creation. Uses FilterMap to create different security-related filters for ClouTrail LogGroup, corresponding metrics and notifications for suspicious or dangerous events. You can customise filter per environment basis.

Predefined filters are:

• rds-change: RDS related changes
• iam-change; IAM changes
• srt-instance: Start, Reboot, Terminate instance
• large-instance: launching large instances
• massive-operations: massive operations- more 10 in 5 min 
• massive-terminations: massive terminations- more 10 in 5 min 
• detach-force-ebs: force detachment of the EBS volume from the instance
• change-critical-ebs: any changes related to the critical EBS volumes
• change-secgroup: any changes related to the security group
• create-delete-secgroup: creation and deletion of the security group 
• secgroup-instance: attaching security group to the instance
• route-change: routing  changes
• create-delete-vpc: creation and deletion of a VPC
• netacl-change: changes at Network ACL
• cloudtrail-change: changes in the CloudTrail configuration
• cloudformation-change: changes related to the CloudFormation
• root-access: any root access events
• unauthorised: failed and unauthorised operations
• igw-change: Internet Gateway related changes
• vpc-flow-logs: Delete or Create VPC flow logs
• critical-instance: any operation on the critical instances
• eip-change: Elastic IP changes
• net-access: Any access outside of predefined known IP ranges

4 preconfigured notification topics : 

• InfosecEmailTopic, 
• DevOpsEmailTopic
• InfosecSMSTopic
• DevOpsSMSTopic

awsconfig.global.json - nested template for Global AWS Config Service configuration.

• creates S3 bucket for the config dumps
• creates AWS Config-related IAM roles and policies
• creates AWS config delivery channel and schedule config dumps (hourly)
• creates and enables AWS config recorder 
• creates SNS ans SQS configuration for easy integration with Splunk AWS app.

cloudwatchsubs.global.json - nested template for configuring AWS CloudWatch Subscription Filter to extract and analyse most severe CloudTrail events using custom Lambda function:

• creates Lambda function and all requred roles and permissions
• creates Subscription filter as a compilation of the filters from the FilterMap
    

Currently uses following filters and aggregate them to one due to the AWS CloudWatch Logs subscription limitation (only one filter supported):

• critical-instance
• iam-change
• srt-instance
• cloudtrail-change
• root-access
• net-access
• detach-force-ebs
• unauthorised

iam.global.json - nested template for IAM Global configuration: 

• creates Infosec Team IAM Group and managed policy
• creates DevOps Team IAM Group and managed policy
• creates DBA Team IAM Group and managed policy
• creates Self Service Policy  for users to manage API keys and MFA
• creates  ProtectProdEnviroment to protect production environment from destructive actions 
• creates EnforceMFAPolicy to enforce MFA for sensitive operations
• creates EnforceAccessFromOfficePolicy to restrict some operation to office source IPs
• creates DomainJoin role and all required policy to perform automated domain join
• creates SaltMasterPolicy and Role for Configuration Management Tool (in this case - Salt)
• creates SQLDataBaseInstancePolicy and Instance profile example policy
• creates SIEM system example policy
• creates VPC flow log role
• creates and manages SIEM user and API keys
• creates and manages SMTP user (for the AWS SES service ) and API keys

cloudwatchsubs_kinesis.global.json - PoC template (not linked as nested to the security.global.json)  for configuring AWS CloudWatch Subscription Filter to send most severe CloudTrail events to AWS Kinesis stream using subscription filter similar to the cloudwatchsubs.global.json 

Supported features:

Environments and regions: Stack supports unlimited amount of environments with 4 environments predefined (staging, dev, prod, and dr) and use 1 account and 1 region per environment concept to reduce blast radius (if account become compromised)

AWS services used by stack: CloudTrail, AWS Config, CloudWatch, CloudWatch Logs and Events, IAM,  Lambda, Kinesis.

To deploy:

1. Create bucket using following naming convention: com.ChangeMe.EnviromentName.cloudform, replacing ChangeMe and EnviromentName with your value to make it look like this:            com.it-security.prod.cloudform
2. Enable bucket versioning 
3. in the templates  security.global.json and cloudwatchsubs.global.json replace "ChangeMe" with name used in the bucket creation.
4. In the template cloudtrailalarms.global.json modify SNS endpoints for email notification infosec@ChangeMe.com and devops@ChangeMe.com; Add endpoints with mobile phone numbers for SMS notification to appropriate SNS topics if needed.
5. Modify iam.global.json template to adrress you SQL DataBase bucket location (com-ChangeMe-", {"Ref": "Environment"} , "-sqldb/)  and modify any permission if need according to your organisation structure, roles, responsibilities and services.
6. Modify FilterMap in cloudtrailalarms.global.json and cloudwatchsubs.global.json templates make filters work for your infrastructure (Critical Instance IDs, Critical Volume IDs, you ofiice IP range, you NAT gateways, etc) 
7. Zip example Lambda function LogCritical_lambda_security_global.py like LogCritical_lambda_security_global.zip
8. Upload this function into S3 bucket created at step 1 and copy object version (GUI- show version -object properties ) and insert into cloudwatchsubs.global.json template into "LogCriticalLambdaCodeVer" mapping at the appropriate environment (prod, staging ..)
9. Modify "regions" Environments mapping in the iam.global.json, and cloudwatchsubs.global.json templates to specify correct AWS region you are using for the deployment.
10. Upload all *.global.json templates into S3 bucket created at step 1. 
11. Create new CloudFormation stack  using parent security template security.global.json and your bucket name (Example: ttps://s3.amazonaws.com/com.it-security.prod.cloudform/security.global.json ),  call it "Security" and specify environment name you going to deploy.
12. Done!

Self-Defending Cloud PoC or Amazon CloudWatch Events usage

Problem:  Malicious attacker get privileged access to your AWS account and destroying your production infrastructure in a matter of seconds.

In case of cloud based infrastructures you can't rely on classic SIEM solutions - by the time your SIEM will detect attack, your infrastructure will be gone. We need a "near real time" way to detect and mitigate the attack.

Attack scenario: using compromised aws api key (no MFA) and CLI/SDK to perform destructive actions.

Attack detection and mitigation strategy:  
All destructive actions start with EC2 instance termination. To prevent such scenario, you should always have "TerminationProtection" feature enabled on your production instances. Based on this, attacker must disable termination protection before demolishing your environment. For the PoC, I will use disabling "TerminationProtection" event as a attack detector (sure thing, real attack detection is a way more complicated process).

Starting point: AWS api key with admin policy attached, all production instances protected using AWS Termination Protection.

Possible solutions and attack mitigation delays:

SIEM:

CloudWatch Logs and alarms:

CloudWatch Subscriptions:

CloudWatch Events:

Implementation: 

Design:
Based on the implementation scenarios shown above and their performance (tested during PoC)  - the fastest way is to leverage AWS CloudWatch events and trigger a lambda function.

Speaking AWS technical language we need:

1. choose what type of AWS event we are looking for. Based on the our attack detection strategy, we are looking for ec2 call: modify-instance-attribute. Using exactly this API call you can enable/disable TerminationProtection. So let's look for "AWS API Call Events" type of the CloudWatch events.
2. Create: event rule : "match incoming events and route them to one or more targets for processing" in our case target is a Lambda function.
3. Create all required policies and Lambda function role in IAM.
4. Build the Lambda function itself.

Setting-up event rule:

I was using following event pattern inside the CloudWatch Event rule:
{ "detail-type": [ "AWS API Call via CloudTrail" ], "detail": { "eventSource": [ "ec2.amazonaws.com" ], "eventName": [ "ModifyInstanceAttribute" ] } }


Getting sample event:

To start writing our event-detection-mitigation lambda function we need to get example of the AWS events for the API call we are monitoring.
We can achieve this with the following simple Lambda function:

import json

def lambda_handler(event, context):
    print event


or, if you need nice formatted json to test you lambda function offline:

import json

def lambda_handler(event, context):
    print json.dumps(event, indent=4, sort_keys=False)

You will find output of your lambda function (result of the print statement) in appropriate (naming as your lambda function) CloudWatch Log Stream


Challenges with event format:

During first tests, I've found that Amazon AWS not following any JSON contract (defined format)  even for the same 1 API call. Making the same API call in the 3 different ways produced 3 different event formats:

Disabling TerminationProtection from GUI with MFA:

{u'account': u'150905', u'region': u'eu-west-1', u'detail': {u'eventVersion': u'1.05', u'eventID': u'b3c4d3b4-353e-44bf-8973-37abccd085b5', u'eventTime': u'2016-10-14T17:31:37Z', u'requestParameters': {u'instanceId': u'i-d8916d57', u'disableApiTermination': {u'value': False}}, u'eventType': u'AwsApiCall', u'responseElements': {u'_return': True}, u'awsRegion': u'eu-west-1', u'eventName': u'ModifyInstanceAttribute', u'userIdentity': {u'userName': u'ihork', u'principalId': u'AIDAI3UNW', u'accessKeyId': u'ASIAIN2', u'invokedBy': u'signin.amazonaws.com', u'sessionContext': {u'attributes': {u'creationDate': u'2016-10-14T16:48:44Z', u'mfaAuthenticated': u'true'}}, u'type': u'IAMUser', u'arn': u'arn:aws:iam::150905:user/igor', u'accountId': u'150905'}, u'eventSource': u'ec2.amazonaws.com', u'requestID': u'e7b585e-af38-49d0-88a8-979ef5052f', u'userAgent': u'signin.amazonaws.com', u'sourceIPAddress': u'174.231.5.2'}, u'detail-type': u'AWS API Call via CloudTrail', u'source': u'aws.ec2', u'version': u'0', u'time': u'2016-10-14T17:31:37Z', u'id': u'55084ea-e4bc-45e6-a7a6-0c8e7d16b32', u'resources': []}

Disabling TerminationProtection from aws cli (no MFA):

command:
$ aws ec2 modify-instance-attribute --no-disable-api-termination --instance-id i-378579b8 

event:
{u'account': u'150905', u'region': u'eu-west-1', u'detail': {u'eventVersion': u'1.05', u'eventID': u'f8ae9323-91b0-4100-b27b-dce348641a5c', u'eventTime': u'2016-10-14T17:31:46Z', u'requestParameters': {u'instanceId': u'i-d8916d57', u'disableApiTermination': {u'value': True}}, u'eventType': u'AwsApiCall', u'responseElements': {u'_return': True}, u'awsRegion': u'eu-west-1', u'eventName': u'ModifyInstanceAttribute', u'userIdentity': {u'userName': u'ihork', u'principalId': u'AIDAI3UNW', u'accessKeyId': u'ASIAIN2', u'invokedBy': u'signin.amazonaws.com', u'sessionContext': {u'attributes': {u'creationDate': u'2016-10-14T16:48:44Z', u'mfaAuthenticated': u'true'}}, u'type': u'IAMUser', u'arn': u'arn:aws:iam::150905:user/igor', u'accountId': u'150905'}, u'eventSource': u'ec2.amazonaws.com', u'requestID': u'cd889e-039e-4f8f-bfe9-4d293012335', u'userAgent': u'signin.amazonaws.com', u'sourceIPAddress': u'174.231.5.2'}, u'detail-type': u'AWS API Call via CloudTrail', u'source': u'aws.ec2', u'version': u'0', u'time': u'2016-10-14T17:31:46Z', u'id': u'cd32fc6-39ae-4237-b46d-62d237d4d89', u'resources': []}

Disabling TerminationProtection from aws cli. 2nd variant

command:

$ aws ec2 modify-instance-attribute --attribute disableApiTermination --value false --instance-id i-199a6696 

event:
{u'account': u'150905', u'region': u'eu-west-1', u'detail': {u'eventVersion': u'1.05', u'eventID': u'75fe4852-d3d9-4c9c-a702-7f025e0c4c50', u'eventTime': u'2016-10-14T17:29:24Z', u'requestParameters': {u'instanceId': u'i-d8916d57', u'attribute': u'disableApiTermination', u'value': u'false'}, u'eventType': u'AwsApiCall', u'responseElements': {u'_return': True}, u'awsRegion': u'eu-west-1', u'eventName': u'ModifyInstanceAttribute', u'userIdentity': {u'userName': u'ihork', u'principalId': u'AIDAI3U7LBIY', u'accessKeyId': u'AKIAJL7', u'type': u'IAMUser', u'arn': u'arn:aws:iam::150905:user/igor', u'accountId': u'150905'}, u'eventSource': u'ec2.amazonaws.com', u'requestID': u'd3c46-2def-4450-b3c1-4827d9f78', u'userAgent': u'aws-cli/1.10.45 Python/2.7.11 Linux/4.7.3-100.fc23.x86_64 botocore/1.4.60', u'sourceIPAddress': u'174.231.5.2'}, u'detail-type': u'AWS API Call via CloudTrail', u'source': u'aws.ec2', u'version': u'0', u'time': u'2016-10-14T17:29:24Z', u'id': u'9fb88a9e-025b-4859-9b98-6180cd14a9b', u'resources': []}


Take a precise look on:

'sessionContext': {u'attributes': {u'creationDate': u'2016-10-14T16:48:44Z', u'mfaAuthenticated': u'true'}} - not expect this part of JSON if you are not using Mfa

 u'disableApiTermination': {u'value': True}} and 'attribute': u'disableApiTermination', u'value': u'false'} same API call done using different AWS CLI options, but serving the same purpose produce 2 different events

How we can disable a user in AWS ?

You just can't disable user in AWS. You can delete it, but you need to remove it from the groups first. It takes times, lines of code and API calls. Solution? - attach inline user policy with explicit deny (will override all allows) for all the actions you need to block.

Lambda function:

Here my PoC lambda function: really "dirty" and serving only one simple use case:

def lambda_handler(event, context):
    print event
# analyzing event
    if event['detail']['requestParameters'].get('disableApiTermination')!= None:
        protection_status = event['detail']['requestParameters']['disableApiTermination']['value']
        UserName = event['detail']['userIdentity']['userName']
        UserID = event['detail']['userIdentity']['principalId']
        if event['detail']['userIdentity'].get('sessionContext') != None:
            mfa = event['detail']['userIdentity']['sessionContext']['attributes']['mfaAuthenticated']
        else:
            mfa = "false"
        print protection_status, UserName, UserID, mfa
# disabling user using inline user policy if no MFA being used
        if mfa != "true" and not protection_status:
            iam = boto3.resource('iam')
            user_policy = iam.UserPolicy(UserName,'disable_user')
            response = user_policy.put(PolicyDocument='{ "Version": "2012-10-17", "Statement": [{"Sid": "Disableuser01","Effect": "Deny","Action": ["ec2:StopInstances", "ec2:TerminateInstances"],"Resource": ["*"]}]}')
            print response



How near  this "near real time" events:
My test showed about 40 second delay. IMHO too much for the "near real time". I'm looking for the potential bottleneck and delays that may caused by Lambda function itself on Event type I used. 


Conclusions:
- not a "near real time" to react fast and mitigate attack without additional protective measures.
- could work if you able to detect attack 40 second earlier
- could reduce overall damages
- definitely very very promising if reaction delay will be less (let's say 5-10 sec).

Update:

Fill free to pull from GitHub AWS CloudFormation template for the  PoC above.

To deploy you need: 

1. selfdefence.infosec.vpc.json - template itself.

2. selfdefence_infosec.py - Lambda function. You will need to Zip it and upload to the s3 bucket with versioning enabled.

3. Edit template (selfdefence.infosec.vpc.json) and specify: S3 bucket name in format you.bucket.name.env.cloudform (where env - is your environment name: prod, test, staging, etc) and S3 version for  selfdefence_infosec.zip file. 

4. upload template to the same s3 bucket.

5. Create a stack using this template end specify corresponding environment name at the creation time.

Enjoy! 

Amazon AWS Account level security checklist and how-to

Disclaimer :-):
There are bunch of Amazon AWS security checklists and recommendations online. Definitely the best one is https://d0.awsstatic.com/whitepapers/compliance/AWS_CIS_Foundations_Benchmark.pdf 
I'm not trying to reinvent the wheel, but integrate and summarize lessons I learned and advices given to me by other AWS experts.

This checklist starts from the moment when you begin AWS account creation.

1.  Create dedicated email address for AWS account registration. This email will become you root account login name, so, please, do not use your daily used or published online email
2. Enable MFA  (Multi Factor Authentication) on the root account. Details: http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_mfa.html
3. Remove or DO NOT create any API key associated with root account. API keys has no MFA - anyone who has root API keys gets full  access to you account. Unintentional leaking of the API key quite common security incident.
4. Copy/bookmark/save IAM sign-in url. You will need to access you AWS Web GUI.
5. Create IAM user with  AdministratorAccess policy attached. It will be your new  "root" like account.
6. Create other IAM users required. Minimize their permission using built-in AWS managed policies like: PowerUserAccess; ReadOnlyAccess; AmazonEC2FullAccess , etc
7. Enable MFA on all users created.  Details: http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_mfa.html
8. Enforce strict password policy. Details: http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_passwords_account-policy.html
9. Generate API keys for users who needs it. For "high-power" user make this keys inactive. They will activate keys through MFA protected AWS Web GUI only when it needed.
10. Do not use API keys in applications running inside AWS. Use IAM roles instead. Details: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
11. Enable and configure CloudTrail  for all regions  + s3 bucket for the CloudTrailLogs.  Details: http://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-create-a-trail-using-the-console-first-time.html
12. Send CloudTrails Events to the CloudWatch Logs. Details: http://docs.aws.amazon.com/awscloudtrail/latest/userguide/send-cloudtrail-events-to-cloudwatch-logs.html
13. Configure monitoring of the CloudTrail Log Files using Amazon CloudWatch Logs metric filters and alarms. Details: http://docs.aws.amazon.com/awscloudtrail/latest/userguide/monitor-cloudtrail-log-files-with-cloudwatch-logs.html
14. Configure near-real time Log data processing using Subscriptions or/and using lambda function.  Details: http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/Subscriptions.html
15. Using #13 and 14 configure notification for suspicions events
16. Enable AWS Config Service to get AWS configuration snapshots and change notifications. Details: http://docs.aws.amazon.com/config/latest/developerguide/gs-console.html
17. Enable and configure AWS VPC flow logs to get visibility on network level. Details: http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/flow-logs.html
18. Enforce server side encryption on your S3 buckets: Details: http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html
19. Enable encryption on you EBS volumes: Details: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSEncryption.html

Almost all steps covered above could and must be automated. I already published and will publish more automation examples in this blog.

Check your resulted account security status:

• Check Amazon AWS Trusted advisor 
• Run AWS Credentials Report 
• Run awesome AWS audit Scout2 tool : https://isecpartners.github.io/Scout2/

And do this periodically. 

Checklists and Best Practices:

AWS CIS Foundations Benchmark (must read document)
https://d0.awsstatic.com/whitepapers/compliance/AWS_CIS_Foundations_Benchmark.pdf

AWS Auditing Security Checklist
https://d0.awsstatic.com/whitepapers/compliance/AWS_Auditing_Security_Checklist.pdf

PS. I would like to thank Liem aka Pimpon  for advices in preparing this checklist.

Amazon AWS re:invent 2014 highlights

Videos from AWS re:invent worth watching:

Must to know:

Advance Usage of AWS CLI:
https://www.youtube.com/watch?v=vP56l7qThNs


Architecture:

Deploy High Availability & DR with AWS:
http://bit.ly/1BKo4fu

Infrastructure as a code:
http://bit.ly/11p59ag

From One to Many: Evolving VPC Design:
http://bit.ly/1Hb1Cwk

Security:

Intrusion detection in the Cloud
http://bit.ly/1xjXWUJ

Delegating Acccess to you AWS environment
http://bit.ly/1xV6MZ9

DevSecOps
http://bit.ly/1H31W0d


Network:

Creating Your virtual Data Center(VPC)
http://bit.ly/1uBuWqY

Black-Belt Networking for the cloud Ninja:
http://bit.ly/1wZmqPd

Amazon VPC Deep Dive
http://bit.ly/1oYAElI

Amazon EC2 Networking Deep Dive and Best Practices:
https://www.youtube.com/watch?v=JUw8y_pqD_Y

Elastic Load Balancing Deep Dive
http://bit.ly/1y4nW4S

Performance:

Maximizing Amazon S3 performance:
http://bit.ly/1y4novI


Monitoring:

Amazon CloudWatch Deep Dive:
http://bit.ly/1yavmac


BigData:

Lessons Learned and the best Practices for running Hadoop on AWS:
http://bit.ly/1vn8OTd

Amazon EMR Deep Dive and Best Practices:
http://bit.ly/1BKl8jg


Need more? Sure!

Bunch of other videos to explore on AWS Youtube channel :
https://amazonwebservices.thismoment.com/us-en/youtube/reinvent20142

And on aws blog:
http://aws.amazon.com/blogs/aws/online-content-from-reinvent-2014/

Amazon AWS re:invent 2014. Cloud security for Enterprise

Amazon AWS re:invent 2014 from infosec point of view in one sentence:

 Giant step towards  Enterprise market and by adding following services.

- AWS Directory Service:  

             "AWS Directory Service is a managed service that allows you to connect your AWS resources with an existing on-premises Microsoft Active Directory or to set up a new, stand-alone directory in the AWS Cloud." 

Details: http://aws.amazon.com/directoryservice

- AWS Key Management Service

      " AWS Key Management Service (KMS) is a managed service that makes it easy for you to create and control the encryption keys used to encrypt your data, and uses Hardware Security Modules (HSMs) to protect the security of your keys."

Details: http://aws.amazon.com/kms/

Nor forget about AWS CloudHSMservice: http://aws.amazon.com/cloudhsm/

- AWS Config

Finaly! - configuration management for AWS. "WS Config is a fully managed service that provides you with an AWS resource inventory, configuration history, and configuration change notifications to enable security and governance."

Details: http://aws.amazon.com/config/

- AWS Service Catalog

 Narrow variety of AWS services to the list of services your company use and present this as a cusom portal for your employee. " AWS Service Catalog is a service that allows administrators to create and manage approved catalogs of resources that end users can then access via a personalized portal."

Details: http://aws.amazon.com/servicecatalog/

The following two services allows you to build centralized log collectors with kind very primitive  SIEM (Cloudwatch alarms) in AWS:

Amazon CloudWatch Logs : "You can now use Amazon CloudWatch to monitor and troubleshoot your systems and applications using your existing system, application, and custom log files. You can send your existing log files to CloudWatch Logs and monitor these logs in near real-time."

Details: http://aws.amazon.com/about-aws/whats-new/2014/07/10/introducing-amazon-cloudwatch-logs/

AWS CloudTrail integration with CloudWatch:  "This integration enables you to receive SNS notifications from CloudWatch, triggered by specific API activity captured by CloudTrail. With SNS notifications, you can take immediate action when a pattern of interest is detected."

Details: http://aws.amazon.com/blogs/aws/cloudtrail-cloudwatch-integration-partners-too/

Encryption on any storage:

Amazon EBS encryption now  (may 2014) available: http://aws.amazon.com/about-aws/whats-new/2014/05/21/Amazon-EBS-encryption-now-available/

Details: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSEncryption.html

S3 data encryption 

Details:http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingEncryption.html

http://aws.amazon.com/blogs/aws/s3-encryption-with-your-keys/

RDS (Relationship Database service) encryption:

1. Using EBS built-in encryption

2. Use DB specific encryption: 

Oracle Transparent Encryption http://aws.amazon.com/about-aws/whats-new/2013/04/18/amazon-rds-oracle-encryption/

Microsoft SQL Transparent Data Encryption http://aws.amazon.com/blogs/aws/amazon-rds-for-microsoft-sql-server-transparent-data-encryption-tde-/

Infosec certifications: SAS70, ISO27001, PCI DSS, DoD CSM

Details: http://aws.amazon.com/compliance/