Monday, December 8, 2014

Amazon AWS re:invent 2014 highlights

Videos from AWS re:invent worth watching:

Must to know:

Advance Usage of AWS CLI:


Deploy High Availability & DR with AWS:

Infrastructure as a code:

From One to Many: Evolving VPC Design:


Intrusion detection in the Cloud

Delegating Acccess to you AWS environment



Creating Your virtual Data Center(VPC)

Black-Belt Networking for the cloud Ninja:

Amazon VPC Deep Dive

Amazon EC2 Networking Deep Dive and Best Practices:

Elastic Load Balancing Deep Dive


Maximizing Amazon S3 performance:


Amazon CloudWatch Deep Dive:


Lessons Learned and the best Practices for running Hadoop on AWS:

Amazon EMR Deep Dive and Best Practices:

Need more? Sure!

Bunch of other videos to explore on AWS Youtube channel :

And on aws blog:

Amazon AWS re:invent 2014. Cloud security for Enterprise

Amazon AWS re:invent 2014 from infosec point of view in one sentence:
 Giant step towards  Enterprise market and by adding following services.

- AWS Directory Service:  
             "AWS Directory Service is a managed service that allows you to connect your AWS resources with an existing on-premises Microsoft Active Directory or to set up a new, stand-alone directory in the AWS Cloud." 

- AWS Key Management Service
      " AWS Key Management Service (KMS) is a managed service that makes it easy for you to create and control the encryption keys used to encrypt your data, and uses Hardware Security Modules (HSMs) to protect the security of your keys."
Nor forget about AWS CloudHSMservice:

- AWS Config
Finaly! - configuration management for AWS. "WS Config is a fully managed service that provides you with an AWS resource inventory, configuration history, and configuration change notifications to enable security and governance."

- AWS Service Catalog
 Narrow variety of AWS services to the list of services your company use and present this as a cusom portal for your employee. " AWS Service Catalog is a service that allows administrators to create and manage approved catalogs of resources that end users can then access via a personalized portal."

The following two services allows you to build centralized log collectors with kind very primitive  SIEM (Cloudwatch alarms) in AWS:
Amazon CloudWatch Logs : "You can now use Amazon CloudWatch to monitor and troubleshoot your systems and applications using your existing system, application, and custom log files. You can send your existing log files to CloudWatch Logs and monitor these logs in near real-time."
AWS CloudTrail integration with CloudWatch:  "This integration enables you to receive SNS notifications from CloudWatch, triggered by specific API activity captured by CloudTrail. With SNS notifications, you can take immediate action when a pattern of interest is detected."

Encryption on any storage:
S3 data encryption 
RDS (Relationship Database service) encryption:
1. Using EBS built-in encryption
2. Use DB specific encryption: 

Infosec certifications: SAS70, ISO27001, PCI DSS, DoD CSM

Monday, May 26, 2014

Amazon AWS EC2 volume encryption (LUKS) and performance for database.

One of the most obvious  recommendation for the public clouds - use encryption!
Let's talk how to do this practically for Amazon EC2 instance.

1. As encryption we will choose LUKS as kernel level block device encryption.
2. Our Amazon EC2 disk volume will be EBS (we need persistent storage )

But it leads to a many questions: what encryption algorithm and set ,  key length? Do we need EBS optimized instance and provisioned IOPS? What options will give us better performance?

Almost forget: this encrypted volume will be used by database. So, performance is key requirement for me.
By performance I mean 6 parameters: block write IO and latency, read IO and latency and CPU load during both tests.

Lets start the tests:

Encryption sets test:
To make results more predictable we need to have EBS storage  to be persistent (Have guaranteed IO)
According to CPU  flags amazon instance has aes support and I'll test only aes encryption sets.
For the IO benchmark I'm using bonnie++

Encryption on device without LVM. IOPS3000
-c aes-ecb-plain -s 128
-c aes-ecb-null -s 128
-c aes-ecb-benbi -s 128
-c aes-cbc-null -s 128
-c aes-cbc-benbi -s 128
-c aes-cbc-plain -s 128
-c aes -s 128
-s 128

You can use bon_csv2html program to convert CSV format Bonnie++ to nice looking HTML table like following:

EBS_optimized instance and 3000IOPS encryption test_NO LVM
-c aes-ecb-plain -s 128
Version 1.96Sequential OutputSequential InputRandom
Sequential CreateRandom Create
SizePer CharBlockRewritePer CharBlockNum FilesCreateReadDeleteCreateReadDelete
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU

-c aes-ecb-null -s 128
Version 1.96Sequential OutputSequential InputRandom
Sequential CreateRandom Create
SizePer CharBlockRewritePer CharBlockNum FilesCreateReadDeleteCreateReadDelete
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU

-c aes-ecb-benbi -s 128
Version 1.96Sequential OutputSequential InputRandom
Sequential CreateRandom Create
SizePer CharBlockRewritePer CharBlockNum FilesCreateReadDeleteCreateReadDelete
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU

 In nutshell, tests above showed that there is almost no major difference between aes based encryption sets.We have write IO 48800 and latency about 21000 ms; for read we have IO around 64000 and latency around 21341us. 

To compare results I did a test with default encryption set and key length 128 against worst case scenario - normal EBS with non-EBS optimized instance and encryption on top of LVM. 

Version 1.96Sequential OutputSequential InputRandom
Sequential CreateRandom Create
SizePer CharBlockRewritePer CharBlockNum FilesCreateReadDeleteCreateReadDelete
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU

In this case write IO dropped a bit to 32515 and latency increased to 24535ms. But, surprise, read IO increased up to 76000 (unfortunately latency too: 229 ms)

Additional tests just to show the difference between encrypted and non encrypted EBS volume parameters.

Provisioned IOPS (1000) + aes-cbc 128
K/sec 40808
Latency 13010 ms
CPU % 8
K/sec 56827
Latency 59051us
CPU % 6
Provisioned IOPS (1000) Non encrypted
K/sec 41492
Latency 228 ms
CPU % 8
K/sec 56884
Latency 102ms
CPU % 7

Different tests done on non-provisioned IOPS, non EBS-optimized, encryption on top of LVM. (Another EC2 instance)

default -256
K/sec 26331
Latency 196563ms
CPU % 5
K/sec 89304
Latency 1005
CPU % 10
Default -128
K/sec 27494
Latency 13847ms
CPU % 5
K/sec 102898
Latency 306
CPU % 12
K/sec 23255
Latency 626 ms
CPU % 4
K/sec 95027
Latency 1764
CPU % 12

Bottom line:
1. Encryption heavily increase latency (hundred times).
2. Provisioning IOPS (actually only  reserve IO for your system) and switching to EBS optimized instance (kind of having dedicated network interface to SAN)   won't solve  latency problem. They will help you  only if storage resource become overload by other amazon user or if it already overloaded by others.
3. Encryption set (if as encryption algorithm you use aes) doesn't make a big difference.
4. You can decrease latency by changing a key length - but you will decrease security as well.
5. Using LUKS without LVM could decrease latency (~10 %) as well.

PS. Small bash script below will help you to test encryption sets. You can run this script for different types of  block devices (EBS, ephemeral, EBS+IOPS, LVM or non LVM ) and confirm or refute my conclusions.
Moreover this script will show how you can configure and mount LUKS device :-)

#test function
function enc_test() {
while read encryption;
cryptsetup luksFormat --key-file '/root/keyfile' $encryption -q $TDISK || continue;
cryptsetup luksOpen --key-file '/root/keyfile' $TDISK d02_enc;
mkfs.ext4 /dev/mapper/d02_enc;
mount /dev/mapper/d02_enc /d02;
mkdir /d02/iotest;
echo $encryption >> enc_test_out.txt;
bonnie++ -d /d02/iotest  -r 3078 -u root -f -b -q >> enc_test_out.txt;
umount /d02
cryptsetup luksClose d02_enc
done <-c aes-ecb-plain -s 128
-c aes-ecb-null -s 128
-c aes-ecb-benbi -s 128
-c aes-cbc-null -s 128
-c aes-cbc-benbi -s 128
-c aes-cbc-plain -s 128
-c aes -s 128
-s 128
echo "Encryption on top of LVM" >> enc_test_out.txt
exit 0

Wednesday, April 2, 2014

Using CFEngine for Linux systems hardening

What is CFEngine? The best answer to this question is CFEngine website:

In nutshell CFEngine is "is a popular open source configuration management system, written by Mark Burgess. Its primary function is to provide automated configuration and maintenance of large-scale computer systems, including the unified management ofserversdesktops, embedded networked devices, mobile smartphones, and tablet computers." Wiki

I'm using CFengine for various sysadmin and infosec tasks and it proved to be  reliable and stable configuration management system.

I would like to share with you cfengine promises (  system configuration description written on cfengine language  ) for the RH based Linux systems. These "promises" enforce system to become and stay hardened, provide centralized user management and take care of initial system configuration.

file - covers system configuration and hardening.
file - user and group management
file - allows you to describe different environments or data-centers (Global variables for the system configuration )
file - main file that links all components together.
file and - responsible for promises update in normal operations and in case of failure.

I'll add more comments and descriptions in subsequent code release or upon your request.
IMHO the code is self-explaining and really easy to read  as soon as you become familiar with a basic CFEngine principles.

Saturday, March 22, 2014

Making Juniper SSL VPN network connect applet work on 64-bit Linux (Fedora)

Old Juniper SSL VPN appliances network connect Java applets not working on 64 bit systems without some additional steps. Here these steps.

1. Install older then Java "7 update 51" 32 bit version of java. 
         You can use 7.51 but you need to add your ssl vpn endpoint to the exception list: 
 You can download 32 bit JRE 7.45 from Oracle web site java archive
 I recommend you to download tar.gz archive instead of rpm. It will allow you to have multiply java  installed and switch between them using alternatives

2.  Install java and configure alternatives to use downloaded java: 
$ tar -xvzf ./jre-7u40-linux-i586.tar.gz
# mv /home/myuser/Downloads/jre1.7.0_45/ /usr/java/

configure alternatives for java 32 bit

# alternatives --install /usr/bin/java java /usr/java/jre1.7.0_40/bin/java 32
configure alternatives for java browser plugin (yes, i suggest to use alternatives to manage java plugins)
alternatives --install /opt/google/chrome/plugins/ java_chrome /usr/java/jre1.7.0_40/lib/i386/ 32

switch alternatives to use installed 32bit Java:
# alternatives --config java_chrome
# alternatives --config java

3. Test if java plugins works and has correct version  in chrome or firefox
go to and verify Java version

4. Install xterm and some 32 bit version of libraries: 
    yum install xterm glibc.i686 zlib.i686 
    These components required to install and run network connect application

5. Go to your ssl vpn endpoint using browser, login and launch network connect.
You should now see new xterm window and sudo password request for first time network connect installation.

If not --> check network connect install log for details:
$ less ~/.juniper_networks/network_connect/installnc.log

Normally after this you should see network connect starting and working. 

If not ---> check:
 - presence of network connect aplication , file permissions and ownership:
      $ ls -la ~/.juniper_networks/network_connect/
         -rws--s--x. 1 root   root   1281164 Mar 21 22:17 ncsvc
 - if network connect application could run:  
$ ~/.juniper_networks/network_connect/ncsvc --version
Juniper Network Connect Server for Linux.
Version         : 7.1
Release Version : 7.1-16-Build26805
Build Date/time : Aug 21 2013 01:11:08 
Copyright 2001-2010 Juniper Networks
if you see any error check for the missing 32bit libraries.

Monday, March 17, 2014

Junk Cloud POC project.

Junk Cloud is fast deploying CloudStack environment running on KVM hypervisor and using outdated equipment.

As first attempt to build Junk Cloud I have tested CloudStack baremetal support. What I expect from baremetal support?

  • reduce time to configure and deploy each host
  • you can add computing resources (physical hosts) to the cloud in a minutes instead of hours. 
  • fast deployment of the new private cloud
  • allow easy utilization of outdated hardware

Unfortunately CloudStack built-in baremetal support is limited:

  • not support advanced networking
  • not support central storage (SAN)
  • not support vmotion and HA

So, it was decided to build own version of the barematal support for theCloudStack:

  • network boot (pxe boot)
  • network auto install of lnux + kvm + network + CloudStack
  • storage configuration 
  • upon completion auto selfprovisionng to the CloudStack as a new host (resource)
The Anaconda install script for building and configuring Centos based KVM host for CloudStack (Advanced networking support) on GitHub:

This script is missing auto selfprovisionng to the CloudStack feature for test purpose. It will be updated after final tests.

Tested and suggested deployment architecture (for the small deployments):

From my point of view implementation of proposed architecture will be:

  • core components: cloud controller + network storage installed and configured. 
  • to add new host just connect it to network and power.
  • all rest done automatically.
Private Cloud? it's that simple:

Build a Cloud Controller:
And finally add all your old hardware:

CloudStack-BIND dns integration.

How to integrate CloudStack DNS with your organization DNS environment?
All recommendation bellow based on the approach of using one sub-domain per CloudStack network.
 1. The simplest way: conditional forwarding from windows DNS server to CloudStack VR for the corresponding sub-domain or (BIND) sub-domain delegation to the VR. 
pros: easy to build
con: VR are running as not authoritative DNS for sub-domain, each time new record added dnsmasq service restarts and you have 2-3 sec of downtime (up to v4.1), all request going to VR.
  2.  CloudStack --> BIND full integration:
Following program solves DNS integration issues between CloudStack VR's DNS service and BIND DNS.
This program assumes that you are using sub-domain per network(each network has own sub-domain) (IMHO the best way fro naming instances in CloudStack)
How it works:
On event or on schedule program call CloudStack API and get list of Networks and list of VM. Using theses lists and preconfigured domain settings it creates the zone file for BIND, push it to server and refresh BIND.
This program could be run using 2 different ways: 
  1.  being installed on DNS server and update DNS records on scheduled interval. (schedule driven)
  2.  being installed on CloudStack management server and listen for the new vm deployment using CloudStack catalina.out log. (event driven)

Proposed version running on heavy used CloudStack environment with very frequent SaltStack driven deployments with almost no issue. 

The script is under active development and testing and will be updated.
Version 2.0 released with all parameters now loaded from dns_builder.conf file and local and remote DNS servers support. 
Some times this script fails because of non-expeted  ClaudStack response:
    dns_table = get_dns()
  File "/usr/bin/", line 134, in get_dns
    output+= vm['name'] + "." + net_dict[vm['nic'][0]['networkname']] + "\t\t\t300\tIN\tA\t" + vm['nic'][0]['ipaddress'] +" \n"
IndexError: list index out of range

Instead of exception handling to keep this script running I used supervisord daemon.
It starts the script, makes sure it running, restarts in case of failure and takes care of logs.
Part of supervizord.conf file related to the script:
command=/usr/bin/      ; the program (relative uses PATH, can take args)
priority=100                ; the relative start priority (default 999)
autostart=true              ; start at supervisord start (default: true)
autorestart=true            ; retstart at unexpected quit (default: true)
;startsecs=10                ; number of secs prog must stay running (def. 10)
startretries=5              ; max # of serial start failures (default 3)
;exitcodes=0,2               ; 'expected' exit codes for process (default 0,2)
;stopsignal=QUIT             ; signal used to kill process (default TERM)
;stopwaitsecs=10             ; max num secs to wait before SIGKILL (default 10)
;user=chrism                 ; setuid to this UNIX account to run the program
log_stdout=true             ; if true, log program stdout (default true)
log_stderr=true             ; if true, log program stderr (def false)
logfile=/var/log/dns_builder.log    ; child log path, use NONE for none; default AUTO
logfile_maxbytes=1MB        ; max # logfile bytes b4 rotation (default 50MB)
logfile_backups=2          ; # of logfile backups (default 10)

Monday, March 10, 2014

Firewall rule logging

     When firewall is managed by NOC team and you are part of infosec it's really hard to maintain reasonable log level at firewall rules.
    Even If decision of log do not log this rule is part  of the change request process and made by infosec you have a risk of having to much logs, having duplicated logs of the same event form different firewalls or spending more time on each request to avoid problem mentioned.
     In all other cases you definitely end up having too much logs that could even overload and slow down you FW device (for me it happened on opendbsd pf firewall when syslog had consumed all memory ).
  Sure thing, you can optimize fw rules and reduce amount of logs - but  it's thankless time consuming process especially if you have many FW devices.
    Create special fw rules just for the log purpose. They will be at the begin of the rule list and identical on all your FW devices.

Logging rules that:

  • easy to create (based on network topology and security zones )
  • easy to check
  • easy to tune 
  • easy to distribute and install
For PF firewall those rules will look like: 

match log inet proto tcp from to port 80

Thursday, February 27, 2014

Virtual machines templates. Static or Live?

Virtualization is everywhere. Need a new VM ? - it's easy,  clone from template and spin it up!

But let's take a close look at the classical "static" templates security cons:

- Static disk image file 
- get updated from time to time ( in most cases not often)
- security policies (or hardening) on it updated only after deployment 
- not being monitored by any part of your security monitoring infrastructure (no events coming, no AV or vulnerabilities  scans,  etc.)

Templates are the key component of your infrastructure - you are cloning all you running machines from them.
To keep templates on the same security level you need: spin up a vm from this template, update it, check stability after updates, scan it for vulnerabilities and re-template back (and do not forget to update checksums. You are checking templates' checksums before deployment, aren't you? :-)   ). 

IMHO, the biggest security issue with "static" templates - is the time frame when your recent cloned from template machine boot up and  get updates  being at the same time not fully protected and up to day.  

Instead of using "static" templates, you can use "live" templates - just normal running VMs  (CPU and memory are really cheap now). So ,what the pros:

- part of you live infrastructure
- get updates as soon as updates are issued or/and approved
- get current version of the security(or hardening) policy implemented
- monitored and covered by your security infrastructure(AV scanners, HIDS/NIDS, log management)
- monitored for up time and stability after each update

Script, shown below, being placed in cron.daily will keep your RH based linux  template up to day (for VMware virtualization)


# Doing self-update if we are running on host with template in name and self-destruction on other hosts 

if [[ "$HOSTNAME" == y??0?centos?tmpl* ]]; 
then /usr/bin/yum update -y > /var/log/self_update.log 2>&1
/bin/rm -v /etc/cron.daily/self_update > /var/log/self_update.log 2>&1 

# Comparing installed and running kernel and reboot in case if they differ 

LAST_KERNEL=$(/bin/rpm -q --last kernel | /usr/bin/perl -pe 's/^kernel-(\S+).*/$1/' | /usr/bin/head -1)
CURRENT_KERNEL=$(/bin/uname -r)
/bin/echo '/usr/bin/ -d ; /bin/sed -i "/.*vmware.*/d"  /etc/rc.local'  >> /etc/rc.local
exit 0

PS. Script also address annoying problem which requires rerun vmware configuration utility after each kernel update to make vmware tools run .
PS2 And self-destruct on non-template hosts.