Showing posts with label Virtualization. Show all posts
Showing posts with label Virtualization. Show all posts

Monday, May 26, 2014

Amazon AWS EC2 volume encryption (LUKS) and performance for database.

One of the most obvious  recommendation for the public clouds - use encryption!
Let's talk how to do this practically for Amazon EC2 instance.

1. As encryption we will choose LUKS as kernel level block device encryption.
2. Our Amazon EC2 disk volume will be EBS (we need persistent storage )

But it leads to a many questions: what encryption algorithm and set ,  key length? Do we need EBS optimized instance and provisioned IOPS? What options will give us better performance?

Almost forget: this encrypted volume will be used by database. So, performance is key requirement for me.
By performance I mean 6 parameters: block write IO and latency, read IO and latency and CPU load during both tests.

Lets start the tests:

Encryption sets test:
To make results more predictable we need to have EBS storage  to be persistent (Have guaranteed IO)
According to CPU  flags amazon instance has aes support and I'll test only aes encryption sets.
For the IO benchmark I'm using bonnie++

Encryption on device without LVM. IOPS3000
-c aes-ecb-plain -s 128
1.96,1.96,ip-10-14.eu-west-1.compute.internal,1,1399324041,14674M,,,,48625,6,24487,4,,,63744,4,2639,62,16,,,,,271,1,+++++,+++,293,0,272,1,+++++,+++,293,0,,21978ms,14659ms,,19440us,60963us,142ms,364us,47785us,16979us,37us,50287us
-c aes-ecb-null -s 128
1.96,1.96,ip-10-14.eu-west-1.compute.internal,1,1399322496,14674M,,,,48815,6,24488,4,,,63679,5,2631,63,16,,,,,271,1,+++++,+++,304,0,269,1,+++++,+++,289,0,,21779ms,14606ms,,26422us,43205us,143ms,367us,52647us,34894us,10us,48557us
-c aes-ecb-benbi -s 128
1.96,1.96,ip-10-14.eu-west-1.compute.internal,1,1399319920,14674M,,,,48814,6,24483,4,,,63659,5,2704,64,16,,,,,271,1,+++++,+++,291,0,272,1,+++++,+++,294,0,,21306ms,14538ms,,27702us,57492us,144ms,370us,48660us,39901us,10us,48902us
-c aes-cbc-null -s 128
1.96,1.96,ip-10-14.eu-west-1.compute.internal,1,1399317092,14674M,,,,48812,6,24476,4,,,63682,5,2630,62,16,,,,,268,1,+++++,+++,291,0,272,1,+++++,+++,291,0,,21630ms,14586ms,,36893us,45752us,142ms,372us,53419us,21090us,10us,53516us
-c aes-cbc-benbi -s 128
1.96,1.96,ip-10-14.eu-west-1.compute.internal,1,1399331123,14674M,,,,48815,6,24475,4,,,63680,5,2696,64,16,,,,,270,1,+++++,+++,291,0,271,1,+++++,+++,291,0,,21290ms,14656ms,,28083us,38028us,142ms,371us,52241us,29474us,10us,51325us
-c aes-cbc-plain -s 128
1.96,1.96,ip-10-14.eu-west-1.compute.internal,1,1399331889,14674M,,,,48814,6,24477,4,,,63659,5,2592,61,16,,,,,269,1,+++++,+++,293,0,273,1,+++++,+++,294,0,,22076ms,14607ms,,28562us,14134us,145ms,366us,47393us,29521us,10us,48817us
-c aes -s 128
1.96,1.96,ip-10-14.eu-west-1.compute.internal,1,1399330501,14674M,,,,48815,7,24480,4,,,63677,5,2714,63,16,,,,,269,1,+++++,+++,302,0,272,1,+++++,+++,292,0,,21060ms,14595ms,,23679us,13062us,142ms,396us,54966us,29954us,10us,50945us
-s 128
1.96,1.96,ip-10-14.eu-west-1.compute.internal,1,1399327477,14674M,,,,48817,6,24470,4,,,63685,5,2630,62,16,,,,,269,1,+++++,+++,289,0,269,1,+++++,+++,290,0,,17066ms,14615ms,,29639us,64800us,142ms,378us,51051us,27984us,44us,50100us

You can use bon_csv2html program to convert CSV format Bonnie++ to nice looking HTML table like following:

EBS_optimized instance and 3000IOPS encryption test_NO LVM
-c aes-ecb-plain -s 128
Version 1.96Sequential OutputSequential InputRandom
Seeks		Sequential CreateRandom Create
SizePer CharBlockRewritePer CharBlockNum FilesCreateReadDeleteCreateReadDelete
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU
ip-10-14.eu-west-1.compute.internal14674M488316244853637574263261162640++++++++28402631++++++++2850
Latency22188ms14602ms24751us14736usLatency143ms390us47199us24302us10us50133us

-c aes-ecb-null -s 128
Version 1.96Sequential OutputSequential InputRandom
Seeks		Sequential CreateRandom Create
SizePer CharBlockRewritePer CharBlockNum FilesCreateReadDeleteCreateReadDelete
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU
ip-10-14.eu-west-1.compute.internal14674M488346244694637415264962162630++++++++28502630++++++++2860
Latency22012ms14608ms36346us13559usLatency143ms379us48095us32471us10us51386us

-c aes-ecb-benbi -s 128
Version 1.96Sequential OutputSequential InputRandom
Seeks		Sequential CreateRandom Create
SizePer CharBlockRewritePer CharBlockNum FilesCreateReadDeleteCreateReadDelete
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU
ip-10-14.eu-west-1.compute.internal14674M488356244474636834265561162670++++++++28802621++++++++2860
Latency21447ms14568ms21341us13945usLatency141ms377us49271us19517us9us50122us


 In nutshell, tests above showed that there is almost no major difference between aes based encryption sets.We have write IO 48800 and latency about 21000 ms; for read we have IO around 64000 and latency around 21341us. 

To compare results I did a test with default encryption set and key length 128 against worst case scenario - normal EBS with non-EBS optimized instance and encryption on top of LVM. 

Version 1.96Sequential OutputSequential InputRandom
Seeks		Sequential CreateRandom Create
SizePer CharBlockRewritePer CharBlockNum FilesCreateReadDeleteCreateReadDelete
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU
ip-10-14.eu-west-1.compute.internal14674M325154223373760775450.64161660++++++++17801650++++++++1780
Latency28895ms24535ms229ms153msLatency170ms369us89118us54455us10us104ms


In this case write IO dropped a bit to 32515 and latency increased to 24535ms. But, surprise, read IO increased up to 76000 (unfortunately latency too: 229 ms)

Additional tests just to show the difference between encrypted and non encrypted EBS volume parameters.

Provisioned IOPS (1000) + aes-cbc 128
write
K/sec 40808
Latency 13010 ms
CPU % 8
read
K/sec 56827
Latency 59051us
CPU % 6
Provisioned IOPS (1000) Non encrypted
write
K/sec 41492
Latency 228 ms
CPU % 8
read
K/sec 56884
Latency 102ms
CPU % 7

Different tests done on non-provisioned IOPS, non EBS-optimized, encryption on top of LVM. (Another EC2 instance)


default -256
write
K/sec 26331
Latency 196563ms
CPU % 5
read
K/sec 89304
Latency 1005
CPU % 10
Default -128
write
K/sec 27494
Latency 13847ms
CPU % 5
read
K/sec 102898
Latency 306
CPU % 12
Nonencrypted
write
K/sec 23255
Latency 626 ms
CPU % 4
read
K/sec 95027
Latency 1764
CPU % 12


Bottom line:
1. Encryption heavily increase latency (hundred times).
2. Provisioning IOPS (actually only  reserve IO for your system) and switching to EBS optimized instance (kind of having dedicated network interface to SAN)   won't solve  latency problem. They will help you  only if storage resource become overload by other amazon user or if it already overloaded by others.
3. Encryption set (if as encryption algorithm you use aes) doesn't make a big difference.
4. You can decrease latency by changing a key length - but you will decrease security as well.
5. Using LUKS without LVM could decrease latency (~10 %) as well.

PS. Small bash script below will help you to test encryption sets. You can run this script for different types of  block devices (EBS, ephemeral, EBS+IOPS, LVM or non LVM ) and confirm or refute my conclusions.
Moreover this script will show how you can configure and mount LUKS device :-)

#!/bin/bash
#
#test function
function enc_test() {
while read encryption;
do
cryptsetup luksFormat --key-file '/root/keyfile' $encryption -q $TDISK || continue;
cryptsetup luksOpen --key-file '/root/keyfile' $TDISK d02_enc;
mkfs.ext4 /dev/mapper/d02_enc;
mount /dev/mapper/d02_enc /d02;
mkdir /d02/iotest;
echo $encryption >> enc_test_out.txt;
bonnie++ -d /d02/iotest  -r 3078 -u root -f -b -q >> enc_test_out.txt;
umount /d02
cryptsetup luksClose d02_enc
done <-c aes-ecb-plain -s 128
-c aes-ecb-null -s 128
-c aes-ecb-benbi -s 128
-c aes-cbc-null -s 128
-c aes-cbc-benbi -s 128
-c aes-cbc-plain -s 128
-c aes -s 128
-s 128
EOM
}
TDISK='/dev/test/d02'
echo "Encryption on top of LVM" >> enc_test_out.txt
enc_test
exit 0


Posted by at No comments:
Labels: , , ,

Monday, March 17, 2014

Junk Cloud POC project.

Junk Cloud is fast deploying CloudStack environment running on KVM hypervisor and using outdated equipment.

As first attempt to build Junk Cloud I have tested CloudStack baremetal support. What I expect from baremetal support?

  • reduce time to configure and deploy each host
  • you can add computing resources (physical hosts) to the cloud in a minutes instead of hours. 
  • fast deployment of the new private cloud
  • allow easy utilization of outdated hardware


Unfortunately CloudStack built-in baremetal support is limited: http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.2.0/html/Installation_Guide/choosing-a-hypervisor.html


  • not support advanced networking
  • not support central storage (SAN)
  • not support vmotion and HA

So, it was decided to build own version of the barematal support for theCloudStack:


  • network boot (pxe boot)
  • network auto install of lnux + kvm + network + CloudStack
  • storage configuration 
  • upon completion auto selfprovisionng to the CloudStack as a new host (resource)
The Anaconda install script for building and configuring Centos based KVM host for CloudStack (Advanced networking support) on GitHub:  https://github.com/IhorKravchuk/cloudstack_ingvar/blob/master/Centos_6x64_kvm_cloudstack.cfg

This script is missing auto selfprovisionng to the CloudStack feature for test purpose. It will be updated after final tests.

Tested and suggested deployment architecture (for the small deployments):



From my point of view implementation of proposed architecture will be:

  • core components: cloud controller + network storage installed and configured. 
  • to add new host just connect it to network and power.
  • all rest done automatically.
Private Cloud? it's that simple:

Build a Cloud Controller:
Add NFS NAS:
And finally add all your old hardware:





Posted by at No comments:
Labels: , , , , ,

Thursday, February 27, 2014

Virtual machines templates. Static or Live?

Virtualization is everywhere. Need a new VM ? - it's easy,  clone from template and spin it up!

But let's take a close look at the classical "static" templates security cons:

- Static disk image file 
- get updated from time to time ( in most cases not often)
- security policies (or hardening) on it updated only after deployment 
- not being monitored by any part of your security monitoring infrastructure (no events coming, no AV or vulnerabilities  scans,  etc.)

Templates are the key component of your infrastructure - you are cloning all you running machines from them.
To keep templates on the same security level you need: spin up a vm from this template, update it, check stability after updates, scan it for vulnerabilities and re-template back (and do not forget to update checksums. You are checking templates' checksums before deployment, aren't you? :-)   ). 

IMHO, the biggest security issue with "static" templates - is the time frame when your recent cloned from template machine boot up and  get updates  being at the same time not fully protected and up to day.  

Instead of using "static" templates, you can use "live" templates - just normal running VMs  (CPU and memory are really cheap now). So ,what the pros:

- part of you live infrastructure
- get updates as soon as updates are issued or/and approved
- get current version of the security(or hardening) policy implemented
- monitored and covered by your security infrastructure(AV scanners, HIDS/NIDS, log management)
- monitored for up time and stability after each update

Script, shown below, being placed in cron.daily will keep your RH based linux  template up to day (for VMware virtualization)

#!/bin/sh

# Doing self-update if we are running on host with template in name and self-destruction on other hosts 

if [[ "$HOSTNAME" == y??0?centos?tmpl* ]]; 
then /usr/bin/yum update -y > /var/log/self_update.log 2>&1
else
/bin/rm -v /etc/cron.daily/self_update > /var/log/self_update.log 2>&1 
fi

# Comparing installed and running kernel and reboot in case if they differ 

LAST_KERNEL=$(/bin/rpm -q --last kernel | /usr/bin/perl -pe 's/^kernel-(\S+).*/$1/' | /usr/bin/head -1)
CURRENT_KERNEL=$(/bin/uname -r)
if [ $LAST_KERNEL != $CURRENT_KERNEL ];
then 
/bin/echo '/usr/bin/vmware-config-tools.pl -d ; /bin/sed -i "/.*vmware.*/d"  /etc/rc.local'  >> /etc/rc.local
/sbin/reboot; 
fi
exit 0

PS. Script also address annoying problem which requires rerun vmware configuration utility after each kernel update to make vmware tools run .
PS2 And self-destruct on non-template hosts.







Posted by at No comments:
Labels: , , ,
Subscribe to: Posts (Atom)