Monday, May 26, 2014

Amazon AWS EC2 volume encryption (LUKS) and performance for database.

One of the most obvious  recommendation for the public clouds - use encryption!
Let's talk how to do this practically for Amazon EC2 instance.

1. As encryption we will choose LUKS as kernel level block device encryption.
2. Our Amazon EC2 disk volume will be EBS (we need persistent storage )

But it leads to a many questions: what encryption algorithm and set ,  key length? Do we need EBS optimized instance and provisioned IOPS? What options will give us better performance?

Almost forget: this encrypted volume will be used by database. So, performance is key requirement for me.
By performance I mean 6 parameters: block write IO and latency, read IO and latency and CPU load during both tests.

Lets start the tests:

Encryption sets test:
To make results more predictable we need to have EBS storage  to be persistent (Have guaranteed IO)
According to CPU  flags amazon instance has aes support and I'll test only aes encryption sets.
For the IO benchmark I'm using bonnie++

Encryption on device without LVM. IOPS3000
-c aes-ecb-plain -s 128
1.96,1.96,ip-10-14.eu-west-1.compute.internal,1,1399324041,14674M,,,,48625,6,24487,4,,,63744,4,2639,62,16,,,,,271,1,+++++,+++,293,0,272,1,+++++,+++,293,0,,21978ms,14659ms,,19440us,60963us,142ms,364us,47785us,16979us,37us,50287us
-c aes-ecb-null -s 128
1.96,1.96,ip-10-14.eu-west-1.compute.internal,1,1399322496,14674M,,,,48815,6,24488,4,,,63679,5,2631,63,16,,,,,271,1,+++++,+++,304,0,269,1,+++++,+++,289,0,,21779ms,14606ms,,26422us,43205us,143ms,367us,52647us,34894us,10us,48557us
-c aes-ecb-benbi -s 128
1.96,1.96,ip-10-14.eu-west-1.compute.internal,1,1399319920,14674M,,,,48814,6,24483,4,,,63659,5,2704,64,16,,,,,271,1,+++++,+++,291,0,272,1,+++++,+++,294,0,,21306ms,14538ms,,27702us,57492us,144ms,370us,48660us,39901us,10us,48902us
-c aes-cbc-null -s 128
1.96,1.96,ip-10-14.eu-west-1.compute.internal,1,1399317092,14674M,,,,48812,6,24476,4,,,63682,5,2630,62,16,,,,,268,1,+++++,+++,291,0,272,1,+++++,+++,291,0,,21630ms,14586ms,,36893us,45752us,142ms,372us,53419us,21090us,10us,53516us
-c aes-cbc-benbi -s 128
1.96,1.96,ip-10-14.eu-west-1.compute.internal,1,1399331123,14674M,,,,48815,6,24475,4,,,63680,5,2696,64,16,,,,,270,1,+++++,+++,291,0,271,1,+++++,+++,291,0,,21290ms,14656ms,,28083us,38028us,142ms,371us,52241us,29474us,10us,51325us
-c aes-cbc-plain -s 128
1.96,1.96,ip-10-14.eu-west-1.compute.internal,1,1399331889,14674M,,,,48814,6,24477,4,,,63659,5,2592,61,16,,,,,269,1,+++++,+++,293,0,273,1,+++++,+++,294,0,,22076ms,14607ms,,28562us,14134us,145ms,366us,47393us,29521us,10us,48817us
-c aes -s 128
1.96,1.96,ip-10-14.eu-west-1.compute.internal,1,1399330501,14674M,,,,48815,7,24480,4,,,63677,5,2714,63,16,,,,,269,1,+++++,+++,302,0,272,1,+++++,+++,292,0,,21060ms,14595ms,,23679us,13062us,142ms,396us,54966us,29954us,10us,50945us
-s 128
1.96,1.96,ip-10-14.eu-west-1.compute.internal,1,1399327477,14674M,,,,48817,6,24470,4,,,63685,5,2630,62,16,,,,,269,1,+++++,+++,289,0,269,1,+++++,+++,290,0,,17066ms,14615ms,,29639us,64800us,142ms,378us,51051us,27984us,44us,50100us

You can use bon_csv2html program to convert CSV format Bonnie++ to nice looking HTML table like following:

EBS_optimized instance and 3000IOPS encryption test_NO LVM
-c aes-ecb-plain -s 128
Version 1.96Sequential OutputSequential InputRandom
Seeks
Sequential CreateRandom Create
SizePer CharBlockRewritePer CharBlockNum FilesCreateReadDeleteCreateReadDelete
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU
ip-10-14.eu-west-1.compute.internal14674M488316244853637574263261162640++++++++28402631++++++++2850
Latency22188ms14602ms24751us14736usLatency143ms390us47199us24302us10us50133us

-c aes-ecb-null -s 128
Version 1.96Sequential OutputSequential InputRandom
Seeks
Sequential CreateRandom Create
SizePer CharBlockRewritePer CharBlockNum FilesCreateReadDeleteCreateReadDelete
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU
ip-10-14.eu-west-1.compute.internal14674M488346244694637415264962162630++++++++28502630++++++++2860
Latency22012ms14608ms36346us13559usLatency143ms379us48095us32471us10us51386us

-c aes-ecb-benbi -s 128
Version 1.96Sequential OutputSequential InputRandom
Seeks
Sequential CreateRandom Create
SizePer CharBlockRewritePer CharBlockNum FilesCreateReadDeleteCreateReadDelete
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU
ip-10-14.eu-west-1.compute.internal14674M488356244474636834265561162670++++++++28802621++++++++2860
Latency21447ms14568ms21341us13945usLatency141ms377us49271us19517us9us50122us


 In nutshell, tests above showed that there is almost no major difference between aes based encryption sets.We have write IO 48800 and latency about 21000 ms; for read we have IO around 64000 and latency around 21341us. 

To compare results I did a test with default encryption set and key length 128 against worst case scenario - normal EBS with non-EBS optimized instance and encryption on top of LVM. 

Version 1.96Sequential OutputSequential InputRandom
Seeks
Sequential CreateRandom Create
SizePer CharBlockRewritePer CharBlockNum FilesCreateReadDeleteCreateReadDelete
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU
ip-10-14.eu-west-1.compute.internal14674M325154223373760775450.64161660++++++++17801650++++++++1780
Latency28895ms24535ms229ms153msLatency170ms369us89118us54455us10us104ms


In this case write IO dropped a bit to 32515 and latency increased to 24535ms. But, surprise, read IO increased up to 76000 (unfortunately latency too: 229 ms)

Additional tests just to show the difference between encrypted and non encrypted EBS volume parameters.

Provisioned IOPS (1000) + aes-cbc 128
write
K/sec 40808
Latency 13010 ms
CPU % 8
read
K/sec 56827
Latency 59051us
CPU % 6
Provisioned IOPS (1000) Non encrypted
write
K/sec 41492
Latency 228 ms
CPU % 8
read
K/sec 56884
Latency 102ms
CPU % 7

Different tests done on non-provisioned IOPS, non EBS-optimized, encryption on top of LVM. (Another EC2 instance)


default -256
write
K/sec 26331
Latency 196563ms
CPU % 5
read
K/sec 89304
Latency 1005
CPU % 10
Default -128
write
K/sec 27494
Latency 13847ms
CPU % 5
read
K/sec 102898
Latency 306
CPU % 12
Nonencrypted
write
K/sec 23255
Latency 626 ms
CPU % 4
read
K/sec 95027
Latency 1764
CPU % 12


Bottom line:
1. Encryption heavily increase latency (hundred times).
2. Provisioning IOPS (actually only  reserve IO for your system) and switching to EBS optimized instance (kind of having dedicated network interface to SAN)   won't solve  latency problem. They will help you  only if storage resource become overload by other amazon user or if it already overloaded by others.
3. Encryption set (if as encryption algorithm you use aes) doesn't make a big difference.
4. You can decrease latency by changing a key length - but you will decrease security as well.
5. Using LUKS without LVM could decrease latency (~10 %) as well.

PS. Small bash script below will help you to test encryption sets. You can run this script for different types of  block devices (EBS, ephemeral, EBS+IOPS, LVM or non LVM ) and confirm or refute my conclusions.
Moreover this script will show how you can configure and mount LUKS device :-)

#!/bin/bash
#
#test function
function enc_test() {
while read encryption;
do
cryptsetup luksFormat --key-file '/root/keyfile' $encryption -q $TDISK || continue;
cryptsetup luksOpen --key-file '/root/keyfile' $TDISK d02_enc;
mkfs.ext4 /dev/mapper/d02_enc;
mount /dev/mapper/d02_enc /d02;
mkdir /d02/iotest;
echo $encryption >> enc_test_out.txt;
bonnie++ -d /d02/iotest  -r 3078 -u root -f -b -q >> enc_test_out.txt;
umount /d02
cryptsetup luksClose d02_enc
done <-c aes-ecb-plain -s 128
-c aes-ecb-null -s 128
-c aes-ecb-benbi -s 128
-c aes-cbc-null -s 128
-c aes-cbc-benbi -s 128
-c aes-cbc-plain -s 128
-c aes -s 128
-s 128
EOM
}
TDISK='/dev/test/d02'
echo "Encryption on top of LVM" >> enc_test_out.txt
enc_test
exit 0