Amazon AWS EC2 volume encryption (LUKS) and performance for database.

One of the most obvious  recommendation for the public clouds - use encryption!
Let's talk how to do this practically for Amazon EC2 instance.

1. As encryption we will choose LUKS as kernel level block device encryption.
2. Our Amazon EC2 disk volume will be EBS (we need persistent storage )

But it leads to a many questions: what encryption algorithm and set ,  key length? Do we need EBS optimized instance and provisioned IOPS? What options will give us better performance?

Almost forget: this encrypted volume will be used by database. So, performance is key requirement for me.
By performance I mean 6 parameters: block write IO and latency, read IO and latency and CPU load during both tests.

Lets start the tests:

Encryption sets test:
To make results more predictable we need to have EBS storage  to be persistent (Have guaranteed IO)
According to CPU  flags amazon instance has aes support and I'll test only aes encryption sets.
For the IO benchmark I'm using bonnie++

Encryption on device without LVM. IOPS3000
-c aes-ecb-plain -s 128
1.96,1.96,ip-10-14.eu-west-1.compute.internal,1,1399324041,14674M,,,,48625,6,24487,4,,,63744,4,2639,62,16,,,,,271,1,+++++,+++,293,0,272,1,+++++,+++,293,0,,21978ms,14659ms,,19440us,60963us,142ms,364us,47785us,16979us,37us,50287us
-c aes-ecb-null -s 128
1.96,1.96,ip-10-14.eu-west-1.compute.internal,1,1399322496,14674M,,,,48815,6,24488,4,,,63679,5,2631,63,16,,,,,271,1,+++++,+++,304,0,269,1,+++++,+++,289,0,,21779ms,14606ms,,26422us,43205us,143ms,367us,52647us,34894us,10us,48557us
-c aes-ecb-benbi -s 128
1.96,1.96,ip-10-14.eu-west-1.compute.internal,1,1399319920,14674M,,,,48814,6,24483,4,,,63659,5,2704,64,16,,,,,271,1,+++++,+++,291,0,272,1,+++++,+++,294,0,,21306ms,14538ms,,27702us,57492us,144ms,370us,48660us,39901us,10us,48902us
-c aes-cbc-null -s 128
1.96,1.96,ip-10-14.eu-west-1.compute.internal,1,1399317092,14674M,,,,48812,6,24476,4,,,63682,5,2630,62,16,,,,,268,1,+++++,+++,291,0,272,1,+++++,+++,291,0,,21630ms,14586ms,,36893us,45752us,142ms,372us,53419us,21090us,10us,53516us
-c aes-cbc-benbi -s 128
1.96,1.96,ip-10-14.eu-west-1.compute.internal,1,1399331123,14674M,,,,48815,6,24475,4,,,63680,5,2696,64,16,,,,,270,1,+++++,+++,291,0,271,1,+++++,+++,291,0,,21290ms,14656ms,,28083us,38028us,142ms,371us,52241us,29474us,10us,51325us
-c aes-cbc-plain -s 128
1.96,1.96,ip-10-14.eu-west-1.compute.internal,1,1399331889,14674M,,,,48814,6,24477,4,,,63659,5,2592,61,16,,,,,269,1,+++++,+++,293,0,273,1,+++++,+++,294,0,,22076ms,14607ms,,28562us,14134us,145ms,366us,47393us,29521us,10us,48817us
-c aes -s 128
1.96,1.96,ip-10-14.eu-west-1.compute.internal,1,1399330501,14674M,,,,48815,7,24480,4,,,63677,5,2714,63,16,,,,,269,1,+++++,+++,302,0,272,1,+++++,+++,292,0,,21060ms,14595ms,,23679us,13062us,142ms,396us,54966us,29954us,10us,50945us
-s 128
1.96,1.96,ip-10-14.eu-west-1.compute.internal,1,1399327477,14674M,,,,48817,6,24470,4,,,63685,5,2630,62,16,,,,,269,1,+++++,+++,289,0,269,1,+++++,+++,290,0,,17066ms,14615ms,,29639us,64800us,142ms,378us,51051us,27984us,44us,50100us

You can use bon_csv2html program to convert CSV format Bonnie++ to nice looking HTML table like following:

EBS_optimized instance and 3000IOPS encryption test_NO LVM
-c aes-ecb-plain -s 128

Version 1.96

Sequential Output

Sequential Input

Random Seeks

Sequential Create

Random Create

Size

Per Char

Block

Rewrite

Per Char

Block

Num Files

Create

Read

Delete

Create

Read

Delete

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

ip-10-14.eu-west-1.compute.internal

14674M

48831

6

24485

3

63757

4

2632

61

16

264

0

+++++

+++

284

0

263

1

+++++

+++

285

0

Latency

22188ms

14602ms

24751us

14736us

Latency

143ms

390us

47199us

24302us

10us

50133us

-c aes-ecb-null -s 128

Version 1.96

Sequential Output

Sequential Input

Random Seeks

Sequential Create

Random Create

Size

Per Char

Block

Rewrite

Per Char

Block

Num Files

Create

Read

Delete

Create

Read

Delete

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

ip-10-14.eu-west-1.compute.internal

14674M

48834

6

24469

4

63741

5

2649

62

16

263

0

+++++

+++

285

0

263

0

+++++

+++

286

0

Latency

22012ms

14608ms

36346us

13559us

Latency

143ms

379us

48095us

32471us

10us

51386us

-c aes-ecb-benbi -s 128

Version 1.96

Sequential Output

Sequential Input

Random Seeks

Sequential Create

Random Create

Size

Per Char

Block

Rewrite

Per Char

Block

Num Files

Create

Read

Delete

Create

Read

Delete

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

ip-10-14.eu-west-1.compute.internal

14674M

48835

6

24447

4

63683

4

2655

61

16

267

0

+++++

+++

288

0

262

1

+++++

+++

286

0

Latency

21447ms

14568ms

21341us

13945us

Latency

141ms

377us

49271us

19517us

9us

50122us

 In nutshell, tests above showed that there is almost no major difference between aes based encryption sets.We have write IO 48800 and latency about 21000 ms; for read we have IO around 64000 and latency around 21341us. 

To compare results I did a test with default encryption set and key length 128 against worst case scenario - normal EBS with non-EBS optimized instance and encryption on top of LVM. 

Version 1.96

Sequential Output

Sequential Input

Random Seeks

Sequential Create

Random Create

Size

Per Char

Block

Rewrite

Per Char

Block

Num Files

Create

Read

Delete

Create

Read

Delete

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

ip-10-14.eu-west-1.compute.internal

14674M

32515

4

22337

3

76077

5

450.6

4

16

166

0

+++++

+++

178

0

165

0

+++++

+++

178

0

Latency

28895ms

24535ms

229ms

153ms

Latency

170ms

369us

89118us

54455us

10us

104ms

In this case write IO dropped a bit to 32515 and latency increased to 24535ms. But, surprise, read IO increased up to 76000 (unfortunately latency too: 229 ms)

Additional tests just to show the difference between encrypted and non encrypted EBS volume parameters.

Provisioned IOPS (1000) + aes-cbc 128
write
K/sec 40808
Latency 13010 ms
CPU % 8
read
K/sec 56827
Latency 59051us
CPU % 6

Provisioned IOPS (1000) Non encrypted

write
K/sec 41492
Latency 228 ms
CPU % 8
read
K/sec 56884
Latency 102ms
CPU % 7

Different tests done on non-provisioned IOPS, non EBS-optimized, encryption on top of LVM. (Another EC2 instance)

default -256
write
K/sec 26331
Latency 196563ms
CPU % 5
read
K/sec 89304
Latency 1005
CPU % 10

Default -128
write
K/sec 27494
Latency 13847ms
CPU % 5
read
K/sec 102898
Latency 306
CPU % 12

Nonencrypted
write
K/sec 23255
Latency 626 ms
CPU % 4
read
K/sec 95027
Latency 1764
CPU % 12

Bottom line:
1. Encryption heavily increase latency (hundred times).
2. Provisioning IOPS (actually only  reserve IO for your system) and switching to EBS optimized instance (kind of having dedicated network interface to SAN)   won't solve  latency problem. They will help you  only if storage resource become overload by other amazon user or if it already overloaded by others.
3. Encryption set (if as encryption algorithm you use aes) doesn't make a big difference.
4. You can decrease latency by changing a key length - but you will decrease security as well.
5. Using LUKS without LVM could decrease latency (~10 %) as well.

PS. Small bash script below will help you to test encryption sets. You can run this script for different types of  block devices (EBS, ephemeral, EBS+IOPS, LVM or non LVM ) and confirm or refute my conclusions.
Moreover this script will show how you can configure and mount LUKS device :-)

#!/bin/bash
#
#test function
function enc_test() {
while read encryption;
do
cryptsetup luksFormat --key-file '/root/keyfile' $encryption -q $TDISK || continue;
cryptsetup luksOpen --key-file '/root/keyfile' $TDISK d02_enc;
mkfs.ext4 /dev/mapper/d02_enc;
mount /dev/mapper/d02_enc /d02;
mkdir /d02/iotest;
echo $encryption >> enc_test_out.txt;
bonnie++ -d /d02/iotest  -r 3078 -u root -f -b -q >> enc_test_out.txt;
umount /d02
cryptsetup luksClose d02_enc
done <-c aes-ecb-plain -s 128
-c aes-ecb-null -s 128
-c aes-ecb-benbi -s 128
-c aes-cbc-null -s 128
-c aes-cbc-benbi -s 128
-c aes-cbc-plain -s 128
-c aes -s 128
-s 128
EOM
}
TDISK='/dev/test/d02'
echo "Encryption on top of LVM" >> enc_test_out.txt
enc_test
exit 0