One of the most obvious recommendation for the public clouds - use encryption!
Let's talk how to do this practically for Amazon EC2 instance.
1. As encryption we will choose LUKS as kernel level block device encryption.
2. Our Amazon EC2 disk volume will be EBS (we need persistent storage )
But it leads to a many questions: what encryption algorithm and set , key length? Do we need EBS optimized instance and provisioned IOPS? What options will give us better performance?
Almost forget: this encrypted volume will be used by database. So, performance is key requirement for me.
By performance I mean 6 parameters: block write IO and latency, read IO and latency and CPU load during both tests.
Lets start the tests:
Encryption sets test:
To make results more predictable we need to have EBS storage to be persistent (Have guaranteed IO)
According to CPU flags amazon instance has aes support and I'll test only aes encryption sets.
For the IO benchmark I'm using bonnie++
Encryption on device without LVM. IOPS3000
-c aes-ecb-plain -s 128
1.96,1.96,ip-10-14.eu-west-1.compute.internal,1,1399324041,14674M,,,,48625,6,24487,4,,,63744,4,2639,62,16,,,,,271,1,+++++,+++,293,0,272,1,+++++,+++,293,0,,21978ms,14659ms,,19440us,60963us,142ms,364us,47785us,16979us,37us,50287us
-c aes-ecb-null -s 128
1.96,1.96,ip-10-14.eu-west-1.compute.internal,1,1399322496,14674M,,,,48815,6,24488,4,,,63679,5,2631,63,16,,,,,271,1,+++++,+++,304,0,269,1,+++++,+++,289,0,,21779ms,14606ms,,26422us,43205us,143ms,367us,52647us,34894us,10us,48557us
-c aes-ecb-benbi -s 128
1.96,1.96,ip-10-14.eu-west-1.compute.internal,1,1399319920,14674M,,,,48814,6,24483,4,,,63659,5,2704,64,16,,,,,271,1,+++++,+++,291,0,272,1,+++++,+++,294,0,,21306ms,14538ms,,27702us,57492us,144ms,370us,48660us,39901us,10us,48902us
-c aes-cbc-null -s 128
1.96,1.96,ip-10-14.eu-west-1.compute.internal,1,1399317092,14674M,,,,48812,6,24476,4,,,63682,5,2630,62,16,,,,,268,1,+++++,+++,291,0,272,1,+++++,+++,291,0,,21630ms,14586ms,,36893us,45752us,142ms,372us,53419us,21090us,10us,53516us
-c aes-cbc-benbi -s 128
1.96,1.96,ip-10-14.eu-west-1.compute.internal,1,1399331123,14674M,,,,48815,6,24475,4,,,63680,5,2696,64,16,,,,,270,1,+++++,+++,291,0,271,1,+++++,+++,291,0,,21290ms,14656ms,,28083us,38028us,142ms,371us,52241us,29474us,10us,51325us
-c aes-cbc-plain -s 128
1.96,1.96,ip-10-14.eu-west-1.compute.internal,1,1399331889,14674M,,,,48814,6,24477,4,,,63659,5,2592,61,16,,,,,269,1,+++++,+++,293,0,273,1,+++++,+++,294,0,,22076ms,14607ms,,28562us,14134us,145ms,366us,47393us,29521us,10us,48817us
-c aes -s 128
1.96,1.96,ip-10-14.eu-west-1.compute.internal,1,1399330501,14674M,,,,48815,7,24480,4,,,63677,5,2714,63,16,,,,,269,1,+++++,+++,302,0,272,1,+++++,+++,292,0,,21060ms,14595ms,,23679us,13062us,142ms,396us,54966us,29954us,10us,50945us
-s 128
1.96,1.96,ip-10-14.eu-west-1.compute.internal,1,1399327477,14674M,,,,48817,6,24470,4,,,63685,5,2630,62,16,,,,,269,1,+++++,+++,289,0,269,1,+++++,+++,290,0,,17066ms,14615ms,,29639us,64800us,142ms,378us,51051us,27984us,44us,50100us
You can use bon_csv2html program to convert CSV format Bonnie++ to nice looking HTML table like following:
EBS_optimized instance and 3000IOPS encryption test_NO LVM
-c aes-ecb-plain -s 128
-c aes-ecb-null -s 128
-c aes-ecb-benbi -s 128
In this case write IO dropped a bit to 32515 and latency increased to 24535ms. But, surprise, read IO increased up to 76000 (unfortunately latency too: 229 ms)
Additional tests just to show the difference between encrypted and non encrypted EBS volume parameters.
Different tests done on non-provisioned IOPS, non EBS-optimized, encryption on top of LVM. (Another EC2 instance)
Bottom line:
1. Encryption heavily increase latency (hundred times).
2. Provisioning IOPS (actually only reserve IO for your system) and switching to EBS optimized instance (kind of having dedicated network interface to SAN) won't solve latency problem. They will help you only if storage resource become overload by other amazon user or if it already overloaded by others.
3. Encryption set (if as encryption algorithm you use aes) doesn't make a big difference.
4. You can decrease latency by changing a key length - but you will decrease security as well.
5. Using LUKS without LVM could decrease latency (~10 %) as well.
PS. Small bash script below will help you to test encryption sets. You can run this script for different types of block devices (EBS, ephemeral, EBS+IOPS, LVM or non LVM ) and confirm or refute my conclusions.
Moreover this script will show how you can configure and mount LUKS device :-)
#!/bin/bash
#
#test function
function enc_test() {
while read encryption;
do
cryptsetup luksFormat --key-file '/root/keyfile' $encryption -q $TDISK || continue;
cryptsetup luksOpen --key-file '/root/keyfile' $TDISK d02_enc;
mkfs.ext4 /dev/mapper/d02_enc;
mount /dev/mapper/d02_enc /d02;
mkdir /d02/iotest;
echo $encryption >> enc_test_out.txt;
bonnie++ -d /d02/iotest -r 3078 -u root -f -b -q >> enc_test_out.txt;
umount /d02
cryptsetup luksClose d02_enc
done <-c aes-ecb-plain -s 128
-c aes-ecb-null -s 128
-c aes-ecb-benbi -s 128
-c aes-cbc-null -s 128
-c aes-cbc-benbi -s 128
-c aes-cbc-plain -s 128
-c aes -s 128
-s 128
EOM
}
TDISK='/dev/test/d02'
echo "Encryption on top of LVM" >> enc_test_out.txt
enc_test
exit 0
Let's talk how to do this practically for Amazon EC2 instance.
1. As encryption we will choose LUKS as kernel level block device encryption.
2. Our Amazon EC2 disk volume will be EBS (we need persistent storage )
But it leads to a many questions: what encryption algorithm and set , key length? Do we need EBS optimized instance and provisioned IOPS? What options will give us better performance?
Almost forget: this encrypted volume will be used by database. So, performance is key requirement for me.
By performance I mean 6 parameters: block write IO and latency, read IO and latency and CPU load during both tests.
Lets start the tests:
Encryption sets test:
To make results more predictable we need to have EBS storage to be persistent (Have guaranteed IO)
According to CPU flags amazon instance has aes support and I'll test only aes encryption sets.
For the IO benchmark I'm using bonnie++
Encryption on device without LVM. IOPS3000
-c aes-ecb-plain -s 128
1.96,1.96,ip-10-14.eu-west-1.compute.internal,1,1399324041,14674M,,,,48625,6,24487,4,,,63744,4,2639,62,16,,,,,271,1,+++++,+++,293,0,272,1,+++++,+++,293,0,,21978ms,14659ms,,19440us,60963us,142ms,364us,47785us,16979us,37us,50287us
-c aes-ecb-null -s 128
1.96,1.96,ip-10-14.eu-west-1.compute.internal,1,1399322496,14674M,,,,48815,6,24488,4,,,63679,5,2631,63,16,,,,,271,1,+++++,+++,304,0,269,1,+++++,+++,289,0,,21779ms,14606ms,,26422us,43205us,143ms,367us,52647us,34894us,10us,48557us
-c aes-ecb-benbi -s 128
1.96,1.96,ip-10-14.eu-west-1.compute.internal,1,1399319920,14674M,,,,48814,6,24483,4,,,63659,5,2704,64,16,,,,,271,1,+++++,+++,291,0,272,1,+++++,+++,294,0,,21306ms,14538ms,,27702us,57492us,144ms,370us,48660us,39901us,10us,48902us
-c aes-cbc-null -s 128
1.96,1.96,ip-10-14.eu-west-1.compute.internal,1,1399317092,14674M,,,,48812,6,24476,4,,,63682,5,2630,62,16,,,,,268,1,+++++,+++,291,0,272,1,+++++,+++,291,0,,21630ms,14586ms,,36893us,45752us,142ms,372us,53419us,21090us,10us,53516us
-c aes-cbc-benbi -s 128
1.96,1.96,ip-10-14.eu-west-1.compute.internal,1,1399331123,14674M,,,,48815,6,24475,4,,,63680,5,2696,64,16,,,,,270,1,+++++,+++,291,0,271,1,+++++,+++,291,0,,21290ms,14656ms,,28083us,38028us,142ms,371us,52241us,29474us,10us,51325us
-c aes-cbc-plain -s 128
1.96,1.96,ip-10-14.eu-west-1.compute.internal,1,1399331889,14674M,,,,48814,6,24477,4,,,63659,5,2592,61,16,,,,,269,1,+++++,+++,293,0,273,1,+++++,+++,294,0,,22076ms,14607ms,,28562us,14134us,145ms,366us,47393us,29521us,10us,48817us
-c aes -s 128
1.96,1.96,ip-10-14.eu-west-1.compute.internal,1,1399330501,14674M,,,,48815,7,24480,4,,,63677,5,2714,63,16,,,,,269,1,+++++,+++,302,0,272,1,+++++,+++,292,0,,21060ms,14595ms,,23679us,13062us,142ms,396us,54966us,29954us,10us,50945us
-s 128
1.96,1.96,ip-10-14.eu-west-1.compute.internal,1,1399327477,14674M,,,,48817,6,24470,4,,,63685,5,2630,62,16,,,,,269,1,+++++,+++,289,0,269,1,+++++,+++,290,0,,17066ms,14615ms,,29639us,64800us,142ms,378us,51051us,27984us,44us,50100us
You can use bon_csv2html program to convert CSV format Bonnie++ to nice looking HTML table like following:
EBS_optimized instance and 3000IOPS encryption test_NO LVM
-c aes-ecb-plain -s 128
| Version 1.96 | Sequential Output | Sequential Input | Random Seeks | Sequential Create | Random Create | |||||||||||||||||||||
| Size | Per Char | Block | Rewrite | Per Char | Block | Num Files | Create | Read | Delete | Create | Read | Delete | ||||||||||||||
| K/sec | % CPU | K/sec | % CPU | K/sec | % CPU | K/sec | % CPU | K/sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | |||
| ip-10-14.eu-west-1.compute.internal | 14674M | 48831 | 6 | 24485 | 3 | 63757 | 4 | 2632 | 61 | 16 | 264 | 0 | +++++ | +++ | 284 | 0 | 263 | 1 | +++++ | +++ | 285 | 0 | ||||
| Latency | 22188ms | 14602ms | 24751us | 14736us | Latency | 143ms | 390us | 47199us | 24302us | 10us | 50133us | |||||||||||||||
-c aes-ecb-null -s 128
| Version 1.96 | Sequential Output | Sequential Input | Random Seeks | Sequential Create | Random Create | |||||||||||||||||||||
| Size | Per Char | Block | Rewrite | Per Char | Block | Num Files | Create | Read | Delete | Create | Read | Delete | ||||||||||||||
| K/sec | % CPU | K/sec | % CPU | K/sec | % CPU | K/sec | % CPU | K/sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | |||
| ip-10-14.eu-west-1.compute.internal | 14674M | 48834 | 6 | 24469 | 4 | 63741 | 5 | 2649 | 62 | 16 | 263 | 0 | +++++ | +++ | 285 | 0 | 263 | 0 | +++++ | +++ | 286 | 0 | ||||
| Latency | 22012ms | 14608ms | 36346us | 13559us | Latency | 143ms | 379us | 48095us | 32471us | 10us | 51386us | |||||||||||||||
-c aes-ecb-benbi -s 128
| Version 1.96 | Sequential Output | Sequential Input | Random Seeks | Sequential Create | Random Create | |||||||||||||||||||||
| Size | Per Char | Block | Rewrite | Per Char | Block | Num Files | Create | Read | Delete | Create | Read | Delete | ||||||||||||||
| K/sec | % CPU | K/sec | % CPU | K/sec | % CPU | K/sec | % CPU | K/sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | |||
| ip-10-14.eu-west-1.compute.internal | 14674M | 48835 | 6 | 24447 | 4 | 63683 | 4 | 2655 | 61 | 16 | 267 | 0 | +++++ | +++ | 288 | 0 | 262 | 1 | +++++ | +++ | 286 | 0 | ||||
| Latency | 21447ms | 14568ms | 21341us | 13945us | Latency | 141ms | 377us | 49271us | 19517us | 9us | 50122us | |||||||||||||||
In nutshell, tests above showed that there is almost no major difference between aes based encryption sets.We have write IO 48800 and latency about 21000 ms; for read we have IO around 64000 and latency around 21341us.
To compare results I did a test with default encryption set and key length 128 against worst case scenario - normal EBS with non-EBS optimized instance and encryption on top of LVM.
| Version 1.96 | Sequential Output | Sequential Input | Random Seeks | Sequential Create | Random Create | |||||||||||||||||||||
| Size | Per Char | Block | Rewrite | Per Char | Block | Num Files | Create | Read | Delete | Create | Read | Delete | ||||||||||||||
| K/sec | % CPU | K/sec | % CPU | K/sec | % CPU | K/sec | % CPU | K/sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | |||
| ip-10-14.eu-west-1.compute.internal | 14674M | 32515 | 4 | 22337 | 3 | 76077 | 5 | 450.6 | 4 | 16 | 166 | 0 | +++++ | +++ | 178 | 0 | 165 | 0 | +++++ | +++ | 178 | 0 | ||||
| Latency | 28895ms | 24535ms | 229ms | 153ms | Latency | 170ms | 369us | 89118us | 54455us | 10us | 104ms | |||||||||||||||
In this case write IO dropped a bit to 32515 and latency increased to 24535ms. But, surprise, read IO increased up to 76000 (unfortunately latency too: 229 ms)
Additional tests just to show the difference between encrypted and non encrypted EBS volume parameters.
Different tests done on non-provisioned IOPS, non EBS-optimized, encryption on top of LVM. (Another EC2 instance)
default -256
write
K/sec 26331
Latency 196563ms
CPU % 5
read
K/sec 89304
Latency 1005
CPU % 10
write
K/sec 26331
Latency 196563ms
CPU % 5
read
K/sec 89304
Latency 1005
CPU % 10
Default -128
write
K/sec 27494
Latency 13847ms
CPU % 5
read
K/sec 102898
Latency 306
CPU % 12
write
K/sec 27494
Latency 13847ms
CPU % 5
read
K/sec 102898
Latency 306
CPU % 12
Nonencrypted
write
K/sec 23255
Latency 626 ms
CPU % 4
read
K/sec 95027
Latency 1764
CPU % 12
write
K/sec 23255
Latency 626 ms
CPU % 4
read
K/sec 95027
Latency 1764
CPU % 12
Bottom line:
1. Encryption heavily increase latency (hundred times).
2. Provisioning IOPS (actually only reserve IO for your system) and switching to EBS optimized instance (kind of having dedicated network interface to SAN) won't solve latency problem. They will help you only if storage resource become overload by other amazon user or if it already overloaded by others.
3. Encryption set (if as encryption algorithm you use aes) doesn't make a big difference.
4. You can decrease latency by changing a key length - but you will decrease security as well.
5. Using LUKS without LVM could decrease latency (~10 %) as well.
PS. Small bash script below will help you to test encryption sets. You can run this script for different types of block devices (EBS, ephemeral, EBS+IOPS, LVM or non LVM ) and confirm or refute my conclusions.
Moreover this script will show how you can configure and mount LUKS device :-)
#!/bin/bash
#
#test function
function enc_test() {
while read encryption;
do
cryptsetup luksFormat --key-file '/root/keyfile' $encryption -q $TDISK || continue;
cryptsetup luksOpen --key-file '/root/keyfile' $TDISK d02_enc;
mkfs.ext4 /dev/mapper/d02_enc;
mount /dev/mapper/d02_enc /d02;
mkdir /d02/iotest;
echo $encryption >> enc_test_out.txt;
bonnie++ -d /d02/iotest -r 3078 -u root -f -b -q >> enc_test_out.txt;
umount /d02
cryptsetup luksClose d02_enc
done <
-c aes-ecb-null -s 128
-c aes-ecb-benbi -s 128
-c aes-cbc-null -s 128
-c aes-cbc-benbi -s 128
-c aes-cbc-plain -s 128
-c aes -s 128
-s 128
EOM
}
TDISK='/dev/test/d02'
echo "Encryption on top of LVM" >> enc_test_out.txt
enc_test
exit 0
write
K/sec 40808
Latency 13010 ms
CPU % 8
read
K/sec 56827
Latency 59051us
CPU % 6