Junk Cloud POC project.

Junk Cloud is fast deploying CloudStack environment running on KVM hypervisor and using outdated equipment.

As first attempt to build Junk Cloud I have tested CloudStack baremetal support. What I expect from baremetal support?

• reduce time to configure and deploy each host
• you can add computing resources (physical hosts) to the cloud in a minutes instead of hours. 
• fast deployment of the new private cloud
• allow easy utilization of outdated hardware

Unfortunately CloudStack built-in baremetal support is limited: http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.2.0/html/Installation_Guide/choosing-a-hypervisor.html

• not support advanced networking
• not support central storage (SAN)
• not support vmotion and HA

So, it was decided to build own version of the barematal support for theCloudStack:

• network boot (pxe boot)
• network auto install of lnux + kvm + network + CloudStack
• storage configuration 
• upon completion auto selfprovisionng to the CloudStack as a new host (resource)

The Anaconda install script for building and configuring Centos based KVM host for CloudStack (Advanced networking support) on GitHub:  https://github.com/IhorKravchuk/cloudstack_ingvar/blob/master/Centos_6x64_kvm_cloudstack.cfg

This script is missing auto selfprovisionng to the CloudStack feature for test purpose. It will be updated after final tests.

Tested and suggested deployment architecture (for the small deployments):

From my point of view implementation of proposed architecture will be:

• core components: cloud controller + network storage installed and configured. 
• to add new host just connect it to network and power.
• all rest done automatically.

Private Cloud? it's that simple:

Build a Cloud Controller:

Add NFS NAS:

And finally add all your old hardware:

CloudStack-BIND dns integration.

How to integrate CloudStack DNS with your organization DNS environment?

All recommendation bellow based on the approach of using one sub-domain per CloudStack network.

 1. The simplest way: conditional forwarding from windows DNS server to CloudStack VR for the corresponding sub-domain or (BIND) sub-domain delegation to the VR. 

pros: easy to build

con: VR are running as not authoritative DNS for sub-domain, each time new record added dnsmasq service restarts and you have 2-3 sec of downtime (up to v4.1), all request going to VR.

  2.  CloudStack --> BIND full integration:

Following program solves DNS integration issues between CloudStack VR's DNS service and BIND DNS.

This program assumes that you are using sub-domain per network(each network has own sub-domain) (IMHO the best way fro naming instances in CloudStack)

How it works:

On event or on schedule program call CloudStack API and get list of Networks and list of VM. Using theses lists and preconfigured domain settings it creates the zone file for BIND, push it to server and refresh BIND.

This program could be run using 2 different ways: 

1.  being installed on DNS server and update DNS records on scheduled interval. (schedule driven)
2.  being installed on CloudStack management server and listen for the new vm deployment using CloudStack catalina.out log. (event driven)

The source code for the event driven version on GitHub: https://github.com/IhorKravchuk/cloudstack_ingvar/blob/master/dns_builder.py

Proposed version running on heavy used CloudStack environment with very frequent SaltStack driven deployments with almost no issue. 

The script is under active development and testing and will be updated.
Version 2.0 released with all parameters now loaded from dns_builder.conf file and local and remote DNS servers support. 

PS:

Some times this script fails because of non-expeted  ClaudStack response:

 
    dns_table = get_dns()
  File "/usr/bin/dns_builder.py", line 134, in get_dns
    output+= vm['name'] + "." + net_dict[vm['nic'][0]['networkname']] + "\t\t\t300\tIN\tA\t" + vm['nic'][0]['ipaddress'] +" \n"
IndexError: list index out of range

Instead of exception handling to keep this script running I used supervisord daemon.

It starts the script, makes sure it running, restarts in case of failure and takes care of logs.

Part of supervizord.conf file related to the script:

[program:dns_builder]
command=/usr/bin/dns_builder.py      ; the program (relative uses PATH, can take args)
priority=100                ; the relative start priority (default 999)
autostart=true              ; start at supervisord start (default: true)
autorestart=true            ; retstart at unexpected quit (default: true)
;startsecs=10                ; number of secs prog must stay running (def. 10)
startretries=5              ; max # of serial start failures (default 3)
;exitcodes=0,2               ; 'expected' exit codes for process (default 0,2)
;stopsignal=QUIT             ; signal used to kill process (default TERM)
;stopwaitsecs=10             ; max num secs to wait before SIGKILL (default 10)
;user=chrism                 ; setuid to this UNIX account to run the program
log_stdout=true             ; if true, log program stdout (default true)
log_stderr=true             ; if true, log program stderr (def false)
logfile=/var/log/dns_builder.log    ; child log path, use NONE for none; default AUTO
logfile_maxbytes=1MB        ; max # logfile bytes b4 rotation (default 50MB)
logfile_backups=2          ; # of logfile backups (default 10)

CloudStack IaaS insecure password reset

CloudStack is one of the major and popular IaaS  (Infrastructure as a Servile)  platform.
http://en.wikipedia.org/wiki/Apache_CloudStack

Below small review of the password reset process in the CloudStack.

The purpose of password reset procedure - to allow user during deployment of the VM (virtual machine) template or after this to reset root (administrator password) of VM. Because of the main idea of IaaS to give user ability to help himself  this is one of the key functionality.

From user perspective process looks like:

1. start new VM or click reset password on any stopped VM
2. Get popup with new root password
3. Log in using console, rdp or ssh using new password.

Let's see what behind the scene:

Each network in CloudStack has dedicated router (VR) which doing dhcp, dns, loadbalancing, firewalling and password reset for whole subnet.

On VR we have following components of password reset service:

1. Process listening on port 8080:
 socat -lf /var/log/cloud.log TCP4-LISTEN:8080,reuseaddr,crnl,bind=10.0.146.2 SYSTEM:/opt/cloud/bin/serve_password.sh "$SOCAT_PEERADDR"
it actually waiting for request like : DomU_Request: send_my_password

2. script actually doing the job: /opt/cloud/bin/serve_password.sh

3. and password file: /var/cache/cloud/passwords having all passwords in clear text with filesystem permissions -rw-r--r--
  10.0.146.15=rD7nudcze
 10.0.146.13=jB9kbknvq
 10.0.146.181=saved_password

after each password request VR replace corresponding password in password file  by "saved_password"

On VM template and  VM instance you have script: /etc/init.d/cloud-set-guest-password
This script automatically request root password from VR at each system startup and update it .

The password request procedure is:

1. Client VM parse local  network setting and getting DHCP server IP.

2. Client send clear text  request like wget -q -t 3 -T 20 -O - --header "DomU_Request: send_my_password" $PASSWORD_SERVER_IP:8080

3. If it get password it will use it. If it gets "saved_password" it won't do anything.

Security problems:

1. clear text password storage on VR

2. Clear text password transmission over the network

3. Missing password sever authentication (only by IP)

4. auto-starting password reset service.

Conclusion:

If attacker has access to the one instance into cloudstack network by spoofing password server(VR) IP he will able to compromise other instances int this subnet after their reboot. Having access to VR - will be able to compromise all nodes.