A brief overview of OpenHAB

What is Openhab?

OpenHAB is a Java based home automation system that allows you to control multiple technologies all from a single location. It is built on the principal of a single “Home automation Bus” where different technologies communicate using a common “Event bus”.

This means that you are able, on any device that can run the JVM, you can run your home automation, without the need to purchase expensive controllers, however, with OpenHAB manual work is required to get the system working, as “batteries are not included”.

OpenHAB is OpenSource which means that the community as a whole are able to fix bugs, and unlike Cloud based services, or propitiatory controllers, you do not need to rely on the company (or in this case, team of people) to still be around to do feature improvements, the community as a whole can do so.

What can OpenHAB connect to?

OpenHAB has a plethora of addons that are available, ranging from HomeAutomation protocols (such as ZWave, EnOcean, RFXCOM (Lightwave RF can be controlled using this)). Some consumer devices are also able to be integrated (Sonos, Samsung TVs, LG TVs) It also has the ability to integrate with various notification systems including EMail,XMPP (Jabber / Google Talk), NotifyMyAndorid. It can also integrate with Calendaring systems like Google Calendar

Lets dive in!

For the examples that we’re going to use in the this document we will use ZWave based equipment

The hardware that will be in use is as follows

Initial Setup of OpenHAB

As OpenHAB doesn’t have an installer it has to be installed by hand. We’re going to do this into /opt

# Create directory structure
cd /opt
mkdir /opt/openhab /opt/oh-addons-dist

# Download Runtime
curl -LO

# Download Addons
curl -LO

# Unzip app
cd /opt/openhab
unzip /opt/*

# Unzip addons
cd /opt/oh-addons-dist
unzip /opt/*

Installation of Addons

We’re going to copy the addons mentioned above into place

cd /opt/openhab/addons

mv /opt/oh-addons-dist/org.openhab.persistence.rrd4j-1.7.0.jar /opt/openhab/addons/

Installation of HABMIN

OpenHAB does not provide a WebUI for administration so we’re going to install a community project called HABMin to provide us this

# Install habmin
cd /opt/openhab/addons
curl -Lo
curl -Lo org.openhab.binding.zave-1.7.0.jar
cd /opt/openhab/webapps/
curl -Lo /opt/openhab/webapps/
mv HABmin-master/ habmin

Configuration of OpenHAB


At this point we have OpenHAB installed, we now however need to provide it a Configuration file, which should be in /opt/openhab/configurations//openhab.cfg


You will need to use your favourite Text Editor (vi, nano, emacs) to edit this file




zwave:port = /dev/ttyACM0

zwave:healtime = 2

zwave:masterController = true











logging:pattern=%date{ISO8601} – %-25logger: %msg%n


In this file we’ve set the ZWave controller to be the first ACM device (/dev/ttyACM0), if you don’t have any 3g Modules or other ZWave controllers plugged into this machine this will (likely) be the device you would need to use.


We are also disabling security for the purpose of this demo, you may want to enable this going forward!


Starting OpenHAB


This is done using the script



Setup of ZWave Devices


We’re going to use HABMin to handle the inclusion of devices into the network.

Open a webbrowser to the machines IP address into the /habmin/ folder. For instance if this is the address you would go to is

Click the Configuration tab.

Click Bindings (Bottom Left corner)

Click the ZWave binding

Click the Devices Tab

For each device you wish to add, click the include button and follow the instructions for the device to include it into the network.


Once this is completed the Devices tab should show the list of devices, similar to below (be aware this has more devices that we’re going to look at configuring!)



It is useful to be able to see what an item’s previous value has been, and as such we want to store these so that they survive over restarts. We’re going to use the rrd4j addon for this. It is configured by putting the following in /opt/openhab/configurations/persistence/rrd4j.persist


// persistence strategies have a name and a definition and are referred to in the “Items” section

Strategies {

   everyHour : “0 0 * * * ?”

   everyDay  : “0 0 0 * * ?”

   everyMinute : “0 * * * * ?”

   // if no strategy is specified for an item entry below, the default list will be used

   default = everyChange




* Each line in this section defines for which item(s) which strategy(ies) should be applied.

* You can list single items, use “*” for all items or “groupitem*” for all members of a group

* item (excl. the group item itself).


Items {

   // persist all items once a day and on every change and restore them from the db at startup

   * : strategy = everyChange, everyMinute, everyDay, restoreOnStartup





In OpenHAB an item is a individual attribute of a device that is configured within OpenHAB.


The first set of Items we will configure is the AeonLabs multi Sesnsor

We need to create a file in /opt/openhab/configurations/items/. Lets create /opt/openhab/configurations/items/livingroom.items

Within this file we define a set of items


Number sensor_1_temp “Temperature [%.1f °C]” {zwave=”3:command=sensor_multilevel,sensor_type=1″}

Number sensor_1_humidity “Humidity    [%.0f %%]” {zwave=”3:command=sensor_multilevel,sensor_type=5”}

Number sensor_1_luminance “Luminance    [%.0f Lux]” {zwave=”3:command=sensor_multilevel,sensor_type=3”}

Contact sensor_1_motion “Motion [%s]” {zwave=”3:command=sensor_binary”}

Number sensor_1_battery “Battery [%s %%]” {zwave=”3:command=battery”}


The format of these files is as follows


ItemType ItemName ItemLabel <ItemIcon> (ItemGroup) {ItemBinding}

We are not going to cover off Icons or Groups in this guide.

The ItemTypes that are available are

  • Color
  • Contact
  • DateTime
  • Dimmer
  • Group
  • Number
  • RollerShutter
  • String
  • Switch


The ItemLabel has the ability to be formatted using standard Java formatter syntax, which will not be covered here other than to say [%.1f °C] will display the temperature to 1 decimal point, ie 23.4.


The Binding is where we configure which device we actually are querying. In this example I’m using ZWave Node 3. We specify the command class that needs be be used, for instance SENSOR_MULTILEVEL and then the sensor type, this is documented at


When this file is saved, you should now be able to see the items in HABMin under Configuration -> Items and Groups


The next devices we will configure are the NorthQ meters. These can be placed in any filename in the items directory. Lets create them as power.items


Group Power <energy>

Number power_1_battery “Electricity Meter Battery [%s %%]” <battery> (Power) {zwave=”4:command=BATTERY,refresh_interval=3600″}

Number power_1_usage_total “KWH [%s]” (Power) {zwave=”4:command=METER,meter_scale=E_KWh,refresh_interval=450″}

Number power_1_usage “Watt [%.2f]” (Power)


Lets put the gas in gas.items


Group Gas

Number gas_1_battery “Gas Meter Battery [%s %%]” (Gas) {zwave=”6:command=BATTERY,refresh_interval=3600″}

Number gas_1_usage_total “m3 [%s]” (Gas) {zwave=”6:command=METER,refresh_interval=450″}

Number gas_1_usage “m3 [%.2f]” (Gas)


You will notice that we have a power_1_usage and gas_1_usage item that does not have a binding. We’ll look at this in a few moments


Our final set of items that we want to create are for the Qubino Relay. Lets create this in relay.items


Switch light1_state “light1 [%s]” {zwave=”10:command=switch_binary”}

Number light1_power “light1 Watt [%s]” {zwave=”10:command=meter,refresh_interval=60″}




Rules allow you to create logic within the OpenHAB system, for instance when there is movement, turn on the light.

We’re going to create two rules, one for our gas meter and one for our power meter.

Both of these devices will, by default, only return the amount of power or gas used since they were installed, but I find it more interesting / useful to have a “point in time” amount that is being used.

Create the file /opt/openhab/configuration/rules/northq.rules


rule gas1_current_usage_update


               // When power meter is updated

               Item gas_1_usage_total received update


               // Update current usage with the difference between this, and the previous update to get our spot usage of m3.






rule power1_current_usage_update


               // When power meter is updated

               Item power_1_usage_total received update


               // Update current usage with the difference between this, and the previous update and multiply by 1000 to give us the total in watts.



                                       * 1000





These rules simply take the last value that the usage total was 5 minutes ago, and sets the usage to the difference between the two readings




Sitemaps provide us a way to display items on a device, such as within OpenHAB’s applications or on the web page.


They can get quite complex and as such I’m not going to cover them off in details, but a sample sitemap would be /opt/openhab/configuration/sitemap/default.xml

sitemap default label=”Home”


                                       Frame label=”Hallway” {

                                               Switch item=light1_state

                                               Text item=light1_power


                                       Frame label=”Sensor1″ {

                                       Text item=sensor_1_temp valuecolor=[>25=”orange”,>15=”green”,>5=”orange”,<=5=”blue”] {

                                       Text item=sensor_1_humidity

                                       Text item=sensor_1_luminance

                                       Text item=sensor_1_battery

                                       Text item=sensor_1_motion


               Frame label=”Energy” {

                       Text item=power_1_usage label=”Power usage [%.0f Watts]” icon=”energy”

                       Chart item=power_1_usage period=h refresh=6000

                       Text item=power_1_battery

               Text item=gas_1_usage label=”Gas usage [%.2f m3]” icon=”fire-on”

                       Chart item=gas_1_usage period=h refresh=6000

                       Text item=gas_1_battery





‘Instant’ Upload to Rackspace CloudFiles

Using inotify and the ‘swift’ client tools it is possible to automatically upload files to cloudfiles as they are written to disk.

This code is untested and might cause planes to drop from the sky, use it at your own risk!

inotifywait -mr --format '%w%f' -e close_write $DIRECTORY | while read filename; do
    if [ ! -e "$1" ]; then
       sleep 2
       if [ ! -e "$1" ]; then
    swift upload  -s -A -U $USERNAME -K $KEY -v $VERSION $CONTAINER $filename 

Google Authenticator F5 IRule

Two Factor authentication is rather hit and miss in terms of support from web apps.

A quick look around the web turns up an article on DevCentral for a solution to implement google authentication with ldap. As I don’t run a LDAP server at home I needed to hack up the script a bit. This iRule implements the two factor side of things from the above article, but skips the LDAP side of things, as it’s not needed!

when RULE_INIT {
  # auth parameters
  set static::auth_cookie "bigip_virtual_auth"
  set static::auth_cookie_aes_key "AES 128 abcdef0123456789abcdef0123456789"
  set static::auth_timeout 86400
  set static::auth_lifetime 86400

  # name of datagroup that holds AD user to Google Authenticator mappings
  set static::user_to_google_auth_class "user_to_google_auth"

  # lock the user out after x attempts for a period of x seconds
  set static::lockout_attempts 3
  set static::lockout_period 30

  # 0 - logging off
  # 1 - log only successes, failures, and lockouts
  # 2 - log every attempt to access virtual as well as authentication process details
  set static::debug 1

  # HTML for login page
  set static::login_page { 

Authorization Required

Google Authenticator code:
} } when CLIENT_ACCEPTED { # per virtual status tables for lockouts and users' auth_status set lockout_state_table "[virtual name]_lockout_status" set auth_status_table "[virtual name]_auth_status" set authid_to_user_table "[virtual name]_authid_to_user" # record client IP, [IP::client_addr] not available in AUTH_RESULT set user_ip [IP::client_addr] # set initial values for auth_id and auth_status set auth_id [md5 [expr rand()]] set auth_status 2 set auth_req 1 } when HTTP_REQUEST { if { $auth_req == 1 } { # track original URI user requested prior to login redirect set orig_uri [b64encode [HTTP::uri]] if { [HTTP::cookie exists $static::auth_cookie] && !([HTTP::path] starts_with "/google/auth/login")} { set auth_id_current [AES::decrypt $static::auth_cookie_aes_key [b64decode [HTTP::cookie value $static::auth_cookie]]] set auth_status [table lookup -notouch -subtable $auth_status_table $auth_id_current] set user [table lookup -notouch -subtable $authid_to_user_table $auth_id_current] if { $auth_status == 0 } { if { $static::debug >= 2 } { log local0. "$user ($user_ip): Found valid auth cookie (auth_id=$auth_id_current), passing request through" } } else { if { $static::debug >= 2 } { log local0. "Found invalid auth cookie (auth_id=$auth_id_current), redirecting to login"} HTTP::redirect "/google/auth/login?orig_uri=$orig_uri" } } elseif { ([HTTP::path] starts_with "/google/auth/login") && ([HTTP::method] eq "GET") } { HTTP::respond 200 content $static::login_page } elseif { ([HTTP::path] starts_with "/google/auth/login") && ([HTTP::method] eq "POST") } { set orig_uri [b64decode [URI::query [HTTP::uri] "orig_uri"]] HTTP::collect [HTTP::header Content-Length] } else { if { $static::debug >= 2 } { log local0. "Request for [HTTP::uri] from unauthenticated client ($user_ip), redirecting to login" } HTTP::redirect "/google/auth/login?orig_uri=$orig_uri" } } } when HTTP_REQUEST_DATA { if { $auth_req == 1} { set user "" set ga_code "" foreach param [split [HTTP::payload] &] { set [lindex [split $param =] 0] [lindex [split $param =] 1] } if { ($user ne "") && ([string length $ga_code] == 6) } { set ga_code_b32 [class lookup $user $static::user_to_google_auth_class] set prev_attempts [table incr -notouch -subtable $lockout_state_table $user] table timeout -subtable $lockout_state_table $user $static::lockout_period if { $prev_attempts = 2 } { log local0. "$user ($user_ip): Starting authentication sequence, attempt #$prev_attempts" } # begin - Base32 decode to binary # Base32 alphabet (see RFC 4648) array set static::b32_alphabet { A 0 B 1 C 2 D 3 E 4 F 5 G 6 H 7 I 8 J 9 K 10 L 11 M 12 N 13 O 14 P 15 Q 16 R 17 S 18 T 19 U 20 V 21 W 22 X 23 Y 24 Z 25 2 26 3 27 4 28 5 29 6 30 7 31 } set l [string length $ga_code_b32] set n 0 set j 0 set ga_code_bin "" for { set i 0 } { $i < $l } { incr i } { set n [expr $n <= 8 } { set j [incr j -8] append ga_code_bin [format %c [expr ($n & (0xFF <> $j]] } } # end - Base32 decode to binary # begin - HMAC-SHA1 calculation of Google Auth token set time [binary format W* [expr [clock seconds] / 30]] set ipad "" set opad "" for { set j 0 } { $j < [string length $ga_code_bin] } { incr j } { binary scan $ga_code_bin @${j}H2 k set o [expr 0x$k ^ 0x5C] set i [expr 0x$k ^ 0x36] append ipad [format %c $i] append opad [format %c $o] } while { $j < 64 } { append ipad 6 append opad \ incr j } binary scan [sha1 $opad[sha1 ${ipad}${time}]] H* token # end - HMAC-SHA1 calculation of Google Auth hex token # begin - extract code from Google Auth hex token set offset [expr ([scan [string index $token end] %x] & 0x0F) <= 2 } { log local0. "$user ($user_ip): Google Authenticator TOTP token matched" } set auth_status 0 set auth_id_aes [b64encode [AES::encrypt $static::auth_cookie_aes_key $auth_id]] table add -subtable $auth_status_table $auth_id $auth_status $static::auth_timeout $static::auth_lifetime table add -subtable $authid_to_user_table $auth_id $user $static::auth_timeout $static::auth_lifetime if { $static::debug >= 1 } { log local0. "$user ($user_ip): authentication successful (auth_id=$auth_id), redirecting to $orig_uri" } HTTP::respond 302 "Location" $orig_uri "Set-Cookie" "$static::auth_cookie=$auth_id_aes;" HTTP::collect } else { if { $static::debug >= 1 } { log local0. "$user ($user_ip): authentication failed - Google Authenticator TOTP token not matched" } HTTP::respond 200 content $static::login_page } } else { if { $static::debug >= 1 } { log local0. "$user ($user_ip): could not find valid Google Authenticator secret for $user" } HTTP::respond 200 content $static::login_page } } else { if { $static::debug >= 1 } { log local0. "$user ($user_ip): attempting authentication too frequently, locking out for ${static::lockout_period}s" } HTTP::respond 200 content "You've made too many attempts too quickly. Please wait $static::lockout_period seconds and try again." } } else { HTTP::respond 200 content $static::login_page } } }

Auditd logging all commands

A common requirement for PCI-DSS is for all commands run by a user who has admin privileges to be logged. There are many ways to do this, most of the time people will opt for a change to the bash configuration or to rely on sudo. There are many ways arround this (such as providing the command that you wish to run as a paramater to SSH). As the linux kernel provides a full auditing system. Using a utility such as auditd, we are able to log all commands that are run. The configuration for this is actually quite simple. In /etc/audit/audit.rules we need to ensure that the following exists.

-a exit,always -F arch=b64 -S execve
-a exit,always -F arch=b32 -S execve

This will capture any execve system call (on exit) and will log this to the auditd log. A log entry will look similar to below.

type=SYSCALL msg=audit(1318930500.123:3020171): arch=c000003e syscall=59 success=yes exit=0 a0=7fff65179def a1=7fff65179ec0 a2=7fff6517d060 a3=7ff54ee36c00 items=3 ppid=9200 pid=9202 auid=0 uid=1000 gid=100 euid=1000 suid=1000 fsuid=1000 egid=100 sgid=100 fsgid=100 tty=(none) ses=4 comm="xscreensaver-ge" exe="/usr/bin/perl" key=(null)
type=EXECVE msg=audit(1318930500.123:3020171): argc=5 a0="/usr/bin/perl" a1="-w" a2="/usr/bin/xscreensaver-getimage-file" a3="--name" a4="/home/welby/Pictures"
type=EXECVE msg=audit(1318930500.123:3020171): argc=3 a0="/usr/bin/perl" a1="-w" a2="/usr/bin/xscreensaver-getimage-file"
type=CWD msg=audit(1318930500.123:3020171): cwd="/home/welby/Downloads"
type=PATH msg=audit(1318930500.123:3020171): item=0 name="/usr/bin/xscreensaver-getimage-file" inode=208346 dev=fe:02 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1318930500.123:3020171): item=1 name=(null) inode=200983 dev=fe:02 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1318930500.123:3020171): item=2 name=(null) inode=46 dev=fe:02 mode=0100755 ouid=0 ogid=0 rdev=00:00

This should keep most auditors happy 🙂


Moving part of an lvm vg from one pv to another

Lets say that you’ve got multiple physical volumes (PV) in a Volume Group (VG) and you want to migrate the extents from one PV to another, this can be acomplished with a quick and easy pvmove command.

For example

pvdisplay -m
--- Physical volume ---
PV Name /dev/sdb1
PV Size 2.73 TiB / not usable 4.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 714539
Free PE 0
Allocated PE 714539
PV UUID XWiRzE-Ol3d-38En-ND6b-qo93-4zeF-xv8zDv

--- Physical Segments ---
Physical extent 0 to 604876:
Logical volume /dev/INTEL_RAID/MEDIA
Logical extents 0 to 604876
Physical extent 604877 to 617676:
Logical volume /dev/INTEL_RAID/backups_mimage_0
Logical extents 25600 to 38399
Physical extent 617677 to 617701:
Logical volume /dev/INTEL_RAID/EPG
Logical extents 0 to 24
Physical extent 617702 to 643301:
Logical volume /dev/INTEL_RAID/backups_mimage_0
Logical extents 0 to 25599
Physical extent 643302 to 714538:
Logical volume /dev/INTEL_RAID/MEDIA
Logical extents 604877 to 676113

--- Physical volume ---
PV Name /dev/sdc1
PV Size 2.04 TiB / not usable 2.00 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 535726
Free PE 430323
Allocated PE 105403
PV UUID laOuKy-5FZa-cJ3h-JffV-qUub-diKC-O0wVqK

--- Physical Segments ---
Physical extent 0 to 25599:
Logical volume /dev/INTEL_RAID/backups_mimage_1
Logical extents 0 to 25599
Physical extent 25600 to 54202:
Logical volume /dev/INTEL_RAID/MEDIA
Logical extents 676114 to 704716
Physical extent 54203 to 67002:
Logical volume /dev/INTEL_RAID/NZB_DOWNLOAD
Logical extents 0 to 12799
Physical extent 67003 to 79802:
Logical volume /dev/INTEL_RAID/backups_mimage_1
Logical extents 25600 to 38399
Physical extent 79803 to 105402:
Logical volume /dev/INTEL_RAID/OLD_VM
Logical extents 0 to 25599
Physical extent 105403 to 535725:

From here you can see that /dev/INTEL_RAID/MEDIA is a Logical Volume (LV) on both PVs within the VG. If I was wanting to grow my mirrored LV, which requires space on both PVs, I’d have to migrate some of the extents of another LV. If I wanted to move some of the MEDIA lv, I should be able to do the following

pvmove /dev/sdb:643302-714538 /dev/sdc

This will move extents 643302-714538 to the next contiguious block on /dev/sdc


Dahdi In LXC

At home we use various VoIP providers to either get free calls to various places (GTalk/GVoice to america for instance) and various other destinations over SIP providers

I’ve been using Asterisk for years (I remeber the 0.7 release) and have implemented it for companies before, usually with no issues, baring the continuall deadlocks in the 1.2 range. Recently I enabled my VoIP network segment for IPv6 only to find that GTalk stoped working on IPV6 Day. After a bit of digging about, it seems that Asterisk 1.8 does support IPV6! But, gtalk and similar are not supported, SIP is infact the only first class citezen it seems.

I’ve toyed with using freeswitch before, but unfortuantly have had varied sucsess with FreeTDM to Dahdi with BT caller ID and the likes. I did hack in support for it, but I’m not too sure if I trust my code, as my C is quite rusty to say the least.

I did however come up with another solution!

As I’m running a moderatly new Linux Kernel I can use LXC – Linux Containers – which are effectilvy the same idea as a wpar, chroot, openvz, whatever. After setting up asterisk in the LXC I needed to expose my Dahdi card to it. LXC allows you to restrict access on a per device basis. I’ve setup Dahdi on the host machine as normal so the kernel modules can be loaded etc. Once this is done I’ve preformed the following within the LXC

cd /
mkdir dev/dahdi
mknod dev/dahdi/1 c 196 1
mknod dev/dahdi/2 c 196 2
mknod dev/dahdi/3 c 196 3
mknod dev/dahdi/4 c 196 4
mknod dev/dahdi/channel c 196 254
mknod dev/dahdi/ctl c 196 0
mknod dev/dahdi/pseudo c 196 255
mknod dev/dahdi/timer c 196 253
mknod dev/dahdi/transcode c 196 250

This creates the Device nodes within /dev/ for my 4 dahdi channels (3FXS 1FXO if anyone is interested). After this I’ve added the following to the lxc config file, to allow the LXC to have access to these devices

# If you want to be lazy just add this line
#lxc.cgroup.devices.allow = c 196:* rwm

#Otherwise use the following
lxc.cgroup.devices.allow = c 196:0 rwm
lxc.cgroup.devices.allow = c 196:1 rwm
lxc.cgroup.devices.allow = c 196:2 rwm
lxc.cgroup.devices.allow = c 196:3 rwm
lxc.cgroup.devices.allow = c 196:4 rwm
lxc.cgroup.devices.allow = c 196:250 rwm
lxc.cgroup.devices.allow = c 196:253 rwm
lxc.cgroup.devices.allow = c 196:254 rwm
lxc.cgroup.devices.allow = c 196:255 rwm

This will obviously only work for the first 4 dahdi channels, but if you need more, just continue adding the 196:x lines, replacing x with the channel number, and also ensuring that you create the device nodes in the same way

Linux Projects Software Uncategorized

A quick (and quite unscientific!) break down of Rackspace CloudFiles UK vs Amazon S3 (Ireland)

(Disclaimer – I’m a Rackspace Employee, the postings on this site are my own, may be bias, and don’t necessarily represent Rackspace’s positions, strategies or opinions. These tests have been preformed independently from my employer by my self)

As Rackspace have recently launched a ‘beta’ Cloudfiles service within the UK I thought I would run a few tests to compare it to Amazon’s S3 service running from Eire (or Southern Ireland).

I took a set of files, totalling 18.7GB, with file sizes ranging from between 1kb and 25MB, text files, and contents being mainly Photos (both JPEG and RAW (cannon and nikon), plain text files, GZiped Tarballs and a few Microsoft Word documents just for good measure.

The following python scripts were used:

Cloud Files

import cloudfiles
import sys,os

local_file_list = sys.stdin.readlines()
cf = cloudfiles.get_connection(api_username, api_key, authurl=auth_url)
containers = cf.get_all_containers()
for container in containers:
    if == dest_container:
            backup_container = container

def upload_cf(local_file):
    u = backup_container.create_object(local_file)

for local_file in local_file_list:
        local_file = local_file.rstrip()
        local_file_size = os.stat(local_file).st_size/1024
        print "uploading %s (%dK)" % (local_file, local_file_size)



import cloudfiles
import sys,os

#Setup the connection
cf = cloudfiles.get_connection(api_username, api_key, authurl=auth_url)

#Get a list of containers
containers = cf.get_all_containers()

# Lets setup the container
for container in containers:
    if == dest_container:
            backup_container = container

#Create the container if it does not exsit
except NameError:
    backup_container = cf.create_container(dest_container)

# We've now got our container, lets get a file list
def build_remote_file_list(container):
    remote_file_list = container.list_objects_info()
    for remote_file in remote_file_list:
        f = open(remote_file['name'],'w')
        rf = container.get_object(remote_file['name'])
        print remote_file['name']
        for chunk in
remote_file_list = build_remote_file_list(backup_container)


from boto.s3.connection import S3Connection
from boto.s3.key import Key
import sys,os

dest_container = "CONTAINER"

s3 = S3Connection('api','api_secret')

buckets = s3.get_all_buckets()

for container in buckets:
    if == dest_container:
                backup_container = container

def build_remote_file_list(container):
    remote_file_list = container.list()
    for remote_file in remote_file_list:
        print remote_file
        f = open(remote_file,'w')
        rf = container.get_key(remote_file)
        #print remote_file['name'

local_file_list = sys.stdin.readlines()

def upload_s3(local_file):
    k = Key(backup_container)
    k.key = local_file

for local_file in local_file_list:
        local_file = local_file.rstrip()
        local_file_size = os.stat(local_file).st_size/1024
        print "uploading %s (%dK)" % (local_file, local_file_size)


from boto.s3.connection import S3Connection
from boto.s3.key import Key
import sys,os

dest_container = "CONTAINER"

s3 = S3Connection('api','apt_secret')

buckets = s3.get_all_buckets()

for container in buckets:
    if == dest_container:
                backup_container = container

def build_remote_file_list(container):
    remote_file_list = container.list()
    for remote_file in remote_file_list:
        f = open(,'w')
        rf = container.get_key(
        #print remote_file['name'

remote_file_list = build_remote_file_list(backup_container)

The test was preformed from a Linux host which has a 100MBit connection (Uncapped/unthrottled) in London, however the test was also preformed with almost identical results from a machine in Paris (also 100mbit). Tests were also run from other locations (Dallas Fort Worth – Texas, my home ISP ( however these locations were limited to 25mbit and 24mbit , and both reached their maximum speeds. The tests were as follows:

  • Download files from Rackspace Cloudfiles UK (these had been uploaded previously) – This is downloaded directly via the API, NOT via a CDN
  • Upload the same files to S3 Ireland
  • Upload the same files to a new “container” at Rackspace Cloudfiles UK
  • Download the files from S3 Ireland – This is downloaded directly via the API, NOT via a CDN
  • The average speeds for the tests are as follows:
    Download: 90Mbit/s
    Upload: 85MBit/s
    S3 Ireland
    Download: ~40Mbit/s
    Upload : 13Mbit/s


  • Cloud files seems to be able to max out a 100mbit connection for both File
  • S3 seems to have a cap of 13mbit for inbound file transfers?
  • S3 seems to either be extremely unpredictable on file transfer speeds for downloading files via the API, or there is some form of cap after a certain amount of data transferred, or there was congestion on the AWS network
  • Below is a graph showing the different connection speeds achieved using CF & S3

    As mentioned before this is a very unscientific test (and I’d say that these results have not been replicated from as many locations or as many times as I’d like to, so I would take them with a pinch of salt) , but it does appear that Rackspace cloudfiles UK is noticeably faster than S3 Ireland


    Lighttpd: mod_security via mod_magnet

    In most large enterprises there is a requirement to comply with various standards. The hot potato in the Ecommerce space at the moment (and has been for a few years!) is PCI-DSS.

    At $WORK we have to comply with PCI-DSS with the full audit and similar occurring due to the number of transactions we perform. Recently we’ve deployed lighttpd for one of our platforms, which has caused an issue for our Information Security Officers and Compliance staff.

    PCI-DSS 6.6 requires EITHER a Code review to be preformed, which whilst this may seem to be an easy task, when you’re talking about complex enterprise applications following a very……… agile development process it’s not always an option. The other option is to use a WAF (Web Application Firewall). Now there are multiple products available that sit upstream and perform this task. There is however an issue if you use SSL for your traffic. Most WAF will not do the SSL decryption / reencryption between the client and server (effectively becoming a Man in the Middle). There are however a few products which do this, F5 networks’ ASM being one that springs to mind. Unfortunately this isn’t always an option due to licensing fees and similar. An alternative is to run a WAF on the server its self. A common module for this is Mod_Security for Apache. Unfortunately, a similar module does not exist for Lighttpd.

    In response to $WORKs requirement for this I’ve used mod_magnet to run a small lua script to emulate the functionality of mod_security (to an extent at least!). Please note that mod_magent is blocking, so will cause any requests to be blocked until the mod_magnet script has completed, so be very careful with the script, and ensure that it’s not causing any lag in a test environment, prior to deploying into live!

    Below is a copy of an early version of the script (most of the mod_security rules that we have are specific to work, so are not being included for various reasons), however I’ll post updates to this soon.


    -- mod_security alike in LUA for mod_magnet
    LOG = true
    DROP = true
    function returnError(e)
            if (lighty.env["request.remote-ip"]) then
                    remoteip = lighty.env["request.remote-ip"]
                    remoteip = "UNKNOWN_IP"
            if (LOG == true) then
                    print ( remoteip .. " blocked due to ".. e .. " --- " ..
                                    lighty.env["request.method"] .. " " .. lighty.request["Host"] .. " " .. lighty.env["request.uri"])
            if (DROP == true) then
                    return 405
    function SQLInjection(content)
            if (string.find(content, "UNION")) then
                    return returnError('UNION in uri')
    function UserAgent(UA)
            UA = UA:gsub("%a", string.lower, 1)
            if (string.find(UA, "libwhisker")) then
                    return returnError('UserAgent - libwhisker')
            elseif (string.find(UA, "paros")) then
                    return returnError('UserAgent - paros')
            elseif (string.find(UA, "wget")) then
                    return returnError('UserAgent - wget')
            elseif (string.find(UA, "libwww")) then
                    return returnError('UserAgent - libwww')
            elseif (string.find(UA, "perl")) then
                    return returnError('UserAgent - perl')
            elseif (string.find(UA, "java")) then
                    return returnError('UserAgent - java')
    -- URI = lighty.env["request.uri"]
    -- POST = lighty.request
    if ( SQLInjection(lighty.env["request.uri"]) == 405) then
           ret = 405
    if ( UserAgent(lighty.request["User-Agent"]) == 405) then
           ret = 405
    return ret

    The following needs to be added to lighttpd.conf to attach this LUA script via mod magnet

    server.modules += ( "mod_magnet" )
    magnet.attract-physical-path-to = ( "/etc/lighttpd/mod_sec.lua")

    *Update – 23 Aug 09* Updated to return code even if one test passes*

    Comments or suggestions are appreciated!