Puppet (http://www.puppetlabs.com/)'s a great tool for system automation - out of the box it comes with most of the functionality that you'll ever need, and a DSL that's easy enough to let junior sysadmins and developers maintain their own recipes. But, depending on the kind of workhorse you want puppet to be, it's possible to hit a wall. For my company, we had too many developers, many system administrators, multiple releases in production simultaneously, and a change control to rule them all (but managed nothing). We also had the odd typo in node definitions causing incidents.

One possible answer to this is to employ an External Node Classifier, which in simple terms means an external script that Puppet calls to get the node definition, or at least the classes and parameters (and the environment, but don't rely on that due to an outstanding bug). An ENC cannot call definitions or puppet functions directly however - it can just pass in classes, class parameters, and global parameters. It's therefore up to you to make your ENC more intelligent.

I think Dan Bode of Puppet Labs gave me the idea of using YAML at first, over a year ago, and I whipped up a basic version in a couple of hours.

Over my long weekend, I decided to write a new ENC library from scratch that would provide the basics that one might want in an ENC, but allows it to be extended with plugins. You instantiate it for a node with some seed data, pass in a loader object, and then include one or more input/override files. It will take care of class, parameter, and substitution-variable overrides, and will spit out a nice Puppet 2.6+ ENC-compatible hash.

The test_puppetenc.rb file is the best source of documentation.

The Code: https://github.com/rmt/puppetenc-ruby

— by Robert Thomson, created 24th Jan, 2012, last modified 24th Jan, 2012 | Tags: Tech

I implemented a HTTP CONNECT-only proxy server in Go, where the endpoint can be a local executable on the proxy server. W/o the executable it's just a standalone HTTP CONNECT proxy server (which can be useful in itself).

It was my first Go program, and I was pleasantly surprised.

I have a desire to do some dynamic redirections based on the requested hostname/port - these redirections would be best served with a shell script.

My exact use-case is that I'd like to be able to connect to VNC ports of KVM VMs in remote datacenters where the host & VNC port may change. I figure it'd be nice to be able to say, "vncviewer appvm01.domain:0" and have the CONNECT to appvm01.domain:5900 lookup the VM->host+vncport relationship for that VM. I may have to use tsocks for VNC clients that don't support a HTTP proxy. If I can find a good free VNC client with proxy support for Windows (know of any?) then I can give Windows-only support staff the same functionality.

Anyway, the code's at http://github.com/rmt/httpconnectproxy

— by Robert Thomson, created 13th Mar, 2011, last modified 13th Mar, 2011 | Tags: Tech

Like many other unfortunates in large companies, I have to deal with networking policies that generally make it a right pain in the arse to do my job. Since we still have a job to do, we naturally work around the limitations (which, since everyone has to do it independently, will inevitably lead to under-the-radar security risks, you networking gits!)

Anyway...I just added another tool to my arsenal. I wrote a simple inetd service that works with Linux 2.6's iptables to find the destination IP and port, then passes control to another program (such as netcat-openbsd or connect-proxy). It will call the other program of your choice with the real destination IP and port appended to the argument list. Simple indeed, but powerful.

With appropriate iptables NAT redirects pointing to my xinetd services, I can provide myself with (apparent) proxy-free access to the Internet and into the various DCs with their screwy non-unified access methods. I don't expect the performance to be great with many connections, but being able to connect to arbitrary ports in the DCs w/o custom configuration will make life easier.

TProxyPass is at https://github.com/rmt/tproxypass

— by Robert Thomson, created 9th Jan, 2011, last modified 9th Jan, 2011 | Tags: Tech

At work, AMQP looks to be the message queueing protocol that will be targeted in the future. One team there implemented some simple clients in C, that would send messages, and also listen for messages and launch a program on receipt, passing in the routing key and pointing to a temporary file containing the message. I didn't get to see their code, but was given RPMs with unnecessary dependencies that I couldn't be bothered to get running on my Ubuntu box, but I thought that low resource shell-scriptable AMQP send and receive programs would be a good idea for the world at large, so over the course of a couple of weekends I extended RabbitMQ-C's examples to have the same functionality. I compile librabbitmq statically into the binary, and the resident memory footprint when running is 500kb.

Let me present rmt's AMQP Tools: http://github.com/rmt/amqptools

For security, you could cryptographically sign the messages, or just rely on AMQP's security. libRabbitMQ-C only supports the plaintext SASL authentication method at the moment, and SSL isn't supported (try stunnel).

— by Robert Thomson, created 24th Oct, 2010, last modified 24th Oct, 2010 | Tags: Tech

A command line interface to XBMC's Navi-X plugin's browser: http://github.com/rmt/navix_cli

XBMC not required.

It's still a little buggy (patches welcome), but I use it on a daily basis already.

— by Robert Thomson, created 24th Jun, 2010, last modified 24th Jun, 2010 | Tags: Tech

The Awesome window manager has me slightly impressed.. who knows whether I will continue to be impressed in a few days, but for now, I am. I found myself somewhat frustrated with the standard GNOME desktop when using multiple monitors in a Xinerama layout (the default if you just plug a screen in). Awesome is a tiling window manager.. in the past, I haven't been inclined to use one, because I work mostly on a single screen, or I have enough screen space for all my current work to be focused. My current work environment, however, involves a bit of task switching, and having a big monitor on my desk as well as the small laptop screen means I want better control of my windows.

The key feature for me is the ability to have independent virtual screens on each monitor, despite it being Xinerama. Most of the time I want the Windows VM showing on the laptop screen, but not always. In my current work environment, having tiling windows is actually proving quite nice, and it's helping me to be a little more organised. I split my tasks between virtual screens, and keep an eye on the Windows VM & outlook on the laptop, so context switching is easy.

Of course, some windows should be floating, and Awesome supports that just fine, as well as allowing each virtual screen to have its own layout style, and easily switch between them.

— by Robert Thomson, created 20th Apr, 2010, last modified 20th Apr, 2010 | 1 comment | Tags: Tech

A little while back I bought an ASRock 330HT as a media PC. Aside from the fan being a little noisier than I hoped, the remote control had no Linux driver so I bought a Hama MCE remote control after reading that it works well under Linux. Unfortunately, while it works fine under Linux, it's got hardcoded key values and not every key is unique. As an example the "i" info button is actually a prolonged right click, a couple of the keys send *exactly* the same keycodes as others (Play and Pause), and there are a bunch of buttons which generate Control-Shift-foo and Alt-bar and so on. I guess under Windows Media Center these all serve a specific purpose. Under Linux and its media center programs (such as XBMC) they don't (and XBMC's input layer is a bit restrictive).

I had my toys, but things weren't working exactly as I wanted. What's a guy to do? I got hacking, of course.

The Linux Input Subsystem is pretty clean nowadays and detecting the correct device and reading a few bytes from /dev/input/eventXX isn't all that hard. Within a couple of hours I was reading the bytes from the remote and pretty printing them as I received them. Unfortunately, this lead to the realisation (as shown above) that the remote's a little screwy and that a few hacks would be necessary to make things work as I wanted. But I was having fun and I was learning something.

The next step was deciding how to use these key events. XBMC has an interface to LIRC and another chap had written a python script making the HAMA MCE remote produce LIRC events. I tried it, fixed some bugs, and it worked.. but it was unsatisfying.. the MCE remote also has a mouse cursor (there are two input devices generated - a mouse and a keyboard - but the mouse device also sends a few key events that I care about as well as the all-important info button).. I started to explore the uinput device which lets you create a fake input device and write to it. Figuring out how to initialise it was a little tricky but I found some similar code and worked it out.. after that it's just writing input events to the device.

My initial idea of just reading all input, munging a few things, and passing the remaining events directly through proved somewhat problematic. If you leave a stranded Control or Alt key pressed down then you have a serious problem and in my tests I had to kill my X session a few times. A better way is to read all the input and have a virtual keyboard that handles the MOD keys, does the processing of normal keystrokes, does the translations, and then converts the translated events back into input events for uinput, ensuring there're no dangling modkeys.

The final step was to create a versatile translated step. I decided that I wanted a configuration file, instead of coding the translations all in python.

Here's some snippets from my config file:

# simple 1-1 translation
KEY_HOMEPAGE-down = send KEY_ESC-down
KEY_HOMEPAGE-up = send KEY_ESC-up
# the info button/right click magic
BTN_RIGHT-down = set rightclick time.time()
BTN_RIGHT-up = if (time.time()-rightclick) > 0.3; send KEY_I; else; send BTN_RIGHT
# compound key sequence example
Alt-KEY_F4-down = wait
Alt-KEY_F4-down KEY_F4-up = send KEY_ESC; clear

The wait action means to add the current key event to the match criteria for the next keystroke. clear then clear's the list.

Code will be online soon at http://github.com/rmt/pyinputevent/

— by Robert Thomson, created 3rd Feb, 2010, last modified 4th Feb, 2010 | 1 comment | Tags: Tech

A couple of months ago I discovered the joys of an Internet Explorer bug revolving around HTTP.. If a HTTP redirect is returned from a server but the Location: header is not in the same TCP packet, Internet Explorer would have some serious issues. This seems to affect MSIE6, MSIE7, & MSIE8. This suggests some poor separation of the network layers in Windows & IE, but it also highlighted the strangeness of SimpleHTTPServer's implementation, which is used by paste's default server.

Every time it sends a header, it flushes the buffer.. the end result is that you often get one TCP packet for each header, and then the data afterwards.. obviously this isn't the most efficient use of the network. Headers and data should be buffered before sending.

The solution was to switch to using FLUP & FastCGI for both the testing and production environments. We're using Nginx, and this combination proved very robust. Wireshark certainly showed much nicer results.

— by Robert Thomson, created 19th Dec, 2009, last modified 19th Dec, 2009 | Tags: Tech

I decided to splurge and purchase this baby: The AsRock ION 330 HT for all my media center needs.

People seem to have got it working fine with Linux, except for the remote control... but I'll look into that.

Now, all I need is a TV (or monitor), and a keyboard+mouse for it. Time to head to Saturn and start evaluating TV sets.

Update: It arrived today, and I didn't have a keyboard nor screen for it... I've now bought a decent Samsung monitor and a cheap microsoft wireless keyboard+mouse. The TV can come in a little while - too big a purchase to buy under pressure.

— by Robert Thomson, created 21st Nov, 2009, last modified 25th Nov, 2009 | Tags: Tech

This works, but is there a better way, like only modifying a Widget or a Field?

class TextCheckboxSelectMultiple(widgets.CheckboxSelectMultiple):
    """
    Set checked values based on a comma separated list instead of a python list
    """
    def render(self, name, value, **kwargs):
        if isinstance(value, basestring):
            value = value.split(",")
        return super(TextCheckboxSelectMultiple, self).render(name, value, **kwargs)

class TextMultiField(forms.MultipleChoiceField):
    """
    Work in conjunction with TextCheckboxSelectMultiple to store a
    comma separated list of multiple choice values in a CharField/TextField
    """
    widget = TextCheckboxSelectMultiple
    def clean(self, value):
        val = super(TextMultiField, self).clean(value)
        return ",".join(val)

— by Rob, created 15th Sep, 2009, last modified 15th Sep, 2009 | 2 comments | Tags: Tech

class RedirectModel(grok.Model):
    """
    RedirectModel + RedirectView allow us to do redirects from Traverser's easily
    """
    def __init__(self, request, url):
        request.setTraversalStack(['@@index'])
        self.redirecturl = url

class RedirectView(grok.View):
    grok.context(RedirectModel)
    grok.name('index')
    def update(self):
        self.redirect(self.context.redirecturl)
    def render(self):
        pass

Then from your Traverser's traverse():

return RedirectModel(self.request, "/newURL/")

— by Robert Thomson, created 29th Jul, 2009, last modified 29th Jul, 2009 | Tags: Tech

The Problem: MSSQL database with "LATIN1" (case-sensitive) encoding, SQLAlchemy and PyMSSQL. Fields with non-ascii characters were sometimes being returned double encoded.. The actual case is that SQLAlchemy passes parameters to the DBAPI as however you pass them to SQLAlchemy.. the convert_unicode option doesn't seem to change the encoding of unicode parameters when saving, only the SQL string.. which seems kind of inconsistent. I'm actually not 100% certain of the interactions here, and who is to blame - PyMSSQL or SQLAlchemy - but let's split the difference and blame Microsoft, they're always a good target.

The Solution: A custom type:

class EncodedString(types.TypeDecorator):
    impl = types.String
    def process_bind_param(self, value, dialect):
        if type(value) == unicode:
            return value.encode(dialect.encoding, 'replace')
        return value
    def process_result_value(self, value, dialect):
        if value and type(value) == str:
            return value.decode(dialect.encoding)
        return value

and instead of defining a Column with types.String or types.Unicode, use EncodedString.

It looks like types.Unicode should do exactly this, but for some reason it's not working for me.

— by Robert Thomson, created 16th Jul, 2009, last modified 16th Jul, 2009 | Tags: Tech

It's possible via a slightly non-standard method to have offline web apps on the Maemo platform. Tear is a webkit based browser, and when using a recent webkit (more info here) you have access to the local storage facilities as specified in HTML5. The problem is that full AppCache functionality doesn't exist/work yet..

However, you can host your apps locally on your maemo device, and by setting "document.domain = your.domain" in javascript, you can use the security context of the remote domain when making requests.. When you're offline, you just return information from the local data stores, but when you're online, you can get new information easily. Neat, eh?

Still, I'm looking forward to AppCache support so I don't have to save pages locally.

— by Robert Thomson, created 10th Jul, 2009, last modified 10th Jul, 2009 | 1 comment | Tags: Tech

LinuxTAG was interesting.. I got a better grasp of the Linux and Open Source crowd in Berlin (and in Germany in general). I think my geek-side will be satisfied there. I've decided to check out Sugar-on-a-stick after chatting to them there (and pointed one of the developers in the direction of Numptyphysics, which just may appear as a Sugar package at some point). I'm looking forward to settling down in a proper apartment again, and setting up a media center for myself - A nice ATOM dual core + NVidia (ION) server combination (TEO-X had one there) should provide all the power that I need (with lower power consumption) for XBMC & a retro (& not so retro) gaming setup. :-)

Also interesting was Büro 2.0, which involves a shared workspace and services for Open Source companies and freelancers in Berlin. If I decide to be a freelancer there, this might be an option.

— by Robert Thomson, created 30th Jun, 2009, last modified 7th Jul, 2009 | Tags: Tech, World

Announcing Seantis Questionnaire, a Django Questionnaire/Survey application that I developed for Seantis GmbH in Switzerland.

Features

• Multiple Questionnaires, multiple pages (QuestionSets) per questionnaire, multiple runs of same questionnaire per subject (eg. for annual surveys)
• Multilingual using the django transmeta application.
• Questions (ie. 6) & SubQuestions (ie. 6a), with appropriate layout tweaks.
• Email Invitations (direct URL to Questionnaire)
• Does _not_ require Django’s authentication system to answer questionnaires
• CSV export of answers.
• Complex dependencies utilising a full Boolean Expression Parser, and Javascript support. eg. (6a=yes and 6b=yes) OR 5=no
• Built-in Question types including:
  • Open (single line), Open (text area)
  • Yes/No, Yes/No/Dont Know, Yes/No/Comment
  • Single Choice, Single Choice + Freeform Option
  • Multiple Choice, Multiple Choice + Freeform Option(s)
  • Number Range
  • Time Period
• Easily extensible question types (Question Processor + Template, and Answer Processor)
• Django Signals on completion of QuestionSets and Questionnaires
• Javascript to hide irrelevant questions (works fine without Javascript too) and disable irrelevant form fields.

Talk to Seantis GmbH about custom development.

Download Seantis Questionnaire from GitHub. It is released under the same license as Django itself.

— by Robert Thomson, created 15th Jun, 2009, last modified 18th Jun, 2009 | Tags: Tech

Zope 3's Pluggable Authentication Utility doesn't automatically store a cookie saying that you're logged in if you've been authenticated once. Instead, it uses credentials and authentication plugins. For each credentials plugin, it passes the credential's output to each of the authentication plugins, and if any succeed, then it returns a Principal. It sounds logical enough, except that it does that every time, instead of storing a variable in my session stating that I'm already authenticated. Since I'll be authenticating against an external system, I didn't like the idea of checking the password on every request.. It wasn't immediately obvious to me how I could simply achieve this within the Pluggable Authentication framework, and I wasn't sure I needed the power of it anyway, so I decided to create my own IAuthentication implementation, and forget about the PAU altogether.

What it does

This implementation stores all the IPrincipal information (id, name, and description) as a signed cookie in the user's browser. By default, the session will expire after 6 hours of non-use, and the cookie's timestamp will be updated every 5 minutes with a new timestamp.

The Code

The signed string code is located in securestring.py, and the authentication code in auth.py. You also need to implement the authentication in your login form, and add it as a local utility to your application/site.

securestring.py

# coding: utf-8
import md5, random

SECRET='SomeRandomStringThatYouShouldNotShare' # CHANGE THIS

def make_sstring(question, string, r = None):
if r is None:
r = ''.join([random.choice('1234567890abcdef') for x in '12345678'])
m = md5.md5(SECRET + question + ":" + string + r)
return "%s|%s|%s" % (string, r, m.hexdigest())

def get_sstring(question, securestring):
string, r, md5 = securestring.split('|',2)
if make_sstring(question, string, r) == securestring:
return string
return False

auth.py

# coding: utf-8
from zope.app.security.interfaces import IAuthentication, IUnauthenticatedPrincipal, PrincipalLookupError, IPrincipal, ILogout
from zope import interface, schema, security
from securestring import make_sstring, get_sstring
from zope.app.component import hooks
from zope.traversing.browser.absoluteurl import absoluteURL
import time
from urllib import urlencode

class Principal(object):
    interface.implements(IPrincipal)

    def __init__(self, id, title, description):
        self.id = id
        self.title = title
        self.description = description

    def __str__(self):
        return "<Principal: %s>" % self.title

def make_authenticated(request, principal):
    id = principal.id
    title = (principal.title or '').replace("::","..")
    description = (principal.description or '').replace("::","..")
    tm = int(time.time())
    sstring = "%d::%s::%s::%s" % (tm, id, title, description)
    sstring = make_sstring('z3c_sstring_login', sstring)
    request.response.setCookie('z3c_sstring_login', sstring, path="/")
    return principal

class SStringAuthenticator(object):
    interface.implements(IAuthentication, ILogout)
    loginpagename = 'login'
    timeout_in_seconds = 60*60*6 # 6 hours
    update_timeout = 60*5 # how often to update the cookie

    def logout(self, request):
        request.response.expireCookie('z3c_sstring_login', path="/")

    def authenticate(self, request):
        sstring = request.cookies.get('z3c_sstring_login', None)
        if sstring is None:
            return None
        sstring = get_sstring('z3c_sstring_login', sstring)
        if not sstring:
            return None
        try:
            tm, id, title, description = sstring.split('::',3)
            tm = int(tm)
            now = int(time.time())
            if (now - tm) < self.timeout_in_seconds:
                principal = Principal(id, title, description)
                if (now-tm) > self.update_timeout:
                    make_authenticated(request, principal)
                return principal
        except:
            pass

    def unauthenticatedPrincipal(self):
        # not really sure what to do here, but it doesnt seem to hurt
        return None

    def unauthorized(self, id, request):
        site = hooks.getSite()
        stack = request.getTraversalStack()
        stack.reverse()
        query = request.get('QUERY_STRING')
        camefrom = '/'.join([request.getURL(path_only=True)] + stack)
        if query:
            camefrom = camefrom + '?' + query
        url = '%s/@@%s?%s' % (absoluteURL(site, request),
            self.loginpagename,
            urlencode({'camefrom': camefrom}))
        request.response.redirect(url)

    def getPrincipal(self, id):
        principal = Principal(id, id, id)
        interface.directlyProvides(principal, IUnauthenticatedPrincipal)
        return principal

Install it as a local utility

In your application, you can add it as a local utility. Since I'm using Grok, I'll give a Grok example:

class MyApp(grok.Application)
    grok.local_utility(SStringAuthenticator, provides=IAuthentication)

You could of course also provide a setup function, to modify the string.

Local Utilities will only appear on NEW objects, so your existing applications/sites won't make use of it.

Authenticate from your login form

I wasn't sure the best way to do the actual authentication here, so I thought I'd just leave it up to the login form. If successful, the login form should just call auth.make_authenticated(request, principal) where Principal is an instance of an IPrincipal (you can use auth.Principal if you like, but there's no doubt a better way).

The Alternative using Pluggable Auth

I realised afterwards that it would be possible to do the same thing within the pluggable auth framework. You could do it almost exactly the same way, except that the credentials plugin should just return the cookie if it's set (credentials just returns a dict, so you're not limited to just a login/password), and the authentication plugin can just check if it's valid. The credentials plugin can still redirect unauthorized users to a login page. http://grok.zope.org/documentation/how-to/authentication-with-grok should be able to give you an idea about how to implement Pluggable Authentication.

— by Robert Thomson, created 9th Jun, 2009, last modified 18th Jun, 2009 | Tags: Tech

My Xen server is now IPv6 enabled, and I have 4bn or so routable IPs. Yay!
And with a little bit of playing, I have IPv6 over the openvpn connection to my laptop, so now I have my very own public IPv6 network at home. And of course, yay for ip6tables!

— by Robert Thomson, created 22nd May, 2009, last modified 22nd May, 2009 | Tags: Tech

I got busy with work just after the N810 arrived, but nevertheless I’ve enjoyed having it and I find it quite a cool gadget. It’s screen is readable in all but direct sunlight, which helps to make it a great device when traveling. I found myself using it mostly as an E-Book reader when on the road, and loving Project Gutenberg and the Baen Library. I still have quite a few application ideas for it – largely revolving around making online content easily searchable and then available offline (eg. train timetable searches, hostel/hotel contact information & reviews).

As a side note, I suspect that the next device I buy will come with a PixelQi screen for it’s easy readability, low power, and versatility, but it will still be a few months until the first commercial device using it makes it to market.

I’ve developed a reasonably complex Django application for questionnaires/surveys for a medical study, which I hope to release publically in the near future, after the internal project is released. We’re still working on a name, as django-questionnaire and django-survey are both taken.

— by Robert Thomson, created 11th May, 2009, last modified 11th May, 2009 | Tags: Tech, World

I received my N810 about 3 weeks ago, and despite the fact it says "N00" instead of "N810" on the front (N00 is the normal number for prototype models from Nokia, but its internal labeling and hardware specs are that of a N810), I am quite taken with it. Surprisingly, I use it mostly as an Ebook reader (it turns out that I really enjoy reading, I just hate carrying books around). As far as Internet goes, I mostly use the weather widget, and receive email updates on it too. Maemo Mapper is quite good to have around also, when one gets lost in a city.. but the GPS doesn't work that well. One of my primary reasons for geting the device itself was that I wanted something that I could hack on and program for easily.. it certainly fits the bill. Some of the standard software release's libraries are a little old, but they're easy enough to update. Python works fine, but I'm partial to using Vala for efficiency. All I need now is a good phone and data plan to go along with it.. but I think that will have to wait until I'm settled in Germany.

On the work front, I've found some programming work with a small Swiss company. I'm currently in Luzern, enjoying the weather and the lovely water views. I miss having the water nearby.. I'm doing some Django and Zope work, and will continue working from home from next week. Sophia's coming up on Thursday and we'll meet some friends in Switzerland and see some of the sights.

— by Robert Thomson, created 7th Apr, 2009, last modified 7th Apr, 2009 | Tags: Tech

Yesterday I placed an order for the N810 Internet tablet. They've been talking about a successor to the N810 for months now (RX-51 / N900), but frankly, the N810 will probably suffice for years whether or not a new (and more expensive) model appears in the next few months.

— by Robert Thomson, created 13th Mar, 2009, last modified 13th Mar, 2009 | Tags: Tech

I’ve been playing around with Parrot again lately, since it’s approaching a 1.0 release. I’m going through the tutorials for the Parrot Compiler Toolkit, and I’m quite impressed. Sure, I keep shooting myself in the foot since I don’t understand how everything interacts yet, but it does work. I like it because it makes it easy to figure out what’s really going on and how the conversion processes work, and it seems to be reasonably well documented.

One of my longer term goals is to have a suitable nice language for MUD world programming (which assumes that the programmers can’t be trusted), and I think I’d like to modify an existing language to fulfill the requirements. Of course, having almost no idea about compiler design seriously limits how much I can tweak without wreaking havoc.

— by Robert Thomson, created 10th Mar, 2009, last modified 10th Mar, 2009 | Tags: Tech

Now that I’m mostly relaxing in Italy instead of earning contract rates in Holland, I force myself to think long and hard before I spend money on gadgets. This means prioritising, and balancing my wants and needs now and in the future against my budget. But the urge for a new toy is growing, and I’ve been doing a bit of research into what’s available and what would be the most practical for me. And also into what’s arriving in the future..

Yesterday I made a spreadsheet trying to figure out what device I wanted most, assigning points for the features that I consider important, and how each gadget rates. I included the Nokia E71 (by all accounts a high quality featureful phone that does most things that I want out of the box), the Nokia N810 Internet Tablet, and the HTC Magic (G2) Android phone. Ease of development under Linux tipped the ratings in favour of the N810 and the Android phone for me. The Android phone has the disadvantage that I dislike programming in Java, and the alternative languages for Android/Dalvik are still largely in the works. At the end, the N810 came out slightly above the HTC Magic, although if the HTC Magic were available today at a reasonable price (and actual features match expected features), the balance would be tipped.

But that then leads to the next question.. what could be better in these devices, and what’s expected in the near future? The N810 has been around for over a year, and a successor is expected shortly (The RX 51, also commonly referred to as the N900) although Nokia hasn’t been forthcoming with the specs. It is expected that it would have an acceleratormeter and probably a digital compass, and it may have some cellular capabilities (although I doubt it will be a replacement for a mobile anytime soon).

I also want to be able to view some e-books and PDFs on any device I have, and this also means that it has to be readable in sunlight. The N810 has a sunlight readable display and a good size & resolution, as well as the ability to view all sorts of documents, which makes it a good pick. The E71 with its small screen is not exactly ideal, and the G2 also falls a bit short.

The technology that I’m waiting for is:
* Pixel Qi’s low power sunlight readable displays (should be excellent for E-books, but also good for normal tablet style uses). We can expect to see the first commercial use of these screens in the second half of 2009.
* Nokia’s RX-51/N900 Internet tablet.. or at least some more information about it, so I can decide whether to wait or buy now.
* HTC Magic (Android G2), and other Android phones. The more hackable the better.
* The Da Vinci Machine (Multi-Language JVM), because Java sucks and anything to bring better programming languages to the JVM is welcome.
* The latest Scala release for Android, as well as Scala for the CLR.

— by Robert Thomson, created 26th Feb, 2009, last modified 26th Feb, 2009 | Tags: Tech

From the website: “Scala is a general purpose programming language designed to express common programming patterns in a concise, elegant, and type-safe way. It smoothly integrates features of object-oriented and functional languages. It is also fully interoperable with Java.”

I quite like it.. I’ve never been a fan of Java, but I have been of good VMs and their well tested highly tuned libraries. Java and the CLR being the two common ones these days. I did briefly look at the website a few months ago, but disregarded it because the funny syntax didn’t appeal to me at the time.. I shouldn’t have. I recommend the book “Programming in Scala” (available as an PDF download from the publisher Artima or from a bookshop near you) in preference to the online tutorials and documentation that exist today.

Of course, given my tendency to get bored easily, I could be learning another language next week. :-)

— by Robert Thomson, created 17th Jan, 2009, last modified 19th Jun, 2009 | Tags: Tech

It was a bit of a struggle to get everything working... but here's how I did it.

1. Install VMWare 2.0.0 under Ubuntu Linux (due to kernel changes, the older version of vmware could no longer compile the kernel modules (asm/semaphore.h changed to linux/semaphore.h and the kill_proc function was removed from the kernel). 1. Browse to vmware's localhost web page and install the Firefox/Mozilla plugin for it, restart firefox. 1. Upgrade VMWare hardware version of my pre-existing Windows vmware image, add USB hardware if not there, let Windows hardware detect and install drivers as necessary, reboot if needed. 1. `mount -t usbfs none /proc/bus/usb -o devgid=500,busgid=500,busmode=664,devmode=664` (I set the devgid and busgid to my uid, which was the admin account in vmware. Maybe you use root and don't need to do something, or maybe it was just strange for me.. no idea.) 1. Browse to http://europe.nokia.com/A4579163 Nokia's website and install the Nokia Software Updater. Reboot windows. 1. Plug in the Nokia phone, select Nokia mode (as opposed to storage mode or another mode).. mount the device in the VMware image. Any read/write errors here are probably permission related in regard to /proc/bus/usb/. 1. Shutdown Windows, find the *vmware.log* file and do `"grep path:" vmware.log`.. find the entry corresponding to the Nokia phone. Copy & Paste the Numbers/Numbers after the path: bit. (maybe it's 1/2/3 for you or just 1/2) 1. Edit the .vmx file in the same directory, add `usb.autoConnect.device0 = "path:THOSE/NUMBERS"`. I also added `usb.generic.skipsetconfig = "true"` but I'm not sure that's necessary. You are actually binding that particular USB port on the host to that VMware image. You may want to undo this afterwards. 1. Start up the Windows guest again. The phone should now connect automatically. 1. Run the Nokia Software Updater.. cross your fingers.. if it doesn't work, uninstall and reinstall the software. Apparently the USB device drivers get screwed up often enough and this fixes them - at least, it did for me. 1. The Nokia Software Updater SHOULD now detect your phone.. if it doesn't, it's probably the device driver.. uninstall and reinstall Nokia Software Updater. It should find any new firmware for your phone and commence installing it.

The reason why you need to use usb autoconnect with the path option is because (it would seem) that the phone reboots into a firmware update mode and changes its identity.. and if it doesn't respond fast enough, then the Nokia Software Updater aborts (causing you to wonder whether you just turned your phone into a brick - luckily it wasn't the case for me).

Good luck!

(For the record, I ended up embarking on this journey when the Nokia Software Updater just stalled at the start screen on my girlfriend's laptop - a common problem, google would suggest.. Please Nokia, give us a Linux firmware updater!)

— by Robert Thomson, created 12th Dec, 2008, last modified 12th Dec, 2008 | 1 comment | Tags: Tech

I held a short memorial service for my SMS archive. At least I'd backed up my phonebook before doing it. Seriously, it looks innocuous if you read the gnokii manpage.

Update: Somebody on IRC said that it should be harmless. For some reason it wasn't on my phone.. but after the "reset" and the subsequent firmware update, everything works.

— by Robert Thomson, created 11th Dec, 2008, last modified 10th Mar, 2009 | Tags: Tech

http://whygitisbetterthanx.com/ WhyGitIsBetterThanX.com prompted me to take more than a superficial look at git (Version Control System), and I'm very impressed. I think I'm a convert, forever destined to look down on other VCS' with a slight air of disgust.

• Update:* Commenter *KK* wanted to point out http://git.or.cz/gitwiki/GitSvnComparsion for a less koolaidy comparison.

— by Robert Thomson, created 7th Dec, 2008, last modified 8th Dec, 2008 | 2 comments | Tags: Tech

We have achieved that which we thought impossible. We now have Internet at our apartment in Perugia! It is a miracle! After over a month of wrangling with technical difficulties (which they never mentioned when we called to find out about our order - instead claiming that we were not customers, and trying to sell us something, thus leading us to assume our order was dropped, and to us placing a second order).. And so it is, that we now have Internet in our apartment. We probably could have had it a couple of weeks ago, had the modem (or instructions, since we had a spare modem) arrived sooner. But now we are officially customers, and we shall cherish our new-found status.

— by Robert Thomson, created 9th Oct, 2008, last modified 9th Oct, 2008 | Tags: Tech

.. if I'm ever that desperate. But seriously, since I'm going to Italy for 10 months, and quitting my job to do it, I will probably look for some projects to keep me busy, and with the current state of the share market, I'd feel better earning money instead of digging into my savings. :-) So if anybody has any 100% remote part-time work for me (python, perl, linux, whatever) at realistic rates, starting November, let me know. The Django remote work job-board looks like an interesting option, if you weed out the adverts wanting something-for-nothing.

— by Robert Thomson, created 2nd Oct, 2008, last modified 19th Jun, 2009 | Tags: Tech

Does it irk other people as much as me that we are expected to read & write XML in order to build and run programs? What ever happened to real configuration files? How about editing tools for XML? Did half the world forget that XML is intended for machines, not humans? I always get a bit annoyed when I see end user documentation telling humans to edit XML. Sure, sometimes the tools come later, but quite often not, or they eat up 200 MB of RAM and require a GUI to edit a 30 line configuration file. EOR.

— by Robert Thomson, created 22nd Jul, 2008, last modified 5th Aug, 2008 | 1 comment | Tags: Tech

The iRiver Clix2 that I ordered finally arrived. It was supposed to be here after 2–3 weeks, but instead it took 8 weeks… in the meantime, I was given a free 1GB iShuffle by T-Systems (my new contracting client following Shell’s outsourcing), which I also quite like due to its small size and the clip, making it perfect for when I’m on my bike.

I’m happy with the Clix2. Right now, I’m using it to listen to the radio (and switching stations whenever they start talking their horrible language). It came configured in USB mass storage device mode, instead of the alternative MTP mode.. I’ve installed some new themes and some flash lite games on it.. I’ve transcoded some TV shows for it (320×240 OLED screen) .. wouldn’t want to watch a full length movie on it though.

In other news, Sophia and I are going to Perugia, Italy, later this month, to see if we can find some accommodation for her Erasmus year and my extended-holiday, and generally check out the place. We’re eagerly counting down the days until we’re both in Perugia!

— by Robert Thomson, created 13th Jul, 2008, last modified 13th Jul, 2008 | Tags: Tech

Defective C++

Scripting languages, managed runtimes and similar (CLR, Parrot and D) make me smile.

— by Robert Thomson, created 24th Apr, 2008, last modified 18th Jun, 2009 | Tags: Tech

I’ve an NVidia card with driver version 169. Sometimes when playing movies with Totem, the colours would be off. It took me a while to figure it out, thinking it was maybe a bug I couldn’t do anything about except for downgrading the NVidia driver (which seemed like too much work with Fedora).. Not sure exactly where the bug lies, but the hue setting for XV changes sometimes, resulting in odd colours. I think gstreamer may reset it sometimes. The quick fix is to install xvattr and run the following:

xvattr -a XV_HUE -v 0

— by Robert Thomson, created 2nd Feb, 2008, last modified 2nd Feb, 2008 | Tags: Tech

I wiped Vista with a fresh copy of Fedora 8. I'm normally a Debian & Ubuntu person, but I decided to give Fedora a go for a change. It installed without any major problems. It would've been nice if it had started NetworkManager for me, so I could easily configure my wireless network.

Yesterday I decided to install a copy of Windows XP, which I had lying around. Once again, I'm not a Windows person and booted into Windows maybe twice a year on my old laptop, but since I have a license, I may as well put it on. There were issues with the boot CD and it not going any further. Recommended BIOS tweaks for the HDD to be detected did nothing either. So I am now trying an alternative method. I installed Windows XP under VMWare, installed all the drivers for the laptop that I could, and will try to copy the vmware image to the real partition, resize using ntfsresize, modify the boot sector and the boot.ini, and then see what happens. I will of course document my steps if I'm successful. Sounds neat, no? :-)

— by RobertThomson, created 26th Nov, 2007, last modified 26th Nov, 2007 | Tags: Tech

I just ordered myself a new toy – a [[http://www.laptopmag.com/Review/Dell-Inspiron-1520.htm Dell Inspiron 1520]]. The specs are:

• 15” LCD (1440×900)
• Intel Core 2 Duo T7500 processor (2.2Ghz, 4MB L2-cache)
• 2GB RAM (667Mhz)
• nVidia 8600M GT with 256MB
• 250GB hdd
• Intel 3945 802.11a/b/g card
• 9 cell battery (adds a little bulk but I’ll appreciate it when traveling)
• Black matte finish with integrated webcam
• And best of all¹: Windows Vista!!!

My current laptop (Compal PowerNote CL50) has served me well for 4 years now, and I hope that this new laptop will serve me just as well. Choosing a new laptop is a tough decision and unfortunately having to make compromises is par for the course. I think I’ve made a good choice… not too small, not too big, and powerful enough to last me for a few years. It’s hard to find similarly spec’d laptops here with the same resolution.

For a case I ordered the [[http://reviews.cnet.com/notebook-accessories/kensington-contour-cargo-messenger/4505–6459_7–31898142.html Kensington Contour Cargo Notebook Messenger]]

For your information, the other main contender was: the Acer Aspire 5920G ([[http://asia.cnet.com/reviews/notebooks/0,39050489,40634980p-2,00.htm mixed review]] – a few differences in the EU market though)

So.. I bet you’re all wondering what I plan to do with a spankin’ new laptop, and what I couldn’t do on my current beloved laptop. Here’s what:

• Virtualisation – My current laptop’s not up to the task, and being an IT guy, having suitable hardware to play with in my free time is vital to keeping my skills up.
• Similarly, having a second laptop means that I can experiment with one while still maintaining my primary working system. It can also serve as a nice quiet server.
• Games – I hardly play them, but every couple of months I get the urge and it would be nice to have a laptop capable of playing most modern games.

¹ That’s a joke, folks. I’ll be installing Linux ASAP. :-)

### Update – 9 Nov 2007

Still in preproduction. :-( Why, oh why, must I wait so long? Newsflash Dell – it’s not Just In Time if you have to wait days or weeks for parts. I’m a child of the Internet age – I want instant satisfaction. Hop Hop.

— by RobertThomson, created 3rd Nov, 2007, last modified 18th Jun, 2009 | Tags: Tech

I’m helping to revamp the build infrastructure here. Until now, we’ve been using a bunch of home-rolled scripts and [[http://autobuild.org/ Test::AutoBuild]].

We decided to look at Fedora’s build system and processes, since we’re developing RHEL and using RHN Satellite, didn’t want to re-invent the wheel, and Red Hat would similarly be trying to keep closely aligned with Fedora’s processes also. Yum repository support is official in RHEL5 and supported by RHN Satellite 5.

Mock was relatively simple. Create a Yum repository (extract latest version information from Satellite and copy the RPMS and comps.xml to a directory, then run createrepo), create a group and the directory structures, the configuration files for our build targets (x86_64 and i386) and it’s a go.

Koji’s a different story. Koji doesn’t support straight Username/Password logins – it uses SSL certificates and/or Kerberos to authenticate. I tried to follow the instructions for the [[http://fedoraproject.org/wiki/Koji/ServerHowTo Koji Server HowTo]] but my setup is a bit different in that I’m installing everything on one machine. For me it meant that I didn’t need (and couldn’t use) a separate cert for kojiweb and kojihub..

For the ClientCert’s, you need a combined PEM file containing both the cert and the private key. If you don’t have that, you will see a python error with PEM something-or-other in it. This applies for ~/.fedora.cert (whatever path’s mentioned in $HOME/.koji/config for the cert), as well as for the WebCert in /etc/httpd/conf.d/kojiweb.conf, that used by kojid (cert= in /etc/kojid/kojid.conf), and of course that used by kojira (as explained in the [[http://fedoraproject.org/wiki/Koji/ServerHowTo Koji Server HowTo]].)

The “Python ProxyDNs” option should correspond to the DN of the web host (or maybe hub, but for me it’s the same) .. if you get this wrong, don’t worry, you will (when the rest is correctly setup) receive an error from the web interface with the proper string.

Now that I can login, I just have to learn how to use Koji. :-)

Update: Koji turned out to be too much of a pain. It really is a complete distribution management solution, and we already use RHN Satellite for that.. so I built my own distributed build system based around Mock. It works quite well, although it lacks a lot of the pizazz of Koji.

— by Robert Thomson, created 25th Oct, 2007, last modified 18th Jun, 2009 | Tags: Tech

I signed up with the VSP (Virtual Service Provider) VoipBuster. It's actually one of the many VOIP websites run by BetaMax, a German company. After paying €11.90 (€10 + 19% VAT) I can now make calls freely or cheaply to many countries for 120 days or until I run out of credit. From home, where I have a VOIP adapter, I can call Australia or Dutch numbers gratis. With my mobile, I can call a landline number in the Netherlands (relatively cheap, plus it's a work mobile) and then use the VoipBuster service to dial to Australia or Germany for 1¢/minute (OK, 1.19¢ with VAT). German mobiles cost just 20¢/minute, thereby undercutting the new EU roaming tariffs, if you ignore the local phonecall required from my mobile (as I do).

VoipBuster and the similar websites do occasionally switch which countries are free to call to, and rates do vary between the websites (smsdiscount.com has 1¢ SMS', whereas VoiceBuster has 5¢), but overall it is (they are?) still one of the cheaper VSPs that exist. The multiple websites and the requirement to register using the Windows client turned me off at first (and the client requirement still does) but I can accept that the multiple websites are largely just a marketing game to generate buzz, differentiate themselves from the competition, and get around the one-size-fits-all model to exploit niches.

— by Robert Thomson, created 24th May, 2007, last modified 18th Jun, 2009 | Tags: Tech

I’ve just purchased a Linksys SPA 3102 SIP gateway.. now I’ll just have to wait for delivery.

It will allow me to use my standard home phone as a land-line, and also route calls to/from PSTN & SIP through it.

I shall configure it to use my voxalot account (sip:111555@voxalot.com) but also register my landline number on E.164.

Not certain which SIP provider(s) I’ll go with yet.. I just know that I really didn’t want to give KPN (the local telco monopoly here) any more than the required €10/month copper-tax here.

Update: The SPA 3102 works quite well, but the configuration was a bit confusing for a first time telephony user – the supplied documentation is virtually non-existent. I’ve also configured it as a PSTN-to-SIP gateway, so that I can call home then make an international call through my VOIP accounts (and land-line calls are cheap). The quality is quite good. Now the only task now is to encourage enough other people to switch to VOIP also, and therefore remove all middlemen. I suspect, unfortunately, that it’s not as easy as one would hope to configure a VOIP adapter or phone, so getting sweet old Aunt Betty online may require a personal visit. Now, all I need is a cheap SMS gateway, since most of the time I’m near a computer and I could save considerable amounts and not be subject to the whims of bastard telcos

— by RobertThomson, created 27th Apr, 2007, last modified 18th Jun, 2009 | Tags: Tech

I'm told that the paperwork will be sorted out by the end of today, but I've given the recruiter permission to accept on my behalf based on our discussed rates and start-dates. Now - to find accommodation. I may be one of the better paid homeless in Den Haag.

I'm actually going to stay with Thom May for a couple/few days when I first arrive.. but I don't want to impose so I'll do some frantic apartment hunting. I'll also use this time to get my SOFI (tax file) number, a bank account, and a dutch phone number.

It's quite sad to be leaving Passau, but Passau has no IT industry to speak of, and this is very much the kind of opportunity I was looking for last time I was looking for work in Europe.. but at that time I wasn't prepared to move too far from Passau. I'll probably keep my apartment here for now - I may sublease it, or just give up the lease.. but I've got a reasonable amount of stuff to move, so I'll have to come back and sort through things soon enough.

Update (9 Jan 2007)

I just heard, I start on Monday, and I get the rate I hoped for. I'm dusting off my suitcase as we speak. Farewell dinner/drinks on Wednesday.. probably train on Thursday.

Update (11 Jan 2007)

Leaving tomorrow morning at 11:22am, arriving around 8:30pm.. changing trains once in Frankfurt, and once in Utrecht in the Netherlands. I'm sure I'll be up for a good meal and a beer after that journey.

— by Robert Thomson, created 8th Jan, 2007, last modified 18th Jun, 2009 | Tags: Passau, Tech, World

A few weeks ago I thought I'd jump into the 21st century and check out a web framework or two. I looked at RoR first, and after a little mind-bending I understood it reasonably well. It really requires you to be one with RoR to grok it fully. Shortly after that, however, I noticed the Netherlands contract advertised, which was of course for a Python developer. Since things were going well there, I decided to focus once again on Python.

There are a few interesting web frameworks for Python.. but my eyes fell on Django and I gave it a go. I must say, I'm impressed. It's far less hyped than Rails, and it's really quite simple. I wrote this entire website with under 350 lines of Python code, plus HTML templates. The administrative interface is quite neat, although I wish it were a little more customisable and a little less tied to the database structure. The basics required only 200 lines of code -- comment spam checking, email integration, and custom captcha code took up the rest. It took quite a while to get here, though, as I was learning on the way.

I'm using SQLite, because I really don't need or want anything more. In fact, I still think that a database is overkill for a blog/wiki. I am, however, mildly impressed with the result. I'm also using Markdown (modified) for a more wiki-style of editing.

— by Robert Thomson, created 3rd Jan, 2007, last modified 18th Jun, 2009 | Tags: Tech