Puppet (http://www.puppetlabs.com/)'s a great tool for system automation - out of the box it comes with most of the functionality that you'll ever need, and a DSL that's easy enough to let junior sysadmins and developers maintain their own recipes. But, depending on the kind of workhorse you want puppet to be, it's possible to hit a wall. For my company, we had too many developers, many system administrators, multiple releases in production simultaneously, and a change control to rule them all (but managed nothing). We also had the odd typo in node definitions causing incidents.
One possible answer to this is to employ an External Node Classifier, which in simple terms means an external script that Puppet calls to get the node definition, or at least the classes and parameters (and the environment, but don't rely on that due to an outstanding bug). An ENC cannot call definitions or puppet functions directly however - it can just pass in classes, class parameters, and global parameters. It's therefore up to you to make your ENC more intelligent.
I think Dan Bode of Puppet Labs gave me the idea of using YAML at first, over a year ago, and I whipped up a basic version in a couple of hours.
Over my long weekend, I decided to write a new ENC library from scratch that would provide the basics that one might want in an ENC, but allows it to be extended with plugins. You instantiate it for a node with some seed data, pass in a loader object, and then include one or more input/override files. It will take care of class, parameter, and substitution-variable overrides, and will spit out a nice Puppet 2.6+ ENC-compatible hash.
The test_puppetenc.rb file is the best source of documentation.
The Code: https://github.com/rmt/puppetenc-ruby
— by Robert Thomson, created 24th Jan, 2012, last modified 24th Jan, 2012 | Tags: Tech
I implemented a HTTP CONNECT-only proxy server in Go, where the endpoint can be a local executable on the proxy server. W/o the executable it's just a standalone HTTP CONNECT proxy server (which can be useful in itself).
It was my first Go program, and I was pleasantly surprised.
I have a desire to do some dynamic redirections based on the requested hostname/port - these redirections would be best served with a shell script.
My exact use-case is that I'd like to be able to connect to VNC ports of KVM VMs in remote datacenters where the host & VNC port may change. I figure it'd be nice to be able to say, "vncviewer appvm01.domain:0" and have the CONNECT to appvm01.domain:5900 lookup the VM->host+vncport relationship for that VM. I may have to use tsocks for VNC clients that don't support a HTTP proxy. If I can find a good free VNC client with proxy support for Windows (know of any?) then I can give Windows-only support staff the same functionality.
Anyway, the code's at http://github.com/rmt/httpconnectproxy
— by Robert Thomson, created 13th Mar, 2011, last modified 13th Mar, 2011 | Tags: Tech
Like many other unfortunates in large companies, I have to deal with networking policies that generally make it a right pain in the arse to do my job. Since we still have a job to do, we naturally work around the limitations (which, since everyone has to do it independently, will inevitably lead to under-the-radar security risks, you networking gits!)
Anyway...I just added another tool to my arsenal. I wrote a simple inetd service that works with Linux 2.6's iptables to find the destination IP and port, then passes control to another program (such as netcat-openbsd or connect-proxy). It will call the other program of your choice with the real destination IP and port appended to the argument list. Simple indeed, but powerful.
With appropriate iptables NAT redirects pointing to my xinetd services, I can provide myself with (apparent) proxy-free access to the Internet and into the various DCs with their screwy non-unified access methods. I don't expect the performance to be great with many connections, but being able to connect to arbitrary ports in the DCs w/o custom configuration will make life easier.
TProxyPass is at https://github.com/rmt/tproxypass
— by Robert Thomson, created 9th Jan, 2011, last modified 9th Jan, 2011 | Tags: Tech
At work, AMQP looks to be the message queueing protocol that will be targeted in the future. One team there implemented some simple clients in C, that would send messages, and also listen for messages and launch a program on receipt, passing in the routing key and pointing to a temporary file containing the message. I didn't get to see their code, but was given RPMs with unnecessary dependencies that I couldn't be bothered to get running on my Ubuntu box, but I thought that low resource shell-scriptable AMQP send and receive programs would be a good idea for the world at large, so over the course of a couple of weekends I extended RabbitMQ-C's examples to have the same functionality. I compile librabbitmq statically into the binary, and the resident memory footprint when running is 500kb.
Let me present rmt's AMQP Tools: http://github.com/rmt/amqptools
For security, you could cryptographically sign the messages, or just rely on AMQP's security. libRabbitMQ-C only supports the plaintext SASL authentication method at the moment, and SSL isn't supported (try stunnel).
— by Robert Thomson, created 24th Oct, 2010, last modified 24th Oct, 2010 | Tags: Tech
It's a rather disorganised trip, and I know I've lost touch with many old friends, so send me an email (rob _at_ corporatism.org) or SMS me on +61434247158 and we'll see if we can meet up at some point. I'll be in Sydney for a little while at the start of the trip, and in Newcastle for a bit, but I'll probably travel up north for a couple of weeks. I'll be accompanied by Sophia.
— by Robert Thomson, created 14th Aug, 2010, last modified 20th Aug, 2010 | Tags: World
A command line interface to XBMC's Navi-X plugin's browser: http://github.com/rmt/navix_cli
XBMC not required.
It's still a little buggy (patches welcome), but I use it on a daily basis already.
— by Robert Thomson, created 24th Jun, 2010, last modified 24th Jun, 2010 | Tags: Tech
The Awesome window manager has me slightly impressed.. who knows whether I will continue to be impressed in a few days, but for now, I am. I found myself somewhat frustrated with the standard GNOME desktop when using multiple monitors in a Xinerama layout (the default if you just plug a screen in). Awesome is a tiling window manager.. in the past, I haven't been inclined to use one, because I work mostly on a single screen, or I have enough screen space for all my current work to be focused. My current work environment, however, involves a bit of task switching, and having a big monitor on my desk as well as the small laptop screen means I want better control of my windows.
The key feature for me is the ability to have independent virtual screens on each monitor, despite it being Xinerama. Most of the time I want the Windows VM showing on the laptop screen, but not always. In my current work environment, having tiling windows is actually proving quite nice, and it's helping me to be a little more organised. I split my tasks between virtual screens, and keep an eye on the Windows VM & outlook on the laptop, so context switching is easy.
Of course, some windows should be floating, and Awesome supports that just fine, as well as allowing each virtual screen to have its own layout style, and easily switch between them.
— by Robert Thomson, created 20th Apr, 2010, last modified 20th Apr, 2010 | 1 comment | Tags: Tech
A little while back I bought an ASRock 330HT as a media PC. Aside from the fan being a little noisier than I hoped, the remote control had no Linux driver so I bought a Hama MCE remote control after reading that it works well under Linux. Unfortunately, while it works fine under Linux, it's got hardcoded key values and not every key is unique. As an example the "i" info button is actually a prolonged right click, a couple of the keys send *exactly* the same keycodes as others (Play and Pause), and there are a bunch of buttons which generate Control-Shift-foo and Alt-bar and so on. I guess under Windows Media Center these all serve a specific purpose. Under Linux and its media center programs (such as XBMC) they don't (and XBMC's input layer is a bit restrictive).
I had my toys, but things weren't working exactly as I wanted. What's a guy to do? I got hacking, of course.
The Linux Input Subsystem is pretty clean nowadays and detecting the correct device and reading a few bytes from /dev/input/eventXX isn't all that hard. Within a couple of hours I was reading the bytes from the remote and pretty printing them as I received them. Unfortunately, this lead to the realisation (as shown above) that the remote's a little screwy and that a few hacks would be necessary to make things work as I wanted. But I was having fun and I was learning something.
The next step was deciding how to use these key events. XBMC has an interface to LIRC and another chap had written a python script making the HAMA MCE remote produce LIRC events. I tried it, fixed some bugs, and it worked.. but it was unsatisfying.. the MCE remote also has a mouse cursor (there are two input devices generated - a mouse and a keyboard - but the mouse device also sends a few key events that I care about as well as the all-important info button).. I started to explore the uinput device which lets you create a fake input device and write to it. Figuring out how to initialise it was a little tricky but I found some similar code and worked it out.. after that it's just writing input events to the device.
My initial idea of just reading all input, munging a few things, and passing the remaining events directly through proved somewhat problematic. If you leave a stranded Control or Alt key pressed down then you have a serious problem and in my tests I had to kill my X session a few times. A better way is to read all the input and have a virtual keyboard that handles the MOD keys, does the processing of normal keystrokes, does the translations, and then converts the translated events back into input events for uinput, ensuring there're no dangling modkeys.
The final step was to create a versatile translated step. I decided that I wanted a configuration file, instead of coding the translations all in python.
Here's some snippets from my config file:
# simple 1-1 translation
KEY_HOMEPAGE-down = send KEY_ESC-down
KEY_HOMEPAGE-up = send KEY_ESC-up
# the info button/right click magic
BTN_RIGHT-down = set rightclick time.time()
BTN_RIGHT-up = if (time.time()-rightclick) > 0.3; send KEY_I; else; send BTN_RIGHT
# compound key sequence example
Alt-KEY_F4-down = wait
Alt-KEY_F4-down KEY_F4-up = send KEY_ESC; clear
The wait action means to add the current key event to the match criteria for the next keystroke. clear then clear's the list.
Code will be online soon at http://github.com/rmt/pyinputevent/
— by Robert Thomson, created 3rd Feb, 2010, last modified 4th Feb, 2010 | 1 comment | Tags: Tech
A couple of months ago I discovered the joys of an Internet Explorer bug revolving around HTTP.. If a HTTP redirect is returned from a server but the Location: header is not in the same TCP packet, Internet Explorer would have some serious issues. This seems to affect MSIE6, MSIE7, & MSIE8. This suggests some poor separation of the network layers in Windows & IE, but it also highlighted the strangeness of SimpleHTTPServer's implementation, which is used by paste's default server.
Every time it sends a header, it flushes the buffer.. the end result is that you often get one TCP packet for each header, and then the data afterwards.. obviously this isn't the most efficient use of the network. Headers and data should be buffered before sending.
The solution was to switch to using FLUP & FastCGI for both the testing and production environments. We're using Nginx, and this combination proved very robust. Wireshark certainly showed much nicer results.
— by Robert Thomson, created 19th Dec, 2009, last modified 19th Dec, 2009 | Tags: Tech
I decided to splurge and purchase this baby: The AsRock ION 330 HT for all my media center needs.
People seem to have got it working fine with Linux, except for the remote control... but I'll look into that.
Now, all I need is a TV (or monitor), and a keyboard+mouse for it. Time to head to Saturn and start evaluating TV sets.
Update: It arrived today, and I didn't have a keyboard nor screen for it... I've now bought a decent Samsung monitor and a cheap microsoft wireless keyboard+mouse. The TV can come in a little while - too big a purchase to buy under pressure.
— by Robert Thomson, created 21st Nov, 2009, last modified 25th Nov, 2009 | Tags: Tech
