Playing with Meteor Again

It’s been awhile. I’m looking at Meteor again after my first look about 2 or 3 years ago. It seems to have matured into a rather cool web framework. It’s USP is real time updates between server and clients. Anything entered into the server database, is reflected (almost) immediately on each of the clients. If a client makes changes, those are reflected immediately on the local, and are background updated to the server, and then out to the other clients. It’s really slick.

It uses (by default) a mongodb as backend, and minimongos on each client, which live in the browser memory and have a subset of the full database cached locally. These listen for updates from the server and push updates to the server form teh minimongo. The client only reads and acts on the minimongo.

I decided to pay proper attention this time instead of skimming, so I’ve bought the Discover Meteor Book by Tom Coleman and Sacha Greif.

I’ve also done some digging into the architecture of Meteor because I have always wondered about how much control and scalability I might be sacrificing for the “magic”. This page from The Meteor Cookbook helped a lot (thanks). I now understand that I can remove the mongodb from the meteor server itself and deploy and admin as I wish in production. I am still a little hazy on scaling the meteor app itself, but that is coming in later chapters!

All in all, I’m enjoying myself. The long winter evenings will just fly.

Installing and Running Django Tutorial on Anaconda Python

So I want to do some more fluffy web based stuff with python, and, ultimately, mongodb. I’m doing things properly and starting with the Django tutorial. I’ve already installed Anaconda to provide me with a  full python 2.7 and the key data analysis packages pre-installed. The Anaconda windows installer works perfectly.

An aside, but the ipython notebook is brilliant. Runs a local webserver on your machine with the python command line running. You can record everything you do, download copies and send to others, inline graph generation, etc. It’s lovely.

Fire up a DOS command line.

So I started by checking to see if Django was already installed:

python -c "import django; print(django.get_version())"

It wasn’t. easy_install and pip often don’t work directly from the command line, with either python 2.7 installed manually  or with anaconda, so I went for the safe option that seems to work everytime and:

where python
cd C:\Users\me\AppData\Local\Continuum\Anaconda\Scripts
pip install Django

That installed Django for me fine. I checked again to see what version was installed:

python -c "import django; print(django.get_version())"

So now I need the Django Tutorial for this version. Note that if your version of Django/python is different to mine, you will need to select the right Documentation version. You can do that with the control atthe bottom RHS of the tutorial page.

Next, it wants me to create the project. I have to cd to my source folder and run a command. That command blows up:

C:\me\projects\reporting\incubator> startproject mysite
Traceback (most recent call last):
File "C:\Users\me\AppData\Local\Continuum\Anaconda\Scripts\", line 2, in <module>
from django.core import management
ImportError: No module named django.core

Sad Face. But let’s work around those pesky path issues like we did with pip:

C:\me\projects\reporting\incubator>python C:\Users\me\AppData\Local\Continuum\Anaconda\Scripts\ startproject mysite

Success. Happy Face. Mental note made. Continuing with the tutorial.

I didn’t have the sqlite3 shell installed, so downloaded it from


Finding a WinSxs Solution For Windows Server 2008

On Windows Server 2008 R2 there is no real way to manage the growth of the folder C:\windows\winsxs. This is an issue if you have virtual machines with small system partitions. Hell, it’s an issue because install files from 3 years ago are in a security protected folder that you can’t get rid of without major surgery.

There are lots of articles online from MCEs and MVPs etc saying you do DISM /this /that or COMPCLN.exe. These are all missing the point. They cleanup the stuff prior to the last Service Pack. What about the ~7 gigabytes, and growing, of OS partition space taken by this flaming folder that has no use to it all?

This technet discussion is typical.

However, I did find an article on the microsoft blog about introducing some cleanup to winsxs on Windows 7 that is non-typical, and provides a little clarity. The author, Charity Shelbourne, engages with the commenters specifically about Windows 2008. Hurrah! A comment from GerVoeten on 30th Sept, 2013 is most accurate:

The answer is the only way to clean the winsxs folder up is to uninstall updates and software. There is no guarantee that this will return back to its original size as the files maybe hard linked somewhere.
For 99.9% of us uninstalling updates and software is not really an option, these articles on cleaning up odd files in the os and using disk cleanup is a joke.

In summary, after digging around for 2 days on and off, I think there should be a canonical answer:  if you have Server 2008 R2 and you want to safely cleanup some old install files, wash your hands, pull on your rubber gloves, <small voice of reason>make some backups</small voice of reason>, you’re going in.

Please note this is specific to Windows Server 2008 R2 and doesn’t include Windows 7, Windows 2012.

Basic Motorcycle Kit

I recently got asked by a friend what he’d need kit-wise to start motorcycling. I emailed my thoughts and then realised this might be useful another time. So here it is, slightly tidied up and hopefully not too patronising (if it is, pat-on-the-head). If you’re already a motorcyclist, yes, I know: comfort and safety levels are subjective and differ for every person.

First thing you need is a hat. This is key, it’s the one thing you wear whatever the weather and it needs to fit right and not frustrate you. I also suggest you want your first one to be cheap enough so that you can ditch it with minimal agonising over how much it cost if it turns out you made a terrible mistake.
So two recommendations:
1. don’t spend over 150 quid on it (that’s not a target to aim for!);
2. go and try lots on. 
My opinions, which come down to personal preference: 
  • AGV and Shark are very good brands that have some good budget offerings; 
  • a seat belt style chin strap is minimal fuss on and off;
  • an integrated sun visor is a life saver, esp on autumn evenings when sun is low in the sky;
  • I like flip face helmets as it reduces on and offs and they are the ones that usually have integrated sun visors. On the flip side (ahem), they tend to be noisier;
  • If you go top job (300 quid plus) on an Arai or a Shoei, they tend to be Marmite. You’ll either love the Arais and hate the Shoeis, or vice versa. Tends to be on shape of forehead. I always get on with Arai and AGVs, hate Shoeis and Sharks.
  • A good place to research them is . They are a pretty good shop too. I’ve bought some kit from them in the past.


Good boots will keep you dry and warm and will stop your feet and ankles getting crushed in an accident or when you drop the bike. You can spend an outrageous amount on boots. My most expensive boots (so far) were £180 and were not waterproof or warm, they were racing boots. The ones I wear now were fitted dual purpose bike and walking boots (Hoggs I think) at 150 quid from Altberg. Worth every penny! However, I recommend you start with off-the-peg and work out what you like. Try lots on. Don’t spend over 100 quid on your first pair. Make sure they’re waterproof but breathable if possible.
Jackets and Trousers
  • traditional: leathers. You can get one piece or two piece. One piece are good in summer and if you’re doing a lot of track days. Chances are, you’re thinking about touring and commuting. I’d steer clear of leathers until you know what you want. However, leather is pretty much ultimate sliding and abrasion resistance, although textiles are pretty excellent these days.
  • touring solutions: armoured textile. You can get a jacket and trousers as a set or get separate. Trousers should always be able to zip to jackets – so make sure you test any that don’t come as a set. I went set and got waterproof, detachable thermal lining. Make sure they have CE approved armour with them, including, preferably, a back protector. Textiles are so good these days that they perform almost as well as leathers in abrasion tests, with the added bonus that you’re more likely to have them done up and on in most weathers, unlike leathers :-). My 5 year old Hein Gericke Sheltex suit is great. I’ve talked to a couple of guys at work who have more recent versions of HG jacket and they rate them too. Depending on how much you’re spending, suggest you look at ones that hi-viz built in. Adding a hi-viz vest later is just a faff, so you stop doing it. To get an idea have a look at 
  • sports jacket – good for summer. You’ll freeze and get soaked if it rains. But they are nice to just chuck on and go. I got one with an R1 in 1998 and still wear it occasionally. 
  • armoured jeans – I think they’re expensive for what they are, but they do look good. I have no experience with them. 
Armoured leather ones for summer (and learning to ride), and (prob) textile thermal armoured ones for rain and winter. Learning to ride you need to be able feel really well through the glove, so a waterproof thermal glove that feels as though you have five copies of the Telegraph wrapped around each finger are not what you want. A decent-ish, 50 quid plus race glove is my recommendation. Don’t go mental on Handroids and what not. And don’t get white ones if you can’t stand the sight of road muck on your previously pristine purchase. 
You will get cold hands in winter. I wear a thin set of normal thinsulate stretchy gloves underneath my winter gloves to keep warm. Other solutions include: heated handlebar grips; electrically heated glove liners; and bar mitts (which are permanently attached to bars and you slide your hands in).
Get some earplugs. It’s about 10 quid for hundreds of them. Boring, but will save your hearing. Future you will thank you.
Switch off your Phone
Seriously, every time you get on the bike, turn it off. You’re learning. You don’t need some dodgey pyramid selling SMS to vibrate your moobs as you negotiate weaving to the front of the traffic light queue for the first time or Billie Jean starting playing as you lay the bike into that 50 mph bend just right. What are you going to do anyway? You’ll have to stop, get your gloves off (at least), and dig inside your jacket or rucksack to deal with it. 
That should get you started. there’s loads of other stuff you can get (back protectors, airbags, armoured airtex vests, sat-navs, comms gear, etc), but getting this basic set will protect you and keep you comfortable so that you concentrate on the bike and riding, rather than how cold/hot/tight/loose/wet you are or how to turn on or off the gadgets.
At today’s prices, 2013, I reckon that’ll cost you between £650 and <insert huge figure here/>. 

Unzipping Huge Files from the Command Line

Lucky me. I have 7GB zip which has about 8K files in it, across 160 odd folders. Unzipped it’s about 32GB of data. Whatever zipped it used 64 bit extensions to enable it to be that massive. 

So can’t be unzipped by the standard *nix unzip which doesn’t do use 64 extensions (AFAIK). The Windows 7 compact -u command borks completely (and unhelpfully). I’ve had a look at winrar, command line doesn’t like zips, and all the usual suspects for handling zips in Windows. None are really simple, available, and just, well, THERE, when I need them. What I need is something simple that’s on just about everything I use and doesn’t mutter “shareware” or “trial version” at me and I can use in a bash one liner.

Step forward java. The jar utility uses zip compression as part of it’s packaging, and can unzip zip files for you. It also uses whatever extensions necessary to get at those pesky enormous zip files that no-one ever expected anyone to be able to use.

On windows:

C:\tmp> "<path to your jdk>\bin\jar" xf

On bash:

jar xf

Ta-da. It ain’t quick but it works.

Check SHA or MD5 sums on Windows

I know I should do this more often on my own machine, but I am weak. However, it is a necessity when installing Open Source software at work that I check the checksums after download. I always forget how to do it and often laboriously check the whole thing with MK 1 human eyeball. Dumb.

So, to make it better, here’s the simple, command line way, to check the checksums.

You’ll need:

  1. cygwin installed
  2. cd’ed to the folder where you downloaded the installation file to.

echo "9819c6d21360f805cc17157c64c24672 apache-tomcat-7.0.42.exe" | sha1sum -c -

If it’s good you should get (depending on your gitbash/bash/etc shell you’re using):

apache-tomcat-7.0.42.exe: OK

If bad you get: 

apache-tomcat-7.0.42.exe: FAILED
md5sum: WARNING: 1 computed checksum did NOT match

or maybe

no properly formatted MD5 checksum lines found

Thanks to Les on upthescala

EDIT 20130816: Fixed typo in command to remove pipe and replace with redirect.
EDIT 20140128: What was I thinking on previous edit? Anyway, fixed and added a little more colour.

Serving HTTPS out of Bottle using CherryPy

I wanted to bash a couple of ideas together and see how quickly I could get something serving data from mongodb over https. I’d just started messing with python and the pymongo driver and I was (and still am) loving the quick hit python coding gives. Adding bottle as a framework made it all webby and useful feeling into the bargain :-). What is not not to like?

What I really wanted was a sweet single point solution that bottle already gave me, but with https. Past self takes over at this point….

After a bunch of digging around and reading lots of blurb, the upshot is a lightweight https server called CherryPy using bottle querying a mongodb using a regex and outputting whatever is in the first result to the screen, with a list of some of the other results, allowing the user to refine their search. The app is mickey mouse, and you do have a post where I’d prefer a redirect and using a cookie (so that browser refreshes work nicely), but the amount of data you can potentially return using the re would blow out the cookie size pretty quickly. There’s a probably a better way(tm) 😉

To do a test run you will need:

  • mongod running
  • If you want it not to throw a wobbler (I haven’t put much exception handling in it), you’ll need at least one row in testbottle.trades collection with at least one attribute of ‘Trade_Id’.
    So create a test db and record or two to play with. Start mongo shell and add some:

    use testbottle
    //bung one in with an easy one match re to remember
    		"Trade_Id" : "000457390G-1",
    		"Version_Number" : 1,
    		"Notes" : "",
    		"Settlement_Consideration" : 1721.72
    //let us bung a small shedload in starting with x
    for (var i = 0; i < 1000; i+=1) { 
    		"Trade_Id" : "x" + i, 
    		Version_Number : i, 
    		Settlement_Consideration : Math.random() * i * 100
  • python installed
  • pip install pymongo – OR if on windows – use the windows msi
  • pip install bottle
  • pip install cherrypy
  • pip install pyOpenSSL – OR if on windows – use the windows msi
  • Create a top level folder for this project. I used pyMongoBottlePoc
  • create your own ssl cert –
    openssl req -new -x509 -keyout testserver.pem -out testserver.pem -days 365 -nodes

    Put this in your project top level folder.

  • The code is here. Put this in your project top level directory
  • The hello page and show found pages are here. Put these in “views” subdirectory
  • Run it by typing at the command line
  • Use it by pointing your web browser at
    https://localhost:8082/ is a webserver on your localhost port 8082 which initially shows a welcome page hello_get_value.tpl asking you to input a trade id. When you click submit, does a regex search in the mongo database returns a cursor (trades) and the first result (trade). Bottle renders them using show_trade.tpl.

I leaned really heavily on this post from Diego XA and on the 10Gen Dev course youtube videos week 1 which have an excellent intro to python course in them. Thanks guys.

Installing Python, pyMongo, and Bottle on Windows

I had no problem installing all of the above on my Mac. On Windows, however, it was a bit of a pain. Thinking about it, it maybe that the mac install was Python 2.7, which everyone has been there and done that, whereas the Windows install was python 3.3, which may be a little later to the party.

Anyway, what I did:

  1. Install python 3.3 from Use the msi installer
  2. Install the pymongo using the msi from > Look down the bottom to find the windows installers. I suggest you don’t do an easy_install, it won’t compile properly and will either not work at all, or will have degraded performance.
  3. Install distribute. The instructions are here. I found it easiest to install from cygwin/git-bash using:
     $ curl -O
    $ python
  4. Install pip using easy_install supplied by distribute. NB: you must do this from the DOS command line (and, it appears, be in the Scripts folder):
     C:\> cd c:\Python33\Scripts\
    C:\> easy_install pip
  5. I also installed bottle at this point.
     C:\> pip install bottle

Many experienced python developers are recommending using virtualenv. It allows you have a virtual python install where you can pip install anything without blowing up your base python install by accident. You may have to use the performance retarded pymongo as Windows 7 (32-bit) doesn’t have the c libraries needed to build the necessary modules.

Hat tip to khattam for the distribute/easy_install usage.

GIT: Find commits for a specific file

git log --stat component/src/main/resources/META-INF/spring/context-osgi.xml

 Gets you a list of all the commits including that file.

git diff $commit_hash_old $commit_hash_current component/src/main/resources/META-INF/spring/context-osgi.xml

Shows you the diffs between the old file and the one you have now.

Cross ref: git blame might be more useful. Tells you which lines of a file were changed by whom:

git blame component/src/main/resources/META-INF/spring/context-osgi.xml

6d4e6fd3 (Dick Dastardly 2012-07-26 15:28:33 +0100 24)
6d4e6fd3 (Dick Dastardly 2012-07-26 15:28:33 +0100 25) /data/error/
c3cedfe7 (Dick Dastardly 2012-06-13 16:10:19 +0100 26)
6d713a0a (Muttley 2012-10-18 13:44:33 +0100 27)
3233b294 (Peter Perfect 2012-11-07 13:16:34 +0000 28) mappingPersistenceUnit
3233b294 (Peter Perfect 2012-11-07 13:16:34 +0000 29) HSQL
c3cedfe7 (Dick Dastardly 2012-06-13 16:10:19 +0100 30) org.apache.openjpa.jdbc.sql.HSQLDictionary
c3cedfe7 (Dick Dastardly 2012-06-13 16:10:19 +0100 31) org.hsqldb.jdbc.JDBCDriver
47e39616 (Dick Dastardly 2012-06-21 14:12:40 +0100 32) jdbc:hsqldb:data/resource/lookup.db
3233b294 (Peter Perfect 2012-11-07 13:16:34 +0000 35) 1