Peter Hicks' Blog

Author: Peter Hicks

  • Importing Ordnance Survey Open Data in to PostgreSQL with PostGIS

    Some time ago, I looked at some uses for Ordnance Survey Open Data, coming to the conclusion that a sensible way to work with it would be to import it in to a geospatial-enabled database.
    Each set of data is provided in ESRI Shapefile format, and has four files:

    • shp – shape format
    • shx – shape index format
    • dbf – attribute format in dBase IV format
    • prj – projection format

    The shp2pgsql command converts SHP files in to a set of SQL commands which will effectively import the data in to PostgreSQL. Here’s a ridiculously simple guide to importing a file:
    createdb os_opendata
    psql -d os_opendata -c "CREATE EXTENSION POSTGIS"
    shp2pgsql <filename>.shp <table_name> | psql -d os_opendata
    Depending on the speed of your machine, in a few seconds you’ll find a new table in your database with all the data included.
    And finally, what if you just want to import all of the data at once? Try this:
    find Data/ -name "*.shp" | xargs -I % -n1 shp2pgsql % | psql -d os_opendata

  • Ubuntu 14.04 for Productive People

    Way back in 2011, I blogged about Ubuntu 11.10 for Productive People, which took the form of a mini tutorial on how to wrestle some of Ubuntu’s UI candy away and replace it with something better suited to being productive.
    I’m still standing by my assertation that Ubuntu is too ‘pretty’ on the desktop now, and lacks a ‘power user’ mode, but I won’t argue with anyone who says it’s great. It’s not a false dichotomy – you can have a power mode and a pretty mode in a desktop operating system.
    Updated for the current beta of Ubuntu 14.04LTS, here are the instructions on how to get the latest release of Ubuntu in to shape:

    • Install Ubuntu 14.04
    • Install Gnome using apt-get install gnome – use lightdm as the display manager
    • Remove the slightly obstructive overlay scrollbar with apt-get remove overlay-scrollbar
    • Log out, then log back in again but click the Ubuntu logo by your username and select ‘GNOME Flashback (Metacity)’
    • Run gnome-tweak-tool, select Fonts and set the text scaling factor to 0.9, then under Appearance, set the Icon theme to Gnome and Cursor theme to Adwaita. Under Top Bar, check ‘Show date’ and ‘Show seconds’

    Refreshingly easy, isn’t it? I’m going to be updating to 14.04LTS when it’s released!

  • Open Rail Data – Two Years On

    After a brief, but really interesting visit to the former Bletchley PSB (or signalbox, if you’re less of a railway geek), I popped in to OpenTech 2013 to present an update to the presentation I gave two years ago.

    In some ways, we’ve come a long way – in others, maybe not. Regardless, there’s scope for opening up more data to make us all more aware of what’s going on – suggestions immediately afterwards included getting data on cable theft incidents, counts of people going through ticket barriers at stations in real-time, plus passenger counts from trains.

    My presentation is available if you missed it, or if you want to cut-out and keep. Exciting times 🙂

  • Really Useful Things

    Here are five things that help my quest to reduce the amount of time I spend supporting and increase the amount of time I spend doing:

    • Google Apps for Business – £3.30 per user per month, and it means I can use Google Drive (although an Ubuntu client that isn’t InSync would rock) to hold PDFs and reference data I use all the time, and not have to worry about email hosting.
    • LastPass – password management. Much easier than having a GPG-encrypted text file that you sync to DropBox. Premium is only $12 (£8) per year.
    • GitHub – online project hosting using Git. I use GitHub by default for almost everything I do, private and public. A snip at $7 (£4.60) per month.
    • Atlassian Hosted JIRA – which, love it or hate it, works well for OpenTrainTimes bug tracking. Reasonably priced at $10 (£6) per month for up to 10 users.
    • Evernote Premium – write notes, sync them automatically and go search. I still love my Moleskine notebook for taking most meeting notes, and it looks great too – but sometimes you just gotta type electronically. £4 per month, but I really wish they’d come up with a sensible client for Ubuntu.

  • Installing an Olive on VirtualBox

    Although it exists in many other places, I’ve not found a comprehensive set of instructions for installing JunOS 11.4 under VirtualBox that actually works. As I found, It isn’t too difficult, and only took me a day or so.
    You’ll need to create a FreeBSD machine in VirtualBox with 1Gb of RAM and 5Gb of disk space. Select one or more network interfaces as the Intel PRO/1000 MT Desktop adapter. If you’re running on a UNIX system, additionally redirect the COM1 serial port to a host pipe called /tmp/com1. Use the command socat /tmp/com - to show the output from the serial console, which is useful after booting the Olive for the first time.

    Installing FreeBSD

    • Download the FreeBSD 4.4 mini ISO from ftp://ftp-archive.freebsd.org/pub/FreeBSD-Archive/old-releases/i386/ISO-IMAGES/4.4/4.4-mini.iso
    • Create a FreeBSD machine with 1Gb of RAM. Create a VDI startup disk (either dynamically allocated or fixed size) of 4Gb
    • Edit the machine settings and enable network adapters as Intel PRO/1000 MT Desktop (82540EM) adapters. Although FreeBSD 4.4 won’t support these, JunOS 11.4 will.
    • Attach the ISO image to the CD/DVD drive in the machine, and boot it.
    • When FreeBSD boots, select ‘Skip kernel configuration and continue with installation’.
    • At the /stand/sysinstall menu, select a Standard installation.
    • At the ‘FDISK Partition Editor’ screen, delete any existing slices and create a single FreeBSD slice covering the entire disk by pressing ‘A’. Press ‘Q’ to finish – changes are automatically saved.
    • At the ‘Install Boot Manager for drive ad0?’ page, select ‘Standard’ so as not to install a boot manager.
    • At the ‘FreeBSD Disklabel Editor’ screen, create partitions as follows:
      • 1G filesystem mounted on /
      • 512M swap partition
      • 512M filesystem mounted on /config
      • Remaining space in a filesystem mounted on /var
    • Press ‘Q’ to finish – changes are automatically saved.
    • At the ‘Choose Distributions’ page, select ‘X’ to exit’.
    • At the ‘Choose Installation Media’ page, select ‘Install from a FreeBSD CD/DVD’.
    • The disk will now be partitioned, filesystems created and FreeBSD installed.
    • After installation, the following questions will appear. Answer ‘No’ to each:
      • Would you like to configure any Ethernet or SLIP/PPP network devices?
      • Do you want this machine to function as a network gateway?
      • Do you want to configure inetd and simple internet services?
      • Do you want to have anonymous FTP access to this machine?
      • Do you want to configure this machine as an NFS server?
      • Do you want to configure this machine as an NFS client?
      • Do you want to select a default security profile for this host?
      • Would you like to customise your system console settings?
    • Answer ‘Yes’ to “Would you like to set this machine’s time zone now?”. Select ‘No’ to “Is this machine’s CMOS clock set to UTC?”, then select ‘8 – Europe’, ’42 – United Kingdom’ then ‘1 – Great Britain’. Answer ‘Yes’ to “Does the abbreviation ‘BST’ look reasonable?”
    • Answer ‘No’ to “Would you like to enable Linux binary compatibility?”
    • Answer ‘No’ to “Does this system have a USB mouse attached to it?”, then select ‘Exit’ at the “Please configure your mouse” menu
    • Answer ‘No’ to the question regarding browsing the FreeBSD package collection.
    • Answer ‘No’ to the question regarding adding initial user accounts.
    • Set a password for the ‘root’ user.
    • Answer ‘No’ to the question regarding the last chance to set options.
    • Select ‘X’ to exit installation, detach the ISO image and select ‘Yes’ to the “Are you sure you wish to exit?” question.
    • The virtual machine will now restart.

    Creating a JunOS installation image

    Download junos-olive-patch.sh and run it against a standard JunOS installation image, for example:
    user@host:~$ ./junos-olive-patch.sh jinstall-11.4R2.14-domestic.tgz
    This will unpack and patch the installation file, replacing ‘checkpic’ in the pkgtools archive with a symbolic link to /bin/true so the package will install on an Olive.
    To get this installation package on to VirtualBox, make it in to an ISO file using mkisofs:
    user@host:~$ mkisofs jinstall-11.4R2.14-domestic.tgz > olive.iso
    Attach the ISO image to the Olive in VirtualBox, then mount the ISO file on FreeBSD by typing mount /cdrom. Install the package by running pkg_add -f jinstall-11.4R2.14-domestic.tgz.
    Reboot, and wait for the BTX loader screen to disappear – this may take several minutes. If you’re using socat to monitor the output of the console, you’ll see JunOS being installed.

  • Installing WebSphere MQ 7.5 on Ubuntu 12.04LTS

    An item on my mental To Do list for some time has been getting WebSphere MQ installed on Ubuntu. I don’t use it in a production environment, therefore an IBM PartnerWorld Software Access agreement gets me the software for a year, and a virtual machine with Ubuntu circumvents the need to buy Red Hat Enterprise Linux.
    With many things I do that could potentially turn out complicated or non-trivial, I document the entire process. I’m assuming you’re running a 64-bit version of Ubuntu 12.04LTS at this point, and that you’ve also read Quick Beginnings for Linux on IBM’s site.
    First, install the ‘rpm’, ‘pax’ and ‘default-jre’ packages. IBM distribute MQ as a set of RPMs, and I decided to install these rather than rebuild them as .deb files.
    Next, create /etc/sysctl.d/50-webspheremq.conf with the following:
    kernel.msgmni=1024
    kernel.shmmni=4096
    kernel.shmmax=73834496
    kernel.shmall=2097152
    kernel.sem=500 256000 250 1024
    fs.file-max=32768
    net.ipv4.tcp_keepalive_time=500
    Make these changes live by running sudo sysctl -p.
    Run sudo ./mqlicence.sh, read and agree to the licence.
    ‘crtmqpmg’ expects to find ‘pax’ in /usr/bin, so create symbolic link to ‘pax’ by running sudo ln -s /bin/pax /usr/bin/pax, then create /usr/lib64 using sudo mkdir /usr/lib64 as the installation process will want to write files in there.
    Create a set of MQ packages by running ./crtmqpkg $ID, where $ID is an installation identifier, such as ‘dev’ or ‘prod’. This may take some time. When it’s finished, change to /var/tmp/mq_rpms/$ARCH and install the following RPMs using the command sudo rpm -ivh --nodeps --force-debian:

    • MQSeriesRunTime
    • MQSeriesServer
    • MQSeriesSDK
    • MQSeriesClient
    • MQSeriesJava
    • MQSeriesSamples
    • MQSeriesGSKit
    • MQSeriesMan

    Set this MQ installation as default using sudo /opt/mqm/bin/setmqinst -i -p /opt/mqm, then run dspmqver to verify the installation is correct.
    Finally, add your user in to the ‘mqm’ group by running sudo addgroup $USER mqm. Log out and log in for these changes to take effect.

  • Opening Great Britain’s Rail Data

    It’s my first time in Helsinki, and the weather is much the same as a September day in London – wet.
    I finished preparing for my talk at OKFestival a little under 24 hours ago, and it went without a hitch. These things are normally OK once you’ve finished worrying about them.
    Anyway, the slides and video (which no longer works as of February 2020) of my presentation on Open Train Times and Opening Great Britain’s Rail Data are now online. Enjoy!

  • The Chancellor's Autumn Statement

    Many people reading my blog are interested in Open Data – here are the three important paragraphs from the Chancellor’s Autumn Statement earlier, as they relate to Open Data:
    “1.125 Making more public sector information available will help catalyse new markets and innovative products and services as well as improving standards and transparency in public services. The Government will open up access to core public datasets on transport, weather and health, including giving individuals access to their online GP records by the end of this Parliament. The Government will provide up to £10 million over five years to establish an Open Data Institute to help industry exploit the opportunities created through release of this data.”
    “A.146 Open Data Institute – The Government will provide up to £10 million over five years, with match-funding from industry and academia, to establish the world’s first Open Data Institute to help business exploit the opportunities created by release of public data”
    “A.140 Rail fares data – The Government will consult in early 2012, through the Fares and Ticketing Review, on providing open access to rail fares data, giving passengers and business better information and enabling them to make the most cost-effective travel choices.”
    The Cabinet Office website has further details in a PDF here.
    I’ll leave it at that for the moment – other people will doubtless be writing their take on it, but I’ll leave you with one word from me: positive.

  • VirtualBox is too clever

    I have a desktop machine connected to a Gigabit Ethernet switch, plugged in to my Cisco 1801 router. The switch is dumb and doesn’t support VLAN tagging, which is fine as I have everything in a single VLAN.
    My desktop machine works fine with IPv6, however when I brought up a virtual machine on VirtualBox, IPv6 breaks for that particular VM. “ip addr show” tells me that it’s autoconfigured three IPv6 addresses, each on separate subnets – one of them correct, two wrong (but valid on other subnets here).
    What’s going on? It turns out I’d set the port to my GigE switch up as a trunk port, with the native VLAN being the user access VLAN, and two other VLANs allowed. What this does is to send frames in VLAN 4 on to the wire untagged, and any untagged frames received automatically drop in to VLAN 4. However, any broadcast frames from VLANs 5 and 6 are sent on to the wire with an 802.1q header – and VirtualBox, being very clever, seems to pass these up to the virtual machine without their 802.1q headers.
    I may file a bug – that’s too clever 🙂

  • SSL Certificate error whilst using rvm

    Given that I have a lovely clean install of Ubuntu 11.10, I decided to use rvm to manage by Ruby installation. The only problem with that is the SSL certificate beginrescueend.com uses isn’t known to Ubuntu.
    I’ve seen a post advocating using -k to ignore CA certificate validation, but that’s not the right thing to do. It works, but only by defeating a security mechanism.
    Cut to the chase – how to get around it:

    • Read the error message curl produces, which suggests visiting http://curl.haxx.se/docs/sslcerts.html
    • From there, download cacert.pem and copy this to your home directory
    • Create ~/.curlrc, with the single line cacert = ~/cacert.pem
    • Test using curl https://rvm.beginrescueend.com/ > /dev/null – the CA error should not appear