Requesting a URL with Python

After a year at the bottom of my drawer, I have busted out my Pimoroni Flotilla. It has a Python API and so I figured this was as good a time as any to use Python3 for the first time in many years.

Part of the kit is an LED matrix. I want to use this to display how many people there are in space right now.

There is a great website which will answer this question (howmanypeopleareinspacerightnow.com) which also has a JSON endpoint. It turns out there are lots of ways of getting this information from Python. Here are three I tried.

requests

Requests calls itself “HTTP for Humans”, and is mostly short and concise, but I need to send some extra headers with the request because of some user-agent filtering on the server.


import requests

url = 'http://www.howmanypeopleareinspacerightnow.com/peopleinspace.json'
headers = {'user-agent': 'space-requestor/0.1'}

response = requests.get(url, headers=headers)

print(response.status_code) # 200
print(response.text) # response body as text

inspacenow = response.json() # response body as JSON object

print(inspacenow["people"][0]["name"]) # 'Peggy Whitson'

urllib3

urllib3 describes itself as a ‘powerful, sanity-friendly HTTP client’ and it’s more verbose than I’d like for my simple case, but feels like it might scale into a larger application quite well.


import urllib3
import json

url = 'http://www.howmanypeopleareinspacerightnow.com/peopleinspace.json'

http = urllib3.PoolManager()
response = http.request('GET', url)

print(response.status) # 200
print(response.data) # response body as byte string

inspacenow = json.loads(response.data.decode('utf-8')) # response body as JSON object

print(inspacenow["people"][0]["name"]) # Peggy Whitson

unirest

Unirest is a collection of eight HTTP client libraries for multiple programming languages, but supporting near-identical request idioms, which makes it easy to use if you are using multiple languages yourself.

Sadly, the Python library, which I’ve used before very happily, does not work with Python3.

If it worked, the main call would have looked something like response = unirest.get(url), which is a brevity I deeply appreciate!

Conclusion

Requests seems very popular, and seems like a good solid choice for HTTP requests in Python3 applications. It’s what I’ll be using for my humans in space monitor!

The surprise of Spring Boot

I am a long time Java programmer. It was what I used to write my final year project at university, and it was my first, second and third jobs.

That third job lasted more than a decade and, alongside a huge amount of other things, I got to deliver a number of projects in both PHP and Ruby. PHP was interesting and I’d happily work in it again, but Ruby was fun.

Terse, expressive, powerful and (mostly) lightweight in nature, it feels like a blessing of a language.

The time came to change job again though, and although I looked for some Ruby roles, I’d been a bit too hands-off to be able to be seriously considered and so I got another Java role. But! What a surprise! Whilst I wasn’t looking, Java frameworks learned some lessons from their “convention over configuration” brethren, and those lessons are mostly reflected in Spring Boot by Pivotal Software. Here is a complete Hello World web application in Spring Boot:

@RestController
class ThisWillActuallyRun {

    @RequestMapping("/")
    String home() {
        "Hello World!"
    }

}

This is borrowed from the Quick Start example, and is in Groovy, so there’s no other requirement for imports, classpaths etc. Just use the Spring Boot command line tool to run it, and you’re done!

The Getting Started guides cover a wealth of activities to do almost anything you can imagine, but the framework is perfect for writing microservice-based applications with very little configuration required beyond adding some annotations to your classes and methods.

It makes writing Java a pleasure again, and an experience far from the heavyweight EJB and Spring2 I was recoiling from a few years ago!

Resolution #1

Weigh myself every day. Nothing about losing weight, just about recording it.

That was one of my New Year’s resolutions. Let’s see how it’s going:

A chart of my weight on each of the days in January 2017

My weight in January 2017

I missed four days out of a possible 31, which means I had an 87% success rate, which I think is pretty good going for a New Year’s resolution 🙂

I am simply recording my weight on a normal set of bathroom scales at roughly the same time each day and immediately adding the data to the Withings app on my phone, which I already had because I have a Withings Activité. I am a pretty steady 69kg, with small peaks after each weekend.

Whether this brings about any change in diet or exercise is yet to be seen. In the meantime, I’m just going to keep recording the data.

Full fat

8 years ago The Guardian proudly announced that they were providing full-text feeds of, well, everything:

Today guardian.co.uk rolled out a major upgrade to the RSS feeds. Our feeds now contain the full content of each article so that you can take guardian.co.uk with you wherever you prefer to get your news.

Fast-forward back to today and not only do they not provide the full content of articles in their feeds (those clickthroughs and ad impressions being all-important), but not even their developer blog has been spared. This is pretty disappointing.

My first thought was to run it through the nice full-text RSS feed creator at fivefilters.org but it looks like The Guardian have asked them to not allow this for their domain (damn those ads!). Luckily there are still tools out there which will convert their truncated feed into full text for me, so I was saved from having to write the relevant dozen lines of code myself, although maybe that was the point, eh developer blog?

Government Digital Service developer docs

GDS has an absolute boat-load of repositories on GitHub, and no clear way to access or browse them.

One of those repositories, govuk-developers, is an ordered index to those repositories, allowing you to find exactly what you’re looking for. This is a static site generated with Middleman and served from Heroku, except that you need a GDS login to be able to access it (presumably for cost reasons).

In the description for that repo it says This project is being rebuilt and is undergoing rapid change. but there haven’t been any changes for nearly a week, so I’ve forked it.

It’s now running in my Heroku account with no username and password (there’s a prompt, but just hit <ENTER>), so you too can browse the GDS repos and docs in a useful way.

Reading feeds again

When Google Reader went away I raged and sulked and tried half a dozen services touted as replacements, but eventually I installed FeedDemon on the computer I used the most often and read my feeds daily, there.

A few weeks ago I found the Newsfold app feed reader for Android, logged back in to my Feedly account and imported the OPML from FeedDemon.

Since then I have been happily checking my feeds every day on my phone, more or less like I used to with the Google Reader mobile app. I only have a dozen or so going at the moment, but hopefully this will slowly step up and I will actually know what’s going on the world again without having to suffer Twitter.

The Guardian’s Christmas Gift Guide – on the web, not of the web

Screenshot of The Guardian's Christmas Gift List app

Congratulations to The Guardian for making me write a blog post.

It’s coming up to Christmas, and like all newspapers The Guardian have a Christmas Gift Guide. It actually looks quite nice, with well-sized images, responsive layout, good filtering and obligatory animated falling snow. When I opened it there were a couple of items which I was interested in buying. So far, so Christmas.

Screenshot of The Guardian's Christmas Gift List app

Screenshot of The Guardian’s Christmas Gift List app

The whole thing is very similar to Amazon’s ‘stream’ app (“Interesting finds, updated daily”) for browsing items. The difference comes when you actually click an item.

On both the Guardian and Amazon sites the page is dimmed and a modal display appears. This gives you more information about the product (Amazon’s one also has the decency to support pressing the Escape button to dismiss the modal), and a link to go through to the main product page.

Except that on the Guardian app, that’s not quite what happens.

Where Amazon’s pop-up display has a nice <a href="..."> for its link, meaning that you can open items in a new tab, send them to other tools, copy the link address and so on, the Guardian’s one has a <button>. A button with an attribute of data-target="...".

This means that all you can do is click it and hope for the best. Will it change the location of your current tab, taking you away from the gift list? (no) Will it open a product page in a new tab? (yes) If you’ve already clicked one item in the list and then click another will it open another tab or replace the existing one? (it replaces the existing one – you can’t have two tabs with two different items open at the same time).

As a user, this is incredibly frustrating, especially for something which I was so prepared to like.

Can I therefore take a short moment to suggest to web developers everywhere that if they are adding a link to a web page, they do so by using the markup intended to provide a link to a web page, and not jury-rigging their own link mechanisms with JavaScript on top of whatever element they think they can style the best?

Just in case anyone was wondering, I did go and look up the implementation to see if this was the fault of a crazy JavaScript framework. As far as I can tell, it’s not. Check out lines 237 and 238 of https://giftguide.imd1.uk/js/gui/xmas/view/singularProductView.js:

var targetUrl = $(event.currentTarget).attr('data-target');
window.open(targetUrl,"Christmas gift");

Sigh.

Render Conference 2016

Last week I was in Oxford for RenderConf.

It was brilliant.

Render conference is two packed days of technical front-end talks from a seriously awesome line-up of speakers.

And when they said awesome, they meant it. Go check out the schedule!

My top three talks (in no particular order) were:

Val Head (@vlh): Designing meaningful animation (about web animation from Disney to CSS bezier curves)

Ashley G. Williams (@ag_dubs): If you wish to learn ES6/2015 from scratch, you must first invent the universe (Computer Science theory, Picasso and teaching)

Frederik Vanhoutte (@wblut): Rendering the obvious (Rainbows, categorisation, and Beethoven’s exploding head)

Between these three talks alone there was creativity, practical advice, inspiration and a call to change how you look at the world. Across the other 17 talks you could get everything from demoscene coding in WebGL to how to use the <canvas> element to print a new scarf. There was a really great range of presentations.

All the talks will apparently be available on http://lanyrd.com/2016/renderconf/coverage/ shortly, and the videos will be on Vimeo. My photos, which are mostly of the nice things I saw in between talks because I was too busy listening, are on Flickr.

As well as being an exciting, inspirational conference though, the whole thing was just executed really, really well. Regular breaks, pastries and drinks, retro gaming (enjoying losing at videogames is my superpower), provided lunch, table tennis, skittles, 3D printing pens, hacking competitions and more meant there was always something to do and always someone interesting or cool to talk to.

There was a party at the venue on the first evening where the food even came to us – freshly-made burritos and pizzas from vans outside the venues complemented the wine, beers and alcohol-free options inside. Hats off to all the organisers!

I think my only complaint was that given this was my first visit to Oxford, I was disappointed to find out that not every single building is a 1,000-year old cathedral. Absolutely gutted.

Probably lucky though, I hear they have terrible wifi. See you next year, Render!

Pimoroni Flotilla

Flotilla – Friendly Electronics for All” is the headline, which sounds pretty damn good to me, someone who is permanently afraid of blowing his Arduino and other kit by connecting something the wrong way around.

So this, which has a ton of sensors and outputs, and a central dock for making them talk to each other, which all works over USB, is perfect for me.

We bought the Mega Treasure Chest starter kit and it arrived the other day, but it’s early enough in its life that there are very very few instructions on how to use it just yet. There are three “secret” URLs for getting going (the whole thing is controlled by a Raspberry Pi so you’ll need one of those too!):

  1. http://flotil.la/start/ – instructions on setup are here
  2. http://flotil.la/rockpool – the web interface for controlling your flotilla is here
  3. http://flotil.la/cookbook/ – the two demos which do exist are here

We also used a first version of the “Getting Started” booklet to see us through the initial stages.

Once you’ve done those, you can watch the video for how to build your own robot which you can control over your network using all the different sensors and controllers you get. My 6yo was able to put the whole thing together himself and learn how to control it predictably in under two hours, which was brilliant! We did actually run out of the plastic screws used to hold it all together – I think someone in the warehouse was tired and we ended up with fewer than expected but just about managed to get it all together and have been assured more are on their way!

The completed robot! (upside down!)

I was really glad to discover that the flotilla daemon runs on a Kano, which we’ve had for a couple of years now and which my son loves, because the forums had made me a bit nervous about which OSes were supported.

We’re really looking forward to the rest of the instructions arriving, but my son is already completely delighted with the fact that he has built, and can control, his own robot! I just want to know if the flotilla daemon will run from a Pi Zero powered by a power bar so we can put them on the back of the robot and cut the cord it currently needs to the regular Pi!

Estimating is good for your health

Matt Jukes has written a post called Don’t do Agile. Be agile. Or something. where he describes how estimating stories has not worked for them at the Office of National Statistics.

I think there is a more important reason to do estimation that just release planning, and that’s people management.

Over time, estimating allows you to create a sprint which is neither too daunting nor too trivial, and to build a good team cadence. A sprint with too much work in it can be depressing and ultimately slow down delivery, a sprint with too little runs the risk of falling foul of Parkinson’s law.

You could argue that if you have a good, committed team who are always up for the challenge of whatever’s put in front of them then it doesn’t matter – I’d argue the opposite and that you’re at risk of burning out your team.

Velocity, paired with your observations of the team, will give you the data to be able to work out if they are maintaining a sustainable pace or not.

As much as I am not a fan of Pivotal Tracker, it’s taught me that the burndown is not as important as having a graphical view onto historical velocity. Did the team deliver fewer points this sprint? Why? Was it story-related or human-related? If their velocity has been solid and steady for a long time, are they ready to step up, or do they need a rest? Most managers have a gut feel for these things, but having the data to be able to back it up makes it easier to discuss in retrospectives and planning sessions.

Estimation is not always easy, but so long as you don’t put too much importance on getting the “right” estimate, and so long as it’s really treated like an estimate, it can be extremely valuable for both release planning and maintaining the long-term health of your team.