When a bit of government forwards or attaches emails using Outlook, they get sent using a special, strange Microsoft email format. Up until now, WhatDoTheyKnow couldn’t decode it. You’d just see a weird attachment on the response to your Freedom of Information request, and probably not be able to do anything with it.
He then told us about it, and I merged his changes into the main WhatDoTheyKnow code, tested them out on my laptop, then made them live. It all work perfectly first time. Peter even added the new dependency on vpim to WhatDoTheyKnow conf/packages.
Now if you go to an Outlook attachment on WhatDoTheyKnow,such as this one you’ll just see the files, and be able to download them, and view them as HTML as normal. They’ll also get indexed by the search (although I need to do a rebuild for that for it to work with old requests).
If you want to have a go making an improvement to a mySociety site, you can get the code for most of them from our github repositories. For some sites, there’s an INSTALL.txt file explaining how to get a development environment set up. Let us know if you do anything – even incremental improvements to installation instructions are really useful. And new, useful, features like Peter’s are even more so.
I’m enjoying the weather at the moment, seems to be sunnier than the summer, but cool with an atmospheric autumnal taste in the air.
mySociety is changing as ever, leaping forward in our race to try and make it easier for normal people to influence, improve or replace functions of government. More on this as it happens.
Meanwhile, I’ve been continuing to hack away at WhatDoTheyKnow. A little while ago Google decided to deep index all our pages – causing specific problems (I had to tell it to stop crawling the 117th page of similar requests to another request), and also ones from the extra attention. There have been quite a few problems to resolve with authority spam filters (see this FOI officer using the annotation function), and with subtle and detailed privacy issues (when does a comment become personal? if you made something public a while ago, and it is now a shared public resource, can you modify it or take it down?).
Right, I’ve got to go and fix a bug to do with the Facebook PledgeBank app. It’s to do with infinite session keys, and how we send messages when a pledge has completed. Facebook seem to change their API without caring much that applications have to be altered to be compatible with it. This is OK if the Facebook application is your core job, but a pain when you just want your Facebook code to keep running as it did forever.
(the autumn photo thanks to Nico Cavallotto)
It’s the first full working day for the new facility to annotate Freedom of Information (FOI) requests on WhatDoTheyKnow, and people have been hard at it.
Mr Ormerod points out that private information isn’t necessarily so private if someone has died, so perhaps the exemption the MOD used shouldn’t apply.
Trevor R Nunn has posted three annotations (e.g. this one) to show that his three FOI requests are being treated as one. The annotations facility is great for handling edge cases like this, which don’t happen often enough to be worth explicitly adding to the code, but need some mention.
And finally Edward Betts has processed the list of post boxes retrieved by FOI into a more structured data format, and posted up a link to it. Exactly the kind of collaboration I love to see!
And that’s just this morning!
Francis has been furiously adding new features to our Freedom of Information website WhatDoTheyKnow ever since it launched earlier this year. He’s just added one of the most important missing features, the ability to leave annotations or comments on FOI requests.
This is especially useful for providing plain English summaries of what information in a response was actually interesting, or to discuss refusals to supply information and what to do with them. To add one just go to a request page and scroll to the bottom, just like adding a comment on a blog post.
So, whether you’ve made a request in the past, or you’re just interested in helping out, get annotating.
Whereas new sites are lovely, and I talk about Neighbourhood Fix-It improvements further down, there’s still quite a bit of work that needs to go into making sure our current sites are always up-to-date, working, and full of the joys of spring. Here’s a bit of what I’ve been up to recently, whilst everyone else chats about database upgrades, server memory, and statistics.
The elections last week meant much of WriteToThem has had to be switched off until we can add the new election results – that means the following aren’t currently contactable: the Scottish Parliament; the Welsh Assembly; every English metropolitan borough, unitary authority, and district council (bar seven); and every Scottish council. The fact that the electoral geography has changed a lot in Wales means there will almost certainly be complicated shenanigans for us in the near future so that our postcode lookup continues to return the correct results as much as possible.
Talking of postcode lookups, I also noticed yesterday that some Northern Ireland postcodes were returning incorrect results, which was caused by some out of date entries left lying around in our MaPit postcode-to-area database. Soon purged, but that led me to spot that Gerry Adams had been deleted from our database! Odd, I thought, and tracked it down to the fact our internal CSV file of MLAs had lost its header line, and so poor Mr Adams was heroically taking its place. He should be back now.
A Catalan news article about PledgeBank brought a couple of requests for new countries to be added to our list on PledgeBank. We’re sticking to the ISO 3166-1 list of country codes, but the requests led us to spot that Jersey, Guernsey and the Isle of Man had been given full entry status in that list and so needed added to our own. I’m hoping the interest will lead to a Catalan translation of the site; we should hopefully also have Chinese and Belarussian soon, which will be great.
Neighbourhood Fix-It update
New features are still being added to Neighbourhood Fix-It.
Questionnaires are now being sent out to people who create problems four weeks after their problem is sent to the council, asking them to check the status of their problem and thereby keep the site up-to-date. Adding the questionnaire functionality threw up a number of bugs elsewhere – the worst of which was that we would be sending email alerts to people whether their alert had been confirmed or not. Thankfully, there hadn’t yet been any such alert, phew.
Lastly, the Fix-It RSS feeds now have GeoRSS too, which means you can easily plot them on a Google map.
I’m happy to say that with a lot of help from Francis and Matthew I’ve just rolled out a long-requested addition to WriteToThem. You can now use the site to write one message and send it to all your representatives in a multi-member constituency – so, for example, you can send a message to all your MEPs at once. A nice side project along the way has been getting the test suite for WriteToThem running happily on a mySociety server, and adding a few more tests. Run, tests, run! Good tests.
So, there are now over 600 petitions in the petitions system, and we’re getting a steady stream of appeals from our users to add categories.
I’m posting to ask how you all think we should handle this. It seems to me that there are a few options:
- Ask petition creators to pick one very basic top level category of no more than 10 or so, taken from a hierarchical taxonomy like the one the BBC uses.
- Ask petition creators to pick the top level and the subsequent sub-levels to be more specific.
- Go all web 2.0 and simply ask people to tag their petitions with some key words
More than just thinking about the overall philsophy I’d also appreciate thoughts on design. When you come to the homepage, how should the category system be presented to you? Tricky stuff, and I’d really appreciate your thoughts.
Just a short one for today. Here’s a plot of the rate of take-up of Engineers Without Borders UK’s successful pledge to solicit funding for a development project in Suriname (the first of what we hope will be many projects using PledgeBank to raise funds):
[Plot showing takeup of EWB Suriname fundraising pledge – image gone]
— we’ll be incorporating this (and some other handy information, such as how many people are looking at the pledge pages and which sites link to them) into the site some time soon to help pledge creators see how they’re doing at publicising the pledges (and also because graphs look cool).
Today’s other task was to switch the live site over to using the new login system which Francis mentioned the other day. This was complicated by the fact that there might be some users who had received a signup token, but hadn’t clicked through on the token to actually complete their signup before the site was shut down for the upgrade. The solution to this is a bit nasty, but it seems to have worked OK, which is nice.