When you send a Freedom of Information request, a clock starts ticking. Here in the UK, public authorities are bound by law to answer a request “promptly, and in any case within 20 working days” — but of course they can only respond if they’ve received the request.
And, while email is generally reliable, we’re all familiar with the occasional mishaps it can bring: mailboxes that are full, accounts that have been closed down, or messages being returned because they look too much like spam.
Sign here please
Email works a bit like signed-for physical mail. When a letter is delivered to a recipient they either sign to say they’ve received the letter, or the mail company records that there was no-one available to accept the delivery. This lets the mail company keep the sender up to date with where their letter is. Mail servers do the same — the recipient server sends a confirmation that a particular email has been received, or an error code is reported by your mail server if there’s a problem delivering the email.
Like physical mail, we can only verify that the message has been accepted at the destination address. It’s then under the recipient’s control to get it to the right person at that address, a bit like a receptionist receiving a letter for an employee 10 floors above. We think that if an authority’s mail server confirms that one of our emails has been delivered, it’s their responsibility to ensure it reaches the correct people to be able to answer your FOI request.
Proof of receipt
Look at the header of any request on WhatDoTheyKnow, and within 24 hours, in most cases you’ll now see a small green ‘delivered’ confirmation:
Most users can click on this to see further confirmation:
But if you’re the owner of the request, when you click on the green ‘delivered’ link, you’ll see information from the mail logs as the message passed through our server. If there’s ever a query about whether or not a message was delivered, you can hand these on to the authority to help them analyse any issues.
On the rare occasions that something goes wrong, here’s what users will see instead:
– and if it’s your own request, again, you’ll have access to the mail logs.
Small but mighty
This feature might look small, but there’s a lot of thinking behind it — just check the length of the trail on Github, our ticketing system. Anyone will be able to understand the amount of discussion and problem-solving that went into the addition of this small green tick, while the more technically-minded may also find it interesting to see the coding solutions as they unfolded.
This small green tick also gives users something rather powerful: proof that their request was received by the authority’s mail server at a specific time, should that be disputed.
The suggestion for this feature came initially from one of the WhatDoTheyKnow volunteers. It took some time to implement, but we’re pleased to say that it has now been made available for all Alaveteli sites in release 0.25.0.0.
SayIt is still in active development – that’s to say, we’re working on it and adding new features all the time.
One of the significant features that recently went live is that SayIt sites can now have more than one editor.
Who needs to collaborate?
We can think of lots of reasons to collaborate:
- If you’re minuting meetings, you can get others to share the burden of recording, proofing and editing.
- If you are collecting together statements about a wide topic, or expect your project to run over some length of time, you can call in people to help.
- If you’re running a campaign, you can ask your supporters to help you crowd-source relevant material.
- If you are part of an oral history archive, or similar community project, you can share access with others in your group.
…and we’re sure that there are many other uses too.
How to collaborate
You’ll need a SayIt site.
If you haven’t started your SayIt site yet, just go to this page to start one up. If you’ve got one already, log in.
Then invite your collaborators. Just click on the ‘invite friends to help’ button.
Repeat as required. You can invite as many people as you like.
Ways to use SayIt
Check out our series of step-by-step blog posts to explain how to:
That year saw many debates in Parliament on topics that have since become very familiar: the question of whether the tax on cigarettes should be raised; whether cigarettes should be advertised on television, whether smoking should be allowed in public places, and whether warnings should be printed on packets.
Rich and fascinating stuff for any social historian – and it’s all on TheyWorkForYou.com.
Hansard is an archive
Hansard, the official record of Parliament, is a huge historic archive, and whatever your sphere of interest, it is bound to have been debated at some point.
Browsing through past debates is a fascinating way of learning what the nation was feeling: worries, celebrations, causes for sorrow – all are recorded here.
How to use TheyWorkForYou to browse historic debates
TheyWorkForYou contains masses of historic information: House of Commons debates back to 1935, for example, and details of MPs going back to around 1806. You can see exactly what the site covers here.
There are various ways to search or browse the content. Start with the search box on the homepage – it looks like this:
You can do a simple search right from this page, or choose ‘more options’ below the search box to refine your search.
We’ll look at those advanced options later, but let’s see what happens when you input a simple search term like ‘smoking’.
Here (above) are my search results, with my keyword helpfully highlighted.
By default, search results are presented in reverse chronological order, with the most recent results first. If you are particularly interested in historical mentions, you may wish to see the older mentions first.
That’s easy – just click on the word ‘oldest’ after ‘sorted by date’:
You’ll notice a few other options here:
- Sort by relevance orders your results with the most relevant ones first, as discerned by our search engine. This will give you those speeches with the most mentions of your keyword ahead of those where it is only mentioned once or twice.
- Show use by person displays a list of people who have mentioned your keyword, with the most frequent users at the top. This can be fascinating for games such as “who has apologised the most?” or “who has mentioned kittens most often?”
Click through any of the names, and you’ll see all the speeches where that person mentioned your keyword.
That’s a good start – but what if there are too many search results, and you need some way to refine them? You’ll notice from my screenshots above that there are (at the time of writing) over 10,000 mentions of smoking.
That’s where Advanced Search comes in. You can access it from a few places:
- The ‘more options’ link right next to the search box on search results pages (see image below)
- The ‘more options’ link below the search box on the homepage (see image below)
- Or just navigate directly to our dedicated Advanced Search page (see image below)
Whichever way you arrive at it, the Advanced Search page helps you really get to the content you’re interested in.
The pink box on the right gives you some tips for effective searching.
For example, just as with Google, you can search for exact phrases by putting your search term within quotation marks. Otherwise, your results will contain every speech where all your words are mentioned, even if they’re not together. For phrases like “high street”, this could make a real difference.
Even if you are only searching for a single word, you can put it in quotation marks to restrict the use of ‘stemming’ – so, for example, a search for the word house will also return results containing houses, housing and housed, unless you put it in quotation marks.
You can exclude words too: this can be useful for minimising the number of irrelevant results. So, for example, you might want to find information about the town of Barking, but find that many of your results are debates about dogs. Simply enter the search term “barking” -dogs. The minus sign excludes the word from your search.
In the main body of the page, you’ll also see options to restrict your search to within certain dates, or a specific speaker, or a department, section (eg Scottish Parliament or Northern Ireland Assembly) and even political party.
Get stuck in
The best way to see what you can find is to dig in and give it a go. If your search doesn’t work for you the first time, you can always refine it until it does.
Let us know if you find anything interesting!
mySociety is an organisation with many priorities, and they often compete for attention. Right now, we have some time and budget to lavish on TheyWorkForYou. We need your input to help us understand what development we should prioritise for the site.
Note: if you don’t know much about TheyWorkForYou, your opinion still counts! See the foot of this blog post for an overview of the site and its aims.
Some suggested improvements
Below is a list of improvements that other users have suggested, or that we think are desirable. Which improvements would you most like to see – from this list, or based on your own needs?
- Easier sharing via social media If you see a debate you want to share with your Twitter or Facebook buddies, all you’d have to do is click a button. More details
- When your MP voted Letting you know, via email alerts, when your MP has taken part in a vote. More details
- Option to search just headings At the moment, search covers all content of debates, including everything anyone said. This option would allow you to only search headings, meaning that you could be sure the results were entirely focused on your topic. More details
- Tweeting debate headings or future business A Twitter account which would tweet, and link to, every debate in Parliament, or upcoming events. More details
- Signposting of big ‘events’ such as the Budget These are not always easy to find if you don’t know your way around, so we’d make sure the big events were always trumpeted on the site. More details
Great ideas, or utter bunkum? Let us know. You can give us feedback via any of the following methods:
1. Leave a comment under this blog post;
3. If you’d like to see the whole list of suggestions and issues, you can do so on our development list at Github (and the ‘more details’ links in the list above go to the issues on there). Note that anyone is welcome to add comments to these issues, or even to create your own (please search first to make sure you’re not duplicating an existing issue). Github may look complex, but it’s easy enough to use – you just need to set up a user account here.
We’re keen to understand whether we’re serving all kinds of users, so it’d also be helpful if you could tell us whether you consider yourself to be someone who knows a bit about Parliament (through work, interest, or experience) or a novice user.
Note – you can see what we’re currently working on here. Some changes were obvious – for example, we’re improving MPs’ individual pages.
What is TheyWorkForYou for?
TheyWorkForYou has been running since 2004. We know why we launched it, though the way you use it may be totally different – and if so, we want to hear about that. Its aim is to give a window into Parliament, for everyone, but including people who may never have previously thought that parliamentary proceedings had anything to do with them.
TheyWorkForYou does a lot of things. It lets you find out who your MP is – if you don’t know – and then it tells you all about them.
It publishes the written record of debates in Parliament, and lets you search it, and link to it easily.
It allows you to set up alerts, so you get an email every your chosen words or phrases are mentioned in Parliament – or every time a particular person speaks.
It publishes future business (there are alerts for that, too), written answers, Public Bill Committees, and more.
So, it does a lot – but we know that it still doesn’t do everything our users request, and it doesn’t neccessarily do everything in the way that they want, either. Some changes are obvious, and we’re working on them – right now, for example, we are improving individual MPs’ pages. But we want your thoughts too.
Photo by Lindsay Bremner (CC)
Since its launch in 2005, WriteToThem has always covered all parts of the United Kingdom, and the Northern Ireland Assembly was the first body added to TheyWorkForYou after the UK Parliament, in late 2006. So whilst we certainly have not ignored Northern Ireland, it had always been an irritant of mine (and a cause of infrequent emails) that FixMyStreet only covered Great Britain.
This was due to the way it had originally been funded and set up, but those issues were in the past, due to a myriad of changes both internal and external, and it was now more a case of being able to find the resources to implement the necessary work. Late last year, mySociety worked with Channel 4 on the website for their series of programmes on The Great British Property Scandal. This used, in part, code similar to FixMyStreet to let people report empty homes, and it was required to work in all parts of the UK. So as part of that process, code was written or generalised that let aspects of FixMyStreet like the maps and place name lookup work for Northern Ireland locations.
It’s taken a few months since then to allocate the time, but we’ve now been able to take the code written back then, add various other bits, and incorporate it into FixMyStreet – which now covers the 26 councils of Northern Ireland, and the central Roads Service. Issues such as potholes, graffiti, and broken street lighting can be reported to Antrim or Newry and Mourne as easily as Aberdeen or Wyre Forest, and just as in the rest of the UK you can sign up for alerts based around your location or to your council.
Just like the famous Yellow Pages ad, FixMyTransport is not only there for the nasty things in life.
Quite late in FixMyTransport’s development, we realised that praise is a form of feedback and that this channel ought to be open to our users, too – that’s why on every page for a stop or a route, you’ll see a stonking great button inviting you to ‘say something nice’ – here’s an example: a station manager giving the type of good service that gladdens the heart.
There’s also a little love for Pulborough station, nice words for Brighton’s well-informed station staff, an example of a very accommodating bus driver, and a comment that makes me want to take the 37 bus to Clapham Junction right now.
We haven’t really stressed this side of FixMyTransport, so we’re really glad to see people finding it for themselves and putting it to good use. If you encounter particularly excellent service on your next journey, and you want to spread the love too, just remember the big blue button. We’re sure the operators will be very glad to hear from you.
Last week, FixMyStreet gained a number of new features that we hope you will find useful.
Firstly, we’ve thrown away our old maps and replaced them with new, shiny, zoomable maps. This should make it easier for people to find and report problems, especially in sparser locations. We’re using the OS StreetView layer (hosted internally) when zoomed in, reverting to Bing Maps’ Ordnance Survey layer when zoomed out, as we felt this provided the best combination for reporting problems. In urban areas, you can still see individual houses, whilst in more rural areas the map with footpaths and other such features is probably of more use. FixMyStreet tries to guess initially which map would be most appropriate based upon population density, meaning a search for Stroud looks a bit different from that for Birmingham.
OpenStreetMap fans, don’t worry – as part of our mapping technology upgrade, you can now use osm.fixmystreet.com to access your favourite mapping instead.
Secondly, we now have user accounts. We’ve rolled these out alongside our current system of email confirmation, and it’s up to you which you use when reporting a problem or leaving an update. This means that those who come to the site one time only to report a pothole can continue to do so quickly, but have the option of an account if they want. Having an account means you no longer have to confirm reports and updates by email, and you have access to a page listing all the reports you’ve made through FixMyStreet, and showing these reports on a (obviously new and shiny) summary map.
Other improvements include a much nicer All Reports section, so you can see all reports to Adur District Council on a map, paginated and with the boundary of the council marked – and individual wards of councils now each have their own pages too.
I’ll follow up this post with another, more technical, look at the maps and how they work, for anyone who’s interested 🙂
Have you used WhatDoTheyKnow to make a Freedom of Information request?
If so, you can now add your photograph to the site, and some text on your user page about what you’re researching. This can include links to your blog, campaign page or twitter feed.
To add this to your profile, first log into WhatDoTheyKnow, and go to your user page by choosing “my requests”.
There are then links to add a profile photo and/or set some text about you, and what you’re using FOI for.
I’d go and do it while I remember – it will help you and others find and understand each other, hopefully leading to that little bit more collaborative research!
mySociety is lucky enough to have a number of small donors who give us monthly donations, normally ranging from £5-£20 (if you like our work and want to support us, please do join them!). Today we’re announcing a change to TheyWorkForYou which is supported by these donations.
One of the most popular features on TheyWorkForYou is the vote analysis – the bit that tells you that your MP “voted strongly against introducing a smoking ban” and so on. These voting analyses cut through a massive wall of parliamentary opacity whilst still allowing visitors to examine the details first hand. Despite each analysis resulting in just a single line on TheyWorkForYou, each one is rather time-consuming to construct, and TheyWorkForYou has not updated them as much as our users deserve.
Thanks to our small donors we’ve now been able to commission two part time researchers, Marcus Fergusson and Stephen Young, to help add new vote analyses more regularly. We’re pleased to say that we’ve just rolled out the first new policies, covering issues relating to schools, inquests and the House of Lords. We aim to add a couple of new vote analyses a month for the foreseeable future.
We take the business of authoring analyses that are scrupulously fair and neutrally worded extremely seriously. To this end we have replaced our previously ad-hoc approach with a newly instituted process designed to ensure the maximum rigour and balance, and to ensure we focus on issues which MPs thought were important even if they were not so well covered by the media.
The new process for analysing MP’s positions works like this:
- A list of votes in the current Parliament, ordered with the highest turnout at the top, is taken as a starting point. The turnout figure used is corrected to account for party abstentions.
- mySociety’s researchers work down that list writing explanations in easy to read terms describing what the vote was about; they also identify other related votes on the same issue and research those too.
- A “policy position on the issue” is then chosen against which MP’s votes are compared to determine to what extent they agree or disagree with it. Policy positions are written to be intelligible and interesting to a wide range of users and in such a way that votes to change the status quo are ultimately described on TheyWorkForYou MP pages as votes “for” that change.
We hope you find these analyses useful. Thanks to Richard Taylor for his divisions list and help with this post.
When a bit of government forwards or attaches emails using Outlook, they get sent using a special, strange Microsoft email format. Up until now, WhatDoTheyKnow couldn’t decode it. You’d just see a weird attachment on the response to your Freedom of Information request, and probably not be able to do anything with it.
He then told us about it, and I merged his changes into the main WhatDoTheyKnow code, tested them out on my laptop, then made them live. It all work perfectly first time. Peter even added the new dependency on vpim to WhatDoTheyKnow conf/packages.
Now if you go to an Outlook attachment on WhatDoTheyKnow,such as this one you’ll just see the files, and be able to download them, and view them as HTML as normal. They’ll also get indexed by the search (although I need to do a rebuild for that for it to work with old requests).
If you want to have a go making an improvement to a mySociety site, you can get the code for most of them from our github repositories. For some sites, there’s an INSTALL.txt file explaining how to get a development environment set up. Let us know if you do anything – even incremental improvements to installation instructions are really useful. And new, useful, features like Peter’s are even more so.