Alaveteli is our platform for running Freedom of Information websites — it underpins WhatDoTheyKnow as well as many other sites around the world. It’s made up of many interconnecting elements, but a key part is the database of public bodies.

On WhatDoTheyKnow, we list over 42,000 public bodies that are subject to FOI and EIR – possibly one of the largest databases of public bodies in the country. Along with their name, we record information like their FOI request email address, publication scheme and disclosure log URLs, and we categorise them so that they’re easier to browse and make requests to.

We often want to add additional insight to help citizens understand more about the public body. This can vary from body to body, from describing the information they hold, what they don’t hold, guidance around how to challenge their poor FOI-handling practices or why we list them when they’re not currently subject to FOI.

We do this via what we call “notes” – a free-text field per authority that admins can update with whatever information is useful for the particular body.

We often find we need to add or update the same note for a group of authorities — for example, we added the same note to all Business Improvement Districts to explain our reasons for listing them. 

This can be a major challenge. We list nearly 300 BIDs, and updating each manually, one at a time, would be a several-hour ordeal. An alternative is for our developers to write a quick one-off script to update the text, but that comes with a coordination cost, and can be tricky to work around other text that may be present in the free text field that we want to preserve.

One of the behind-the-scenes ways we manage the categorisation of authorities is through tags. Many of our other records have the ability to add tags too.

This led to the thought that it would be great to be able to apply a note based on a tag, rather than only having one note field per authority. This would allow us to more quickly add useful information to groups of authorities.

Another issue we wanted to solve was to be able to add notes to content other than public bodies, like FOI requests for specific types of information and our category lists. In particular we wanted to be able to celebrate requests that led to particularly useful public interest information being released or having wider impact, and also to add clarification around requests that may be misinterpreted.

We’ve received funding from the Joseph Rowntree Charitable Trust to help support marginalised communities to make more well-formed requests, and to more effectively use the information they obtain to engage with and influence authorities in a way that contributes to fairer decision-making. Thanks to this, we were able to add this underlying capability, which we can use to continue to help users more easily understand how to navigate the complexities of public authorities and what information they hold.

The first step was to extract the existing notes to a new generic Note model that could be attached to any other record. Ruby on Rails makes this easy through its polymorphic associations.

We then needed an interface to allow admins to list and search tags for each type of record that has them, and browse all records that are tagged with a particular tag.

As well as it being generally useful to be able to browse our database through our tags, this gave us a place to add our new functionality for adding a note based on a tag.

While Rails’ polymorphic associations make creating a link between one record and another really easy, they don’t cover creating the link through a free-text tag. Fortunately, it took nothing more than a few lines of code to link up notes that are directly associated with a single record, and notes that are applied via a tag.

Now our public bodies can have notes applied to them and only them, and also notes applied to several authorities based on their tag.

In this example, the Environment Agency has some specific notes about it, but since it’s also a local lighthouse authority, we can apply the relevant notes simply by tagging the authority record and the notes for local lighthouse authorities will automatically get applied. Magic!

We’ve also applied this pattern to information requests. Now, we can add notes to individual requests, like this example that points to another source for obtaining the information…

…or applied via a tag so that we can point citizens interested in the climate response to the authority’s CAPE page.

So far we’ve created 46 notes that get applied via a tag. These notes are applicable to 7,998 requests and 15,283 authorities. Using a rough guess of 30 seconds to manually apply a note to a record, it would take 11,640.5 minutes – 8 full days, or 25 working days – to do so for each of these requests and authorities. This just wouldn’t have been possible before.

This new feature unlocks a whole new avenue for us to support citizens and users that we just wouldn’t have had capacity for otherwise.

—

Image: Keila Hötzel

Are you investigating, researching or gathering large quantities of data through Freedom of Information requests? Perhaps you’re a journalist, academic or NGO. We’re looking people based in the UK who’d like to try out our new ‘Projects’ feature for WhatDoTheyKnow Pro.

Projects allows you to crowdsource the extraction of data from multiple (or batch) FOI requests made to multiple authorities. You can set up a project with a brief description of what it is and what you are hoping to achieve, and some tasks that volunteers can complete to help you with this aim (like categorising responses, or answering questions about the data released).

Once that’s done, you can set it up to invite volunteers, who can help you to extract all the information you need from the released responses.

You’ll be able to download your volunteers’ input as a spreadsheet, meaning analysis of the data is much quicker and easier — so you can get on with the task of forming conclusions and writing up your findings.

What we’ll need from you

Projects is still in its nascent stage, so we need feedback from our testers. This will help us improve the service and tailor it to users’ needs, based on real life use cases.

Right now, we handle the setup and importing of the requests you want to work on manually (that is, our developers have to do it) — but we’re working on improving this aspect, and your feedback will be crucial in shaping the direction our development takes. We’re also looking for general comments, once you’ve used the service, on what’s useful and what’s missing; what you tried to do but couldn’t, and what made things easier for you.

If this sounds interesting, please get in touch at pro_team@whatdotheyknow.com. We look forward to hearing from you!

—

Image: Jessica Lee

We’ve added a new functionality to the Alaveteli Pro codebase, allowing you to download a zip file containing all correspondence and attachments from a batch, and a spreadsheet (csv) to show the progress status of every request.

Alaveteli Pro is our tool for professional users of Freedom of Information. If you’re UK-based, you’re probably most familiar with our local iteration WhatDoTheyKnow Pro — but don’t worry: when we talk about improvements to Alaveteli Pro, you can be sure they’re also part of the WhatDoTheyKnow toolkit.

How to export

You’ll find these tools at the foot of the batch container in the requests list.

Why data exports?

Of course, we like to think Alaveteli Pro is a useful tool in its own right: there’s a lot you can do within the Pro interface, and it was built specifically to help you keep track of all your FOI activity in one place.

But sometimes users want to use external tools – either because they’re just more familiar with them, or because they want to do something beyond the functionality we offer.

Now there’s a simple way to get data out of Alaveteli, allowing you to analyse it with the tools of your choice, or perhaps send a progress report to a supervisor or editor.

It’s part of a programme of work to support cross border journalism between European organisations, supported by Adessium Foundation, allowing us to refine and improve the codebase for the benefit of all Pro users.

The technical bit

Those with a bit of coding knowledge may be interested to hear how we approached the zip download functionality. mySociety developer Graeme explains:

“With batch requests potentially going to as many as 500 different authorities, each request can receive several responses and attachments in return.

“All these emails and files mean that compiling the zip for download could be a lengthy job and would normally cause the request to time out. So for this new feature we’re utilising file streaming to send chunks of the zip as they become available.

“This means that the zip starts downloading immediately and you don’t have to sit watching and wondering whether anything is happening – you can see more and more data being transmitted.”

We hope you find this new feature useful. Please do let us know how you’re using it and any feedback you may have.

—

Image: Startup Stock Photos

 

We’ve recently made a small change to WhatDoTheyKnow. Now, if your request has resulted in a news story, you can add the link as a ‘citation’.

If you’re a journalist or a campaigner, we hope this is a useful way to give your stories some more readership (not to mention a nice inbound link from a high-ranked site for your search engine ratings).

And if you’re simply a citizen whose request was picked up by the press, there’s now a way to share how the information you’ve obtained fits into the news agenda.

There’s a benefit for the wider transparency community, too. We think that, in aggregate, these links will serve to show others just what a simple FOI request can do.

Requests often start off as nothing more than an inkling or a nagging question, but there’s always the chance that one of them will reveal important or interesting information, hitting the news, reaching a wider readership and — who knows?  — maybe even changing hearts and minds. For big stories, it will be good to be able to create a permanent record of where it all began.

More broadly, when you use this feature you’ll be helping us to understand what sort of impact the site is having, too. We’re always keen to spot news stories based on WhatDoTheyKnow requests, but papers don’t always cite a source or link back to the site, meaning that our monitoring is often dependent on a manual search where stories look like they might have originated with one of our users.

How to add a citation

You’ll find the feature in the right hand column of your request page. Just click on ‘Let us know’:

…and paste the URL in:

Here’s how it will show up on the request page, as seen on the first request to gain a citation, which informed a story about electoral letters sent in error:

If there’s more than one story, you can click ‘New Citation’ to add another one.

The way we’ve set this feature up, WhatDoTheyKnow users can add a citation to any of their own requests — but if you spot a news story that’s linked to a request that isn’t yours, please do contact the WhatDotheyKnow team.

They’ll assess it and input it if they find it to be valid. Our aim here is, of course, to prevent spammers from adding irrelevant links to the site.

Users of WhatDoTheyKnow Pro, on the other hand, have the ability to add citations to any request.

This work is one of the ongoing improvements that we’re working on thanks to a grant from Adessium.

—

Image: Kaboom Pics

Today we’ve added twelve new topics to the voting pages for every MP on TheyWorkForYou. Covering important areas such as membership of the EU, mass surveillance of data, and military action against Daesh/ISIL, these voting lines will help give an even better at-a-glance picture of what your MP stands for, and how that was reflected in Parliament.

Check your own MP’s votes on these topics by inputting your postcode on the TheyWorkForYou homepage:

and then clicking on the ‘voting record’ tab:

You’ll see all your MP’s voting lines laid out in several topic areas on a single page:

From there, it’s easier to explore further by clicking ‘show votes’, which will take you to a page listing every vote taken into consideration when calculating the MP’s stance.

For a full explanation on the methodology behind these pages — a combination of manual and automated inputs — see our previous blog post.

How do we choose which topics to include? It’s all driven by what MPs have voted on in Parliament, and consequently how much data there is to draw from in any specific subject area (there are also a few ‘topics in waiting’: areas we know we want to cover, but where there need to be some more votes before we have sufficient data to present meaningful stances).

New topics in full

The new voting lines we’ve added are:

• In Constitutional Reform
  • Reducing the voting age
• In Home Affairs
  • Mass surveillance of people’s communications and activities
  • Merging police and fire services under Police and Crime Commissioners
• In Transport (a new topic area)
  • High speed rail network
  • Greater public control of bus services
  • Publicly-owned railways
• In Foreign Policy & Defence
  • UK membership of the EU
  • The right to remain for EU nationals
  • Military action against the group known as ISIS, ISIL or Daesh
• In Taxation & Employment
  • The reduction of Capital Gains Tax
• In Housing (a new topic area)
  • Secure tenancies for life
  • Charging a market rent to high earners renting a council home

We hope this helps you see, even more clearly, the effect your MP has had on a broad range of national and international issues.

Don’t forget that we also offer the chance to discuss these votes (or any other issue) with your MP. Just click the ‘send a message’ button at the top of the page to go to WriteToThem.com, where you can compose a message to your representatives quickly and easily.

—

Image: © UK Parliament/Jessica Taylor

When you send a Freedom of Information request, a clock starts ticking. Here in the UK, public authorities are bound by law to answer a request “promptly, and in any case within 20 working days” — but of course they can only respond if they’ve received the request.

And, while email is generally reliable, we’re all familiar with the occasional mishaps it can bring: mailboxes that are full, accounts that have been closed down, or messages being returned because they look too much like spam.

As Liz’s post yesterday mentioned, our Freedom of Information platform Alaveteli now includes a new feature which verifies that your message actually has been delivered.

Sign here please

Email works a bit like signed-for physical mail. When a letter is delivered to a recipient they either sign to say they’ve received the letter, or the mail company records that there was no-one available to accept the delivery. This lets the mail company keep the sender up to date with where their letter is. Mail servers do the same — the recipient server sends a confirmation that a particular email has been received, or an error code is reported by your mail server if there’s a problem delivering the email.

Like physical mail, we can only verify that the message has been accepted at the destination address. It’s then under the recipient’s control to get it to the right person at that address, a bit like a receptionist receiving a letter for an employee 10 floors above. We think that if an authority’s mail server confirms that one of our emails has been delivered, it’s their responsibility to ensure it reaches the correct people to be able to answer your FOI request.

Proof of receipt

Look at the header of any request on WhatDoTheyKnow, and within 24 hours, in most cases you’ll now see a small green ‘delivered’ confirmation:

 

Most users can click on this to see further confirmation:

But if you’re the owner of the request, when you click on the green ‘delivered’ link, you’ll see information from the mail logs as the message passed through our server. If there’s ever a query about whether or not a message was delivered, you can hand these on to the authority to help them analyse any issues.

On the rare occasions that something goes wrong, here’s what users will see instead:

– and if it’s your own request, again, you’ll have access to the mail logs.

Small but mighty

This feature might look small, but there’s a lot of thinking behind it — just check the length of the trail on Github, our ticketing system. Anyone will be able to understand the amount of discussion and problem-solving that went into the addition of this small green tick, while the more technically-minded may also find it interesting to see the coding solutions as they unfolded.

This small green tick also gives users something rather powerful: proof that their request was received by the authority’s mail server at a specific time, should that be disputed.

The suggestion for this feature came initially from one of the WhatDoTheyKnow volunteers. It took some time to implement, but we’re pleased to say that it has now been made available for all Alaveteli sites in release 0.25.0.0.

—
Image: Pacific Northwest Agricultural Safety and Health Center (CC)

Over the last few days, we’ve looked at various ways to use SayIt. This is the final post in the series for now, and it’s highlighting a feature rather than pointing out a use.

SayIt is still in active development – that’s to say, we’re working on it and adding new features all the time.

One of the significant features that recently went live is that SayIt sites can now have more than one editor.

Who needs to collaborate?

We can think of lots of reasons to collaborate:

• If you’re minuting meetings, you can get others to share the burden of recording, proofing and editing.
• If you are collecting together statements about a wide topic, or expect your project to run over some length of time, you can call in people to help.
• If you’re running a campaign, you can ask your supporters to help you crowd-source relevant material.
• If you are part of an oral history archive, or similar community project, you can share access with others in your group.

…and we’re sure that there are many other uses too.

How to collaborate

You’ll need a SayIt site.

If you haven’t started your SayIt site yet, just go to this page to start one up. If you’ve got one already, log in.

Then invite your collaborators. Just click on the ‘invite friends to help’ button.

Put in the email address of a friend or colleague. We’ll send them an invitation to help add and edit speeches and sections.

Repeat as required. You can invite as many people as you like.

Ways to use SayIt

Check out our series of step-by-step blog posts to explain how to:

• Use SayIt to publish transcripts of meetings
• Make a collection of quotes
• Store research interviews

Image credit: Leaf Cutter Ants by Jon Pinder (CC)

Fifty years ago, in 1964, the causal link between smoking and lung cancer was confirmed by the Surgeon General in the US.

That year saw many debates in Parliament on topics that have since become very familiar: the question of whether the tax on cigarettes should be raised; whether cigarettes should be advertised on television, whether smoking should be allowed in public places, and whether warnings should be printed on packets.

Rich and fascinating stuff for any social historian – and it’s all on TheyWorkForYou.com.

Hansard is an archive

Hansard, the official record of Parliament, is a huge historic archive, and whatever your sphere of interest, it is bound to have been debated at some point.

Browsing through past debates is a fascinating way of learning what the nation was feeling: worries, celebrations, causes for sorrow – all are recorded here.

How to use TheyWorkForYou to browse historic debates

TheyWorkForYou contains masses of historic information: House of Commons debates back to 1935, for example, and details of MPs going back to around 1806. You can see exactly what the site covers here.

There are various ways to search or browse the content. Start with the search box on the homepage – it looks like this:

You can do a simple search right from this page, or choose ‘more options’ below the search box to refine your search.

We’ll look at those advanced options later, but let’s see what happens when you input a simple search term like ‘smoking’.

Here (above) are my search results, with my keyword helpfully highlighted.

By default, search results are presented in reverse chronological order, with the most recent results first. If you are particularly interested in historical mentions, you may wish to see the older mentions first.

That’s easy – just click on the word ‘oldest’ after ‘sorted by date’:

You’ll notice a few other options here:

• Sort by relevance orders your results with the most relevant ones first, as discerned by our search engine. This will give you those speeches with the most mentions of your keyword ahead of those where it is only mentioned once or twice.
• Show use by person displays a list of people who have mentioned your keyword, with the most frequent users at the top. This can be fascinating for games such as “who has apologised the most?” or “who has mentioned kittens most often?”

Click through any of the names, and you’ll see all the speeches where that person mentioned your keyword.

Advanced search

That’s a good start – but what if there are too many search results, and you need some way to refine them? You’ll notice from my screenshots above that there are (at the time of writing) over 10,000 mentions of smoking.

That’s where Advanced Search comes in. You can access it from a few places:

• The ‘more options’ link right next to the search box on search results pages (see image below)

• The ‘more options’ link below the search box on the homepage (see image below)

• Or just navigate directly to our dedicated Advanced Search page (see image below)

Whichever way you arrive at it, the Advanced Search page helps you really get to the content you’re interested in.

The pink box on the right gives you some tips for effective searching.

For example, just as with Google, you can search for exact phrases by putting your search term within quotation marks. Otherwise, your results will contain every speech where all your words are mentioned, even if they’re not together. For phrases like “high street”, this could make a real difference.

Even if you are only searching for a single word, you can put it in quotation marks to restrict the use of ‘stemming’ – so, for example, a search for the word house will also return results containing houses, housing and housed, unless you put it in quotation marks.

You can exclude words too: this can be useful for minimising the number of irrelevant results. So, for example, you might want to find information about the town of Barking, but find that many of your results are debates about dogs. Simply enter the search term “barking” -dogs. The minus sign excludes the word from your search.

In the main body of the page, you’ll also see options to restrict your search to within certain dates, or a specific speaker, or a department, section (eg Scottish Parliament or Northern Ireland Assembly) and even political party.

Get stuck in

The best way to see what you can find is to dig in and give it a go. If your search doesn’t work for you the first time, you can always refine it until it does.

Let us know if you find anything interesting!

Image: National Archives (No Known Restrictions)

mySociety is an organisation with many priorities, and they often compete for attention. Right now, we have some time and budget to lavish on TheyWorkForYou. We need your input to help us understand what development we should prioritise for the site.

Note: if you don’t know much about TheyWorkForYou, your opinion still counts! See the foot of this blog post for an overview of the site and its aims.

 Some suggested improvements

Below is a list of improvements that other users have suggested, or that we think are desirable. Which improvements would you most like to see – from this list, or based on your own needs?

1. Easier sharing via social media If you see a debate you want to share with your Twitter or Facebook buddies, all you’d have to do is click a button. More details
2. When your MP voted Letting you know, via email alerts, when your MP has taken part in a vote. More details
3. Option to search just headings At the moment, search covers all content of debates, including everything anyone said. This option would allow you to only search headings, meaning that you could be sure the results were entirely focused on your topic. More details
4. Tweeting debate headings or future business A Twitter account which would tweet, and link to, every debate in Parliament, or upcoming events. More details
5. Signposting of big ‘events’ such as the Budget These are not always easy to find if you don’t know your way around, so we’d make sure the big events were always trumpeted on the site. More details

Great ideas, or utter bunkum? Let us know. You can give us feedback via any of the following methods:

1. Leave a comment under this blog post;

2. Tweet at us on @TheyworkForYou, comment on our Facebook page, or drop us a mail;

3. If you’d like to see the whole list of suggestions and issues, you can do so on our development list at Github (and the ‘more details’ links in the list above go to the issues on there). Note that anyone is welcome to add comments to these issues, or even to create your own (please search first to make sure you’re not duplicating an existing issue). Github may look complex, but it’s easy enough to use – you just need to set up a user account here.

We’re keen to understand whether we’re serving all kinds of users, so it’d also be helpful if you could tell us whether you consider yourself to be someone who knows a bit about Parliament (through work, interest, or experience) or a novice user.

Note – you can see what we’re currently working on here. Some changes were obvious – for example, we’re improving MPs’ individual pages.

What is TheyWorkForYou for?

TheyWorkForYou has been running since 2004. We know why we launched it, though the way you use it may be totally different – and if so, we want to hear about that. Its aim is to give a window into Parliament, for everyone, but including people who may never have previously thought that parliamentary proceedings had anything to do with them.

TheyWorkForYou does a lot of things. It lets you find out who your MP is – if you don’t know – and then it tells you all about them.

It publishes the written record of debates in Parliament, and lets you search it, and link to it easily.

It allows you to set up alerts, so you get an email every your chosen words or phrases are mentioned in Parliament – or every time a particular person speaks.

It publishes future business (there are alerts for that, too), written answers, Public Bill Committees, and more.

So, it does a lot – but we know that it still doesn’t do everything our users request, and it doesn’t neccessarily do everything in the way that they want, either. Some changes are obvious, and we’re working on them – right now, for example, we are improving individual MPs’ pages. But we want your thoughts too.

Photo by Lindsay Bremner (CC)

Since its launch in 2005, WriteToThem has always covered all parts of the United Kingdom, and the Northern Ireland Assembly was the first body added to TheyWorkForYou after the UK Parliament, in late 2006. So whilst we certainly have not ignored Northern Ireland, it had always been an irritant of mine (and a cause of infrequent emails) that FixMyStreet only covered Great Britain.

This was due to the way it had originally been funded and set up, but those issues were in the past, due to a myriad of changes both internal and external, and it was now more a case of being able to find the resources to implement the necessary work. Late last year, mySociety worked with Channel 4 on the website for their series of programmes on The Great British Property Scandal. This used, in part, code similar to FixMyStreet to let people report empty homes, and it was required to work in all parts of the UK. So as part of that process, code was written or generalised that let aspects of FixMyStreet like the maps and place name lookup work for Northern Ireland locations.

It’s taken a few months since then to allocate the time, but we’ve now been able to take the code written back then, add various other bits, and incorporate it into FixMyStreet – which now covers the 26 councils of Northern Ireland, and the central Roads Service. Issues such as potholes, graffiti, and broken street lighting can be reported to Antrim or Newry and Mourne as easily as Aberdeen or Wyre Forest, and just as in the rest of the UK you can sign up for alerts based around your location or to your council.