1. Clearing the map

    When you consider that FixMyStreet has been running for over a decade, it’s not really surprising that the maps in some areas are a little over-crowded with pins.

    That can be a problem for anyone trying to make a new report — even when you zoom right in, we were beginning to find that in some very congested areas, it was difficult to place a new pin without clicking on an existing one.

    A FixMyStreet screenshot, crowded with pins

    We’ve tried to remedy this in various ways in the past. For a while we only displayed newer reports by default, a decision which we discarded when we brought in pagination, allowing users to click through batches of reports rather than seeing them all in one long list on a single page.

    For some time now we’ve also provided the option to hide the pins completely, via this button both on the desktop and app versions:

    location of hide pins button on FixMyStreet

    And there’s also a ‘hide pins’ option at the foot of the map:

    location of hide pins button on FixMyStreet

    But even so, arriving at a map absolutely covered in pins and having to look around for that button doesn’t exactly seem like a nice, smooth user journey, so we’ve revisited the matter.

    Why not just delete the old reports?

    We’ve always had a policy of keeping every report live on FixMyStreet (unless it’s reported to us as abusive, or its maker contacts us to ask us to remove it — and even in this latter case we’d prefer to retain the content of the report while anonymising it).

    This is because the reports made to councils build up to create an invaluable archive of the issues that various regions of the country face, through time.

    The historic collection of reports allows planners to understand recurring or seasonal problems; and researchers use this data as well, to get insights into all sorts of issues. For examples, see Réka Solymosi’s presentation at TICTeC on using FixMyStreet data to understand what counts as ‘disorder’ in the environment, or mySociety’s own research on why some areas of the country report on FixMyStreet more than others.

    And so here’s what we’ve done

    • When you visit a map page on the main FixMyStreet site, by default, you’ll again only see reports that are less than six months old, and that are still open.

    A report remains ‘open’ until the council marks it as ‘closed’, or a user or the council marks it as ‘fixed’. ‘Closed’ means that the council doesn’t intend to do further work on the issue, which can be for reasons such as the issue not falling within their responsibilities or because it is part of their regular maintenance schedule and will be seen to in time.

    • You can still opt to see closed and fixed reports by selecting from the dropdown at the top of the list:Dropdown options at the top of a FixMyStreet map page
    • And you can also still see reports older than six months by clicking the checkbox:
    • The two filters work together, giving you the options of displaying:
      • Open reports less than six months old (the default)
      • Open reports of any age
      • All reports less than six months old
      • All reports of any age
      • Any combination of open/closed/fixed reports less than six months old
      • Any combination of open/closed/fixed reports of any age

    To keep things simpler for app users, the display there is set to only show newer, open reports, so if you want the full range of options, you’ll need to switch to viewing the site on a desktop.

    Additionally, reports that have been closed for six months without any update being made will now no longer allow updates. If you need to update an issue that falls into this category, we recommend starting a new report (possibly linking to the old one for reference if it provides useful information for the council).

    But you might not see this everywhere

    Some councils use FixMyStreet Pro as their own fault-reporting software. These councils can opt whether or not to adopt these defaults, so your experience may be slightly different when visiting FixMyStreet via your local council’s own site.

    We think that we’ve arrived at a more intuitive solution than those we tried before — and we hope that these options will suit everyone, whether you’re a user in a hurry coming to make a quick report, or someone who’d like to see a more in-depth history of the area. Give it a go, and then let us know your thoughts.

     


    Banner image: Jim Tilley (CC by-nc-nd/2.0)

  2. Alaveteli Pro: batch requests, a news story, and going international

    Jenna Corderoy, Alaveteli Professional Advocate, brings us an update on the project.

    Since our last blog post on Alaveteli Professional — our Freedom of Information toolkit for journalists, campaigners and activists — there have been a few exciting developments.

    Batch work

    The batch request feature is coming along nicely: this will allow users of the service to send one Freedom of Information request to multiple authorities and help them to easily manage large volumes of responses.

    We’re going to be working with a small group of our beta testers to develop this feature and make sure we release it in a useful and responsible form (click here to apply as a beta tester and get a year’s free access to WhatDoTheyKnowPro, the UK version of the service).

    Getting results

    We’ve been pleased to see the first news story to emerge as the result of a request made through WhatDoTheyKnowPro: a response to the Foreign and Commonwealth Office showed which are the countries where UK holidaymakers are most likely to get arrested. The full list was covered in the Birmingham Mail.

    Further afield

    But we also have plans for this Freedom of Information toolkit to go international: Alaveteli Professional will be a bolt-on option for anyone already running an FOI site on our software platform Alaveteli.

    In April, mySociety team members traveled to roll out the first such project, with Info Pro Všechny, the Czech Republic’s Alaveteli site.

    We were able to introduce beta users to the features we’ve been developing, such as the ability to keep requests private until the story has been published.

    While in the Czech Republic, we held a roundtable discussion with journalists and campaigners, swapping Freedom of Information battle stories and sharing tips and tricks for getting the best results when submitting requests for information, as well as experiences of filing requests to European Union institutions.

    mySociety was also invited to give a talk to student journalists based in Olomouc about Info Pro Všechny and Alaveteli Professional in general, discussing success stories generated from our Freedom of Information sites from around the world.

    Next up

    We’re currently working on subscription options, which will allow us to officially launch WhatDoTheyKnowPro as a paid-for service in the UK, and later in the year, we plan to introduce the Pro toolkit to the Belgian Alaveteli site Transparencia.be, which has been making a splash in Belgian politics.

    Photo by Amador Loureiro (Unsplash)

  3. Visualising transparency, for AllTrials

    You may know Dr Ben Goldacre from his ‘Bad Science’ and ‘Bad Pharma’ campaigns, which fight misinformation around medicine. Ben has just launched his latest project, the AllTrials Transparency Index — and mySociety helped with the website side of things.

    The AllTrials campaign focuses around the fact that a shockingly large proportion of clinical trials do not have their results publicly published.

    Not only does this devalue the time, goodwill and even potential risk put in by participants, but there are also issues around bias. Those trials published tend to be the ones which show positive results: if that doesn’t sound like such a terrible situation to you, try playing this game from the Economist magazine, which graphically depicts the problems with skewed coverage. At worst, such selective publication can be dangerous, or lead to poor choices from bodies making medical purchasing decisions.

    Transparency and data visualisation are two areas where mySociety has a long history, and so it came to be that we fashioned the deceptively simple AllTrials Transparency Index site, on which anyone can browse the transparency index of the world’s major drug companies, and dive in deeper to the data to see how it was compiled. The source data is free for others to download too, so anyone can integrate it into other projects.

    All Trials transparency ranking

    AllTrials are also tracking whether companies register new trials:

    AllTrials graph, showing registered trials

     

    That’s the part that anyone can understand — and now, notes for the more technically-inclined who may be wondering how we took the data on each drugs company and presented it in a way that can be quickly and easily taken in.

    This is an ongoing campaign with a commitment to future audits, so we wanted to make it easy for the AllTrials team to update the site and republish the source data each time they do.

    It’s a static Jekyll site. We wrote a custom plug-in to parse a CSV and produce a page for each company within that CSV, as well as creating some summary data that feeds into the graphs on the front page.

    This data is then pulled from the CSV, and D3 is employed to build the graphs and insert them into the generated pages.

    The end result is a site that looks good and which can automatically update whenever the underlying data changes. We hope we’ll have played a small part in helping to ensure that it does — and for the better.

    All Trials company profile


    Header image: Komu News (CC by/2.0)

  4. WhatDoTheyKnow Pro: ready for beta users

    If you’ve been watching the progress of our Freedom of Information toolkit for journalists, campaigners and activists, you might be interested to know that we are now accepting applications for pre-launch access.

    Successful applicants will have an early opportunity to put the service through its paces. WhatDoTheyKnowPro will launch as a paid-for service, but as a beta tester you’ll have up to a year’s access for free, because we’re keen to see how you’ll use it — and to hear your feedback on which features are useful.

    What’s WhatDoTheyKnowPro?

    WhatDoTheyKnowPro is our first launch of Alaveteli Professional, accessed via the WhatDoTheyKnow website and specifically for UK users who utilise FOI in their work or campaigning.

    It’s the first instance of the service we plan to make available to other Freedom of Information sites running around the world on our Alaveteli platform.

    What you’ll get

    Since we last caught up with Alaveteli Professional, we’ve made really concrete progress with several of the features that, at the time of that blog post, were just entries on our long to-do list.

    Here’s what users will access:

    • The ability to keep requests private until your story has been published
    • A powerful private dashboard that helps you track and manage your FOI projects
    • A super-smart to-do list that makes it easier to follow the progress of your requests
    • Action alerts that nudge you when it’s time to take the next step in a request

     

    screenshot-requests-78dd427b981787204f3ddfe4eda11cb1939e93263d29398d309b48f14cdb1be2  screenshot-dashboard-34a9936fd33b54729b033c322bc1bc360116ae85b2d4fec5182dd22f58283745
     

    Batch benefits

    And very soon, we’ll also be carefully rolling out the batch request features, which will allow you to:

    • Make one request to multiple authorities
    • Manage large volumes of responses and easily keep track of the status of each request
    • Get regular updates as the responses come in, without overwhelming your inbox

     
    screenshot-batch-list-7f71f614b844db3686cc6731d7725aec10e244dc327f88b5c87c02382b6f6d71 screenshot-batch-selection-d59dda9d21302f321502c8033a254a27e6cb3f2579c4ac16c5f8006eed2a43ac

     

    We’re excited about the batch feature in particular, and we know that many of our prospective users are, too. At the same time, we’ve heard some concerns that it might encourage a scattergun approach that wastes authorities’ time.

    Our planned development will ensure that people use this feature responsibly, and, consequently, get the best returns from it. This will include a prompt to send a smaller batch initially, so that the remainder of the requests can be refined based on the quality of information that is returned — there’s nothing worse that asking every council in the country for information and then realising that you’ve worded your question in a way that means you can’t use the resulting data!

    At the moment, batch requests can’t be made on WhatDoTheyKnow without help from the site administrators. We’re aware that many journalists and activists already make many batch requests outside WhatDoTheyKnow for this very reason. We’d like more of these requests to be released in public (we estimate that around 15% of UK FOI requests are made via our site): so by including this capability in WhatDoTheyKnowPro, we hope we’ll not only be steering people to use those powers sensibly, but that much more information will also end up in the public domain — maximising its usefulness.

    Apply for free access

    If that all sounds exciting, then apply here. We’d love to hear how you plan to use WhatDoTheyKnowPro.

  5. Forward into alpha: what we’ve learned about Alaveteli Professional

    As we recently mentioned, one of mySociety’s current big projects is Alaveteli Professional, a Freedom of Information toolset for journalists.

    It’s something we wanted to build, and something we believed there was a need for: but wanting and believing do not make a sound business case, and that’s why we spent the first few weeks of the project in a ‘discovery’ phase.

    Our plan was to find out as much as we could about journalists, our prospective users — and particularly just how they go about using FOI in their work. Ultimately, though, we were seeking to understand whether journalists really would want, or need, the product as we were imagining it.

    So we went and talked to people at both ends of the FOI process: on the one hand, journalists who make requests, and on the other, the information officers who respond to them.

    Since we’re planning on making Alaveteli Professional available to partners around the world, it also made sense to conduct similar interviews outside the UK. Thanks to links with our Czech partner, running Informace Pro Všechny on Alaveteli, that was a simple matter. A recent event at the Times building in London also allowed us to present and discuss our findings, and listen to a couple of interesting expert presentations: Matt Burgess of Buzzfeed talked about some brilliant use of FOI to expose criminal landlords, and listed FOI officers’ biggest complaints about journalists. Josh Boswell of the Sunday Times was equally insightful as he ran through the ways that he uses FOI when developing stories.

    These conversations have all helped.

    The life of an investigative journalist is never simple

    Alaveteli Professional process diagram drawn by Mike Thompson

    The insights our interviewees gave us were turned by Mike Thompson (formerly of mySociety, and brought back in for this phase) into a simple process model showing how journalists work when they’re pursuing an investigation using FOI.

    After conceiving of a story that requires input from one or more FOI request, every journalist will go through three broad phases: research; request and response; and the final data analysis and writing. The more complicated cases can also involve refused requests and the appeals process.

    For a busy working journalist, there are challenges at every step. Each of these adds time and complexity to the process of writing a story, which is an anathema to the normal daily news cycle. FOI-based stories can be slow, and timing unpredictable — editors do not particularly like being told that you’re working on a story, but can’t say when it will be ready, or how much value it will have.

    During the research phase diligent journalists will make a time-consuming trawl through resources like authorities’ own disclosure logs and our own site WhatDoTheyKnow (or its equivalents in other countries), to see if the data they need has already been released.

    Where a ‘round robin’ request is planned, asking for information from multiple authorities — sometimes hundreds — for information, further research is needed to ensure that only relevant bodies are included. In our two-tired council system, where different levels of authority deal with different responsibilities, and not always according to a consistent pattern, that can be a real challenge.

    Wording a request also takes some expertise: get that wrong and the authorities will be coming back for clarification, which adds even more time to the process.

    Once the request has been made it’s hard to keep on top of correspondence, especially for a large round robin request. Imagine sending a request to every council in the country, as might well be done for a UK-wide story, and then dealing with each body’s acknowledgements, requests for clarifications and refusals.

    When the responses are in journalists often find that interpretation is a challenge. Different authorities might store data or measure metrics differently from one another; and pulling out a meaningful story means having the insight to, for example, adjust figures to account for the fact that different authorities are different sizes and cater for differently-dispersed populations.

    Sadly, it’s often at this stage that journalists realise that they’ve asked the wrong question to start with, or wish that they’d included an additional dimension to the data they’ve requested.

    What journalists need

    As we talked through all these difficulties with journalists, we gained a pretty good understanding of their needs. Some of these had been on our list from the start, and others were a surprise, showing the value of this kind of exploration before you sit down to write a single line of code.

    Here’s what our final list of the most desirable features looks like:

    An embargo We already knew, anecdotally, that journalists tend not to use WhatDoTheyKnow to make requests, because of its public nature. It was slightly sobering to have this confirmed via first person accounts from journalists who had had their stories ‘stolen’… and those who admitted to having appropriated stories themselves! Every journalist we spoke to agreed that any FOI tool for their profession would need to include a way of keeping requests hidden until after publication of their story.

    However, this adds a slight dilemma. Using Alaveteli Professional and going through the embargo-setting process introduces an extra hurdle into the journalist’s process, when our aim is, of course, to make the FOI procedure quicker and smoother. Can we ensure that everything else is so beneficial that this one additional task is worthwhile for the user?

    Talking to journalists, we discovered that almost all are keen to share their data once their story has gone live. Not only does it give concrete corroboration of the piece, but it was felt that an active profile on an Alaveteli site, bursting with successful investigations, could add to a journalist’s reputation — a very important consideration in the industry.

    Request management tools Any service that could put order into the myriad responses that can quickly descend into chaos would be welcome for journalists who typically have several FOI requests on the go at any one time.

    Alaveteli Professional’s dashboard interface would allow for a snapshot view of request statuses. Related requests could be bundled together, and there would be the ability to quickly tag and classify new correspondence.

    Round-robin tools Rather than send a notification every time a body responds (often with no more than an acknowledgement), the system could hold back, alerting you only when a request appears to need attention, or send you status updates for the entire project at predefined intervals.

    Refusal advice Many journalists abandon a request once it’s been refused, whether from a lack of time or a lack of knowledge about the appeals process. WhatDoTheyKnow Professional would be able to offer in-context advice on refusals, helping journalists take the next step.

    Insight tools Can we save journalists’ time in the research phase, by giving an easy representation of what sort of information is already available on Alaveteli sites, and by breaking down what kind of information each authority holds? That could help with terminology, too: if a request refers to data in the same language that is used internally within the council, then their understanding of the request and their response is likely to be quicker and easier.

    Onwards to Alpha

    We’re currently working on the next part of the build — the alpha phase.

    In this, we’re building quick, minimally-functional prototypes that will clearly show how Alaveteli Professional will work, but without investing time into a fully-refined product. After all, what we discover may mean that we change our plans, and it’s better not to have gone too far down the line at that point.

    If you are a journalist and you would like to get involved with testing during this stage and the next — beta — then please do get in touch at alaveteli-professional@mysociety.org.


    Image: Goodwines (CC by-nc-nd/2.0)

  6. FixMyStreet version 2.0

    bureau of street traffic

    The FixMyStreet codebase is used all over the world by people running versions of the site for their own country or jurisdiction. This week, we’re proud to announce the release of FixMyStreet version 2.0.

    This version contains a wide array of new features that benefit FixMyStreet sites’ users, administrators, and the officials who receive reports. They include elements that the UK FixMyStreet was the first to trial, such as nicer-looking HTML emails for users and authorities, the ability to filter reports by multiple states and categories, a new admin user system with graduated permissions, and various bugfixes and development improvements.

    Over the next few weeks, we’ll be publishing a series of blog posts over on fixmystreet.org/blog/, examining the changes in detail. If you run a FixMyStreet site, or you’re just interested in coding and technical issues, you may find them of interest. Meanwhile, here’s the broad overview.

    New front-end features

    • HTML email: There is now the option for all emails sent by FixMyStreet to be HTML formatted where previously they were plain text only. This includes confirmation and questionnaire emails to the user, and report emails to the public body. These emails include any image added to the report, plus a small static map of the problem’s location.

    • State/category filtering and sorting of list pages: When viewing a list of reports, you can now filter and sort them in pretty much any way you choose, including sorting by most- or least-recently updated, newest or oldest, or most commented. You can also select multiple categories or states (e.g. “fixed”).

    • Pretty area highlighting on body pages: The highlighting of areas on a body page has been inverted, so that the unimportant parts of the map are shaded and you can interact more easily with reports on the page.

    • Users can now update their own email address This was a frequent request from users and we’re glad to report that they can now do it themselves on their account page.

    • Performance improvements: When looking at reports from a list page, the other report pins stay visible so that it is easier to switch between them. The report itself is being pulled in behind the scenes, meaning the whole page does not need to reload. The map no longer extends underneath the sidebar and header, which makes things easier, and a scroll wheel can now zoom the map in and out.

    • Making privacy options clearer: The reporting form has been separated into public and private sections, to make it clearer which parts of what you provide will be made visible on the site.

      Showing the relevant recipient: If you live in an area where there’s more than one body, the category you pick normally dictates which body we send your report to. Now, when you select the category we update the name of the body given at the top of the report page, if we know that the report will be sent there.

    New admin user system

    Admin users can now use the same log-in right across the site – whether they’re making a report like a standard user, or logging in to make edits and moderate the site.

    In the past, the distinction between admin and other users was black and white. As an admin user, you had access to every part of the site, but users can now be given individual permissions for various layers of access. These include:

    • Proxy users This layer grants the ability to create a report or update on behalf of a body, or as another user. We envisage this being useful in a body’s contact centre, where they receive a report over a phone and enter it into FixMyStreet as that user;
    • Report editors Giving the power to edit a report’s category, state, or location. If the admin user changes the category, and that change means that a different body is now responsible for the report, it will be re-sent;
    • List makers, who can compile their own shortlist of reports they wish to go and inspect. This may be useful for a contractor or team who wishes to compile the day’s tasks;
    • Quick responders These users have access to response templates, allowing them to edit and publish templated updates;
    • Prioritisers These users may set different priorities on reports;
    • Trusted users A simple reputation system, which e.g. potentially lets reports from trusted users be actioned more quickly.

    The admin report edit form has also been greatly improved, including a map to update a report’s location (and re-sending the report if the body changes), and much tidier layout.

    Bugfixes and development changes

    Bugfixes include updating the top-level domain (TLD) list for email validation, hiding authorities which don’t exist any more on the all reports page, and fixing the previously-broken photo preview display after form submission. We have dropped support for Internet Explorer 6.

    If you’re a re-user of the codebase, there are a number of changes that will hopefully help you out. See the extended version of this blog post on fixmystreet.org for more details.

    If you have any questions, or problems installing the code, please do get in touch, or post on our mailing list.

  7. Asking questions in public: the Alaveteli experiments

    Suppose we sent an automated tweet every time someone made a successful Freedom of Information request on WhatDotheyKnow — would it bring more visitors to the site?

    And, if you get a response to your first FOI request, does it mean you are more likely to make a second one?

    These, and many more, are the kind of questions that emerge as we refine the advice that we’re offering partner organisations.

    Our Freedom of Information platform Alaveteli underpins Freedom of Information sites all around the world. When we first launched it, our only priorities were to make the code work, and to make that code as easy as possible to implement. But, as a community emerged around Alaveteli, we realised that we’d all be better off if we shared advice, successes and ideas.

    And that’s where we began to encounter questions.

    Some of them, like how to get more users, or how to understand where users come from, are common to anyone running a website.

    Others are unique to our partner structure, in which effectively anyone in any part of the world may pick up the Alaveteli code and start their own site. In theory, we might know very little more than that a site is running, although we’ll always try to make contact and let the implementers know what help we can offer them.

    There were so many questions that we soon saw the need to keep them all in one place. At mySociety, we’re accustomed to using Github for anything resembling a to-do list (as well as for its primary purposes; Github was designed to store code, allow multiple people to work on that code, and to suggest or review issues with it), and so we created a slightly unusual repo, Alaveteli-experiments.

    Screenshot of the Alaveteli Experiments repo, showing a table of experiments and summaries of their results

    This approach also gives us the benefit of transparency. Anyone can visit that repo and see what questions we are asking, how we intend to find the answers, and the results as they come in. What’s more, anyone who has (or opens) a Github account will also be able to add their own comments.

    Have a browse and you’ll come across experiments like this one and this one, which attempt to answer the questions with which we opened this post.

    Some of the experiments, like this one to analyse whether people click the ‘similar requests’ links in the sidebar, we’re running on our own site, WhatDoTheyKnow. Others, such as this one about the successful requests listed on every Alaveteli site’s homepage, are being conducted on our partners’ sites.

    Our aims are to find out more about how to bring more users to all Alaveteli sites, how to encourage browsing visitors to become people who make requests, and how to turn one-off requesters into people who come back and make another — and then pass all that on to our partners.

    We hope you’ll find plenty of interest on there. We reckon it’s all relevant, especially to anyone running an FOI website, but in many cases to anyone wondering how best to improve a site’s effectiveness. And we’re very happy to hear your ideas, too: if we’ve missed some obvious experiment, or you’ve thought of something that would be really interesting to know through the application of this kind of research, you’re  welcome to let us know.

    You can open your own ticket on the repo, suggest it in the Alaveteli community mailing list, or email Alaveteli Partnerships Manager Gemma.

     


    Image: Sandia Labs (CC by-nc-nd/2.0)

  8. Alaveteli Release 0.25

    We’ve released Alaveteli 0.25! Here are some of the highlights.

    Visible delivery status

    Gareth and Zarino have added a delivery status feature that shows whether a message has been received by the authority’s mailserver. This should provide reassurance for site users that messages are getting through and makes it difficult for an authority to successfully claim that they didn’t receive the request.

    New delivery status feature

    Clicking on the delivery status indicator reveals a bit more detail about the status itself. Admins are shown more detail here including relevant mail logs to diagnose problems or provide proof to the authority if required.

    Analytics

    We’ve upgraded from the so-called “Legacy” (ga.js) version of Google Analytics to Universal Analytics. For most Google Analytics users there’s nothing to do here except sit back and enjoy continued technical support and new feature rollouts from Google but if your Alaveteli theme has custom analytics scripting, you should check Google’s upgrade guide as well as our upgrade notes to see if you need to make changes. If you’re not ready to move to this release yet, don’t panic – you may not get any shiny new features from Google but they haven’t published an end date for support yet.

    Profile spam

    In addition to spam email, there’s been an increase in the number of accounts that create profiles containing spam links – presumably to boost their search engine ranking score rather than to trick people into clicking through from the site. Having spent some time going through accounts on WhatDoTheyKnow to look for patterns, we’ve added some tools to this release to try to discourage this use of Alaveteli and to make it easier for admins to discover and ban offending user accounts.

    you can't update your profile

    (We also looked at extending our reCAPTCHA use for new account signups but this didn’t seem to help so we are not offering it to reusers.)

    Design

    Martin has been working away on improving page load times and accessibility compliance to make the pages faster to load and easier to navigate. (A process we’re continuing into the next release.) We’ve also updated the help template code so that the examples are in the example theme rather than the core code and added a rake task to help check whether your theme implements the help pages correctly.

    The full list of highlights and upgrade notes for this release is in the changelog.

    Thanks again to everyone who’s contributed!

    Image: Miika Mehtälä (CC)

  9. A new Sass workflow for Alaveteli theming

    We recently made some changes to our Freedom of Information software, Alaveteli, that makes it twice as easy for partners to change the look and feel of their site. This post goes quite in-depth, so it’s probably best suited for designers, developers and other technically-minded readers


    This year we embarked on a ground-up redesign of Alaveteli’s theming capabilities to make customising Alaveteli easier and faster. We used a mixture of techniques to try to achieve this and since releasing the update we’re happy to have seen a decrease in the complexity of our themes, and the time we spend making them.

    Alaveteli, our Freedom of Information platform is one of our most deployed software packages and each deployment requires some customisation, from simple branding changes to complex functionality to support a country’s FOI process. This is either done internally by our own developers, or our partners take this on themselves.

    An Alaveteli deployment consists of two things, the core: the guts of Alaveteli with no customisations, and the theme: the aesthetic and functional customisations made to each site. We ship Alaveteli with a fully-featured base theme that gives partners a good starting point to work from, but they can also start their own theme from scratch.

    After talking to our partners and the Alaveteli team we felt that the process of theming was too complicated and involved. It required a lot of knowledge of Alaveteli’s inner workings, was coded in a way that made it hard to understand, and almost always required the intervention of a front-end expert, even to make trivial changes.

    Our vision for how Alaveteli’s theming should work was:

    • Intuitive Front-end developers and designers should be able to follow and understand the templates without needing expert knowledge about Alaveteli’s inner-workings.
    • Flexible Overriding Alaveteli’s core code should be easy, so a theme should use low-specificity, componentised code to enable this.
    • Supportive Should a partner decide to only change a few colours, or embark on a large-scale redesign Alaveteli, the theming process should never get in the way.

    How we did it

    Using the tools we have

    Alaveteli is built on Ruby on Rails, so our first goal was to make better use of Ruby on Rail’s template inheritance.

    Firstly, we split all of the Sass files into logical components.

    Each component has two associated Sass files, a layout.scss and a style.scss. layout.scss describes only layout of a component, e.g. the physical dimensions and position on a page. style.scss describes the styles, for example typefaces, colours, borders and other presentational properties.

    Should a partner wish to totally redesign or replace a component, they can replace both files with their own. Alternatively they can just choose to replace one part of the component, for example keeping the layout, but replacing the styles.

    Ensuring the lowest possible specificity

    When customising Alaveteli we want to be sure that a partner never has to fight with the base theme to achieve their goals. The most important part of this is to ensure CSS styles are easy to override.

    Firstly we removed all ID selector’s from Alaveteli’s stylesheets and replaced them with class selectors, giving us a good jumping off point as class selectors have much lower specificity than IDs. We then rewrote and refactored Alaveteli’s CSS using the Block Element Modifier methodology (BEM).

    BEM helped us in two ways: it’s written in a way that makes CSS self-describing, for example the code snippet .header__navigation__list tells us that this list is a child of the navigation element and a grandchild of the header element.

    As well as this, using a single class to define the styles for this element means that using BEM gives our styles the lowest possible specificity, so partners can override them with little resistance.

    Using a settings partial

    All of our theme’s styles use the variables defined in _settings.scss when referencing colours, typefaces, spacing, and even logo files so the bulk of Alaveteli’s customisation can be done in our base theme’s settings partial _settings.scss.

    This allows partners to make simple changes which have a huge impact on the look and feel of the site. We’ve found that the large majority of our theme customisation can be done by just changing the variables in this file and it has sped up the theming process markedly.

    Impact

    We’ve quietly rolled out these updates to Alaveteli over the past 12 months and the impact of the changes has been encouraging.

    Internally, we’ve found our theming projects are faster and more efficient. We conservatively estimate that it takes half the time to make a new theme than it used to.

    Thanks to the self-documenting nature of BEM and the styles encapsulated in our settings partial, Alaveteli theming requires a lot less understanding of the platform’s inner workings. Our partners are able to be more self-sufficient and rarely require our intervention to assist with simple customisations.

    Maintenance and support of our themes is greatly simplified, as the structure is consistent across projects, we’re able to roll out further changes and fixes in a more predictable way.
    Overall, we’re really happy with how the changes worked out and the things we’ve learnt from this project we’ll be carrying into our others too, so look out for changes like across our work in the future.

  10. Introducing EveryPolitician’s bot

    mySociety’s EveryPolitician project aims to make data available on every politician in the world. It’s going well: we’re already sharing data on the politicians from nearly every country on the planet. That’s over 68,652 people and 2.9 million individual pieces of data, numbers which will be out of date almost as soon as you’ve read them. Naturally, the width and depth of that data varies from country to country, depending on the sources available — but that’s a topic for another blog post.

    Today the EveryPolitician team would like to introduce you to its busiest member, who is blogging at EveryPolitician bot. A bot is an automated agent — a robot, no less, albeit one crafted entirely in software.

    First, some background on why we need our little bot.

    Because there’s so much to do

    One of the obvious challenges of such a big mission is keeping on top of it all. We’re constantly adding and updating the data; it’s in no way a static dataset. Here are examples — by no means exhaustive — of circumstances that can lead to data changes:

    • Legislatures change en masse, because of elections, etc.
      We try to know when countries’ governments are due to change because that’s the kind of thing we’re interested in anyway (remember mySociety helps run websites for parliamentary monitoring organisations, such as Mzalendo in Kenya). But even anticipated changes are rarely straightforward, not least because there’s always a lag between a legislature changing and the data about its new members becoming available, especially from official national sources.
    • Legislatures change en masse, unexpectedly
      Not all sweeping changes are planned. There are coups and revolutions and other unscheduled or premature ends-of-term.
    • Politicians retire
      Or die, or change their names or titles, or switch party or faction.
    • New parties emerge
      Or the existing ones change their names, or form coalitions.
    • Areas change
      There are good reasons (better representation) and bad reasons (gerrymandering) why the areas in constituency-based systems often change. By way of a timely example, our UK readers probably know that the wards have changed for the forthcoming elections, and that mySociety built a handy tool that tells you what ward you’re in.
    • Existing data gets refined
      Played Gender Balance recently? Behind that is a dataset that keeps being updated (whenever there are new politicians) but which is itself a source of constantly-updating data for us.
    • Someone in Russia updates the wikipedia page about a politician in Japan
      Wikidata is the database underlying projects like Wikipedia, so by monitoring all the politicians we have that are also in there, we get a constant stream of updates. For example, within a few hours of someone adding it, we knew that the Russian transliteration of 安倍晋三’s name was Синдзо Абэ — that’s Shinzo Abe, in case you can’t read kanji or Cyrillic script. (If you’re wondering, whenever our sources conflict, we moderate in favour of local context.)
    • New data sources become available
      Our data comes from an ever-increasing number of sources, commonly more than one for any given legislature (the politicians’ twitter handles are often found in a different online place from their dates of birth, for example). We always welcome more contributions — if you think you’ve got new sources for the country you live in, please let us know.
    • New old data becomes available
      We collect historic data too — not just the politicians in the current term. For some countries we’ve already got data going back decades. Sources for data like this can sometimes be hard to find; slowly but surely new ones keeping turning up.

    So, with all this sort of thing going on, it’s too much to expect a small team of humans to manage it all. Which is where our bot comes in.

    Hello bot

    portrait of bot

    To be honest with you, the bot doesn’t really look like this because, being software, it’s entirely non-corporeal. Sorry.

    We’ve automated many of our processes: scraping, collecting, checking changes, submitting them for inclusion — so the humans can concentrate on what they do best (which is understanding things, and making informed decisions). In technical terms, our bot handles most things in an event-driven way. It springs into action when triggered by a notification. Often that will be a webhook (for example, a scraper finishes getting data so it issues a webhook to let the bot know), although the bot also follows a schedule of regular tasks too. Computers are great for running repetitive tasks and making quantitative comparisons, and a lot of the work that needs to be done with our ever-changing data fits such a description.

    The interconnectedness of all the different tasks the bot performs is complex. We originally thought we’d document that in one go — there’s a beautiful diagram waiting to be drawn, that’s for sure — but it soon became clear this was going to be a big job. Too big. Not only is the bot’s total activity complicated because there are a lot of interdependencies, but it’s always changing: the developers are frequently adding to the variety of tasks the bot is doing for us.

    So in the end we realised we should just let the bot speak for itself, and describe task-by-task some of the things it does. Broken down like this it’s easier to follow.

    We know not everybody will be interested, which is fine: the EveryPolitician data is useful for all sorts of people — journalists, researchers, parliamentary monitors, activists, parliamentarians themselves, and many more — and if you’re such a person you don’t need to know about how we’re making it happen. But if you’re technically-minded — and especially if you’re a developer who uses GitHub but hasn’t yet used the GitHub API as thoroughly as we’ve needed to, or are looking for ways to manage always-shifting data sets like ours — then we hope you’ll find what the bot says both informative and useful.

    everypoliticianbot-small
    The bot is already a few days into blogging — its first post was “I am a busy bot”, but you can see all the others on its own Medium page. You can also follow it on Twitter as @everypolitician. Of course, its true home, where all the real work is done, is the everypoliticianbot account on GitHub.


    Images: CC-BY-SA from the EveryPolitician bot’s very own scrapbook.