1. FixMyStreet version 2.0

    bureau of street traffic

    The FixMyStreet codebase is used all over the world by people running versions of the site for their own country or jurisdiction. This week, we’re proud to announce the release of FixMyStreet version 2.0.

    This version contains a wide array of new features that benefit FixMyStreet sites’ users, administrators, and the officials who receive reports. They include elements that the UK FixMyStreet was the first to trial, such as nicer-looking HTML emails for users and authorities, the ability to filter reports by multiple states and categories, a new admin user system with graduated permissions, and various bugfixes and development improvements.

    Over the next few weeks, we’ll be publishing a series of blog posts over on fixmystreet.org/blog/, examining the changes in detail. If you run a FixMyStreet site, or you’re just interested in coding and technical issues, you may find them of interest. Meanwhile, here’s the broad overview.

    New front-end features

    • HTML email: There is now the option for all emails sent by FixMyStreet to be HTML formatted where previously they were plain text only. This includes confirmation and questionnaire emails to the user, and report emails to the public body. These emails include any image added to the report, plus a small static map of the problem’s location.

    • State/category filtering and sorting of list pages: When viewing a list of reports, you can now filter and sort them in pretty much any way you choose, including sorting by most- or least-recently updated, newest or oldest, or most commented. You can also select multiple categories or states (e.g. “fixed”).

    • Pretty area highlighting on body pages: The highlighting of areas on a body page has been inverted, so that the unimportant parts of the map are shaded and you can interact more easily with reports on the page.

    • Users can now update their own email address This was a frequent request from users and we’re glad to report that they can now do it themselves on their account page.

    • Performance improvements: When looking at reports from a list page, the other report pins stay visible so that it is easier to switch between them. The report itself is being pulled in behind the scenes, meaning the whole page does not need to reload. The map no longer extends underneath the sidebar and header, which makes things easier, and a scroll wheel can now zoom the map in and out.

    • Making privacy options clearer: The reporting form has been separated into public and private sections, to make it clearer which parts of what you provide will be made visible on the site.

      Showing the relevant recipient: If you live in an area where there’s more than one body, the category you pick normally dictates which body we send your report to. Now, when you select the category we update the name of the body given at the top of the report page, if we know that the report will be sent there.

    New admin user system

    Admin users can now use the same log-in right across the site – whether they’re making a report like a standard user, or logging in to make edits and moderate the site.

    In the past, the distinction between admin and other users was black and white. As an admin user, you had access to every part of the site, but users can now be given individual permissions for various layers of access. These include:

    • Proxy users This layer grants the ability to create a report or update on behalf of a body, or as another user. We envisage this being useful in a body’s contact centre, where they receive a report over a phone and enter it into FixMyStreet as that user;
    • Report editors Giving the power to edit a report’s category, state, or location. If the admin user changes the category, and that change means that a different body is now responsible for the report, it will be re-sent;
    • List makers, who can compile their own shortlist of reports they wish to go and inspect. This may be useful for a contractor or team who wishes to compile the day’s tasks;
    • Quick responders These users have access to response templates, allowing them to edit and publish templated updates;
    • Prioritisers These users may set different priorities on reports;
    • Trusted users A simple reputation system, which e.g. potentially lets reports from trusted users be actioned more quickly.

    The admin report edit form has also been greatly improved, including a map to update a report’s location (and re-sending the report if the body changes), and much tidier layout.

    Bugfixes and development changes

    Bugfixes include updating the top-level domain (TLD) list for email validation, hiding authorities which don’t exist any more on the all reports page, and fixing the previously-broken photo preview display after form submission. We have dropped support for Internet Explorer 6.

    If you’re a re-user of the codebase, there are a number of changes that will hopefully help you out. See the extended version of this blog post on fixmystreet.org for more details.

    If you have any questions, or problems installing the code, please do get in touch, or post on our mailing list.

  2. Asking questions in public: the Alaveteli experiments

    Suppose we sent an automated tweet every time someone made a successful Freedom of Information request on WhatDotheyKnow — would it bring more visitors to the site?

    And, if you get a response to your first FOI request, does it mean you are more likely to make a second one?

    These, and many more, are the kind of questions that emerge as we refine the advice that we’re offering partner organisations.

    Our Freedom of Information platform Alaveteli underpins Freedom of Information sites all around the world. When we first launched it, our only priorities were to make the code work, and to make that code as easy as possible to implement. But, as a community emerged around Alaveteli, we realised that we’d all be better off if we shared advice, successes and ideas.

    And that’s where we began to encounter questions.

    Some of them, like how to get more users, or how to understand where users come from, are common to anyone running a website.

    Others are unique to our partner structure, in which effectively anyone in any part of the world may pick up the Alaveteli code and start their own site. In theory, we might know very little more than that a site is running, although we’ll always try to make contact and let the implementers know what help we can offer them.

    There were so many questions that we soon saw the need to keep them all in one place. At mySociety, we’re accustomed to using Github for anything resembling a to-do list (as well as for its primary purposes; Github was designed to store code, allow multiple people to work on that code, and to suggest or review issues with it), and so we created a slightly unusual repo, Alaveteli-experiments.

    Screenshot of the Alaveteli Experiments repo, showing a table of experiments and summaries of their results

    This approach also gives us the benefit of transparency. Anyone can visit that repo and see what questions we are asking, how we intend to find the answers, and the results as they come in. What’s more, anyone who has (or opens) a Github account will also be able to add their own comments.

    Have a browse and you’ll come across experiments like this one and this one, which attempt to answer the questions with which we opened this post.

    Some of the experiments, like this one to analyse whether people click the ‘similar requests’ links in the sidebar, we’re running on our own site, WhatDoTheyKnow. Others, such as this one about the successful requests listed on every Alaveteli site’s homepage, are being conducted on our partners’ sites.

    Our aims are to find out more about how to bring more users to all Alaveteli sites, how to encourage browsing visitors to become people who make requests, and how to turn one-off requesters into people who come back and make another — and then pass all that on to our partners.

    We hope you’ll find plenty of interest on there. We reckon it’s all relevant, especially to anyone running an FOI website, but in many cases to anyone wondering how best to improve a site’s effectiveness. And we’re very happy to hear your ideas, too: if we’ve missed some obvious experiment, or you’ve thought of something that would be really interesting to know through the application of this kind of research, you’re  welcome to let us know.

    You can open your own ticket on the repo, suggest it in the Alaveteli community mailing list, or email Alaveteli Partnerships Manager Gemma.

     


    Image: Sandia Labs (CC by-nc-nd/2.0)

  3. Alaveteli Release 0.25

    We’ve released Alaveteli 0.25! Here are some of the highlights.

    Visible delivery status

    Gareth and Zarino have added a delivery status feature that shows whether a message has been received by the authority’s mailserver. This should provide reassurance for site users that messages are getting through and makes it difficult for an authority to successfully claim that they didn’t receive the request.

    New delivery status feature

    Clicking on the delivery status indicator reveals a bit more detail about the status itself. Admins are shown more detail here including relevant mail logs to diagnose problems or provide proof to the authority if required.

    Analytics

    We’ve upgraded from the so-called “Legacy” (ga.js) version of Google Analytics to Universal Analytics. For most Google Analytics users there’s nothing to do here except sit back and enjoy continued technical support and new feature rollouts from Google but if your Alaveteli theme has custom analytics scripting, you should check Google’s upgrade guide as well as our upgrade notes to see if you need to make changes. If you’re not ready to move to this release yet, don’t panic – you may not get any shiny new features from Google but they haven’t published an end date for support yet.

    Profile spam

    In addition to spam email, there’s been an increase in the number of accounts that create profiles containing spam links – presumably to boost their search engine ranking score rather than to trick people into clicking through from the site. Having spent some time going through accounts on WhatDoTheyKnow to look for patterns, we’ve added some tools to this release to try to discourage this use of Alaveteli and to make it easier for admins to discover and ban offending user accounts.

    you can't update your profile

    (We also looked at extending our reCAPTCHA use for new account signups but this didn’t seem to help so we are not offering it to reusers.)

    Design

    Martin has been working away on improving page load times and accessibility compliance to make the pages faster to load and easier to navigate. (A process we’re continuing into the next release.) We’ve also updated the help template code so that the examples are in the example theme rather than the core code and added a rake task to help check whether your theme implements the help pages correctly.

    The full list of highlights and upgrade notes for this release is in the changelog.

    Thanks again to everyone who’s contributed!

    Image: Miika Mehtälä (CC)

  4. A new Sass workflow for Alaveteli theming

    We recently made some changes to our Freedom of Information software, Alaveteli, that makes it twice as easy for partners to change the look and feel of their site. This post goes quite in-depth, so it’s probably best suited for designers, developers and other technically-minded readers


    This year we embarked on a ground-up redesign of Alaveteli’s theming capabilities to make customising Alaveteli easier and faster. We used a mixture of techniques to try to achieve this and since releasing the update we’re happy to have seen a decrease in the complexity of our themes, and the time we spend making them.

    Alaveteli, our Freedom of Information platform is one of our most deployed software packages and each deployment requires some customisation, from simple branding changes to complex functionality to support a country’s FOI process. This is either done internally by our own developers, or our partners take this on themselves.

    An Alaveteli deployment consists of two things, the core: the guts of Alaveteli with no customisations, and the theme: the aesthetic and functional customisations made to each site. We ship Alaveteli with a fully-featured base theme that gives partners a good starting point to work from, but they can also start their own theme from scratch.

    After talking to our partners and the Alaveteli team we felt that the process of theming was too complicated and involved. It required a lot of knowledge of Alaveteli’s inner workings, was coded in a way that made it hard to understand, and almost always required the intervention of a front-end expert, even to make trivial changes.

    Our vision for how Alaveteli’s theming should work was:

    • Intuitive Front-end developers and designers should be able to follow and understand the templates without needing expert knowledge about Alaveteli’s inner-workings.
    • Flexible Overriding Alaveteli’s core code should be easy, so a theme should use low-specificity, componentised code to enable this.
    • Supportive Should a partner decide to only change a few colours, or embark on a large-scale redesign Alaveteli, the theming process should never get in the way.

    How we did it

    Using the tools we have

    Alaveteli is built on Ruby on Rails, so our first goal was to make better use of Ruby on Rail’s template inheritance.

    Firstly, we split all of the Sass files into logical components.

    Each component has two associated Sass files, a layout.scss and a style.scss. layout.scss describes only layout of a component, e.g. the physical dimensions and position on a page. style.scss describes the styles, for example typefaces, colours, borders and other presentational properties.

    Should a partner wish to totally redesign or replace a component, they can replace both files with their own. Alternatively they can just choose to replace one part of the component, for example keeping the layout, but replacing the styles.

    Ensuring the lowest possible specificity

    When customising Alaveteli we want to be sure that a partner never has to fight with the base theme to achieve their goals. The most important part of this is to ensure CSS styles are easy to override.

    Firstly we removed all ID selector’s from Alaveteli’s stylesheets and replaced them with class selectors, giving us a good jumping off point as class selectors have much lower specificity than IDs. We then rewrote and refactored Alaveteli’s CSS using the Block Element Modifier methodology (BEM).

    BEM helped us in two ways: it’s written in a way that makes CSS self-describing, for example the code snippet .header__navigation__list tells us that this list is a child of the navigation element and a grandchild of the header element.

    As well as this, using a single class to define the styles for this element means that using BEM gives our styles the lowest possible specificity, so partners can override them with little resistance.

    Using a settings partial

    All of our theme’s styles use the variables defined in _settings.scss when referencing colours, typefaces, spacing, and even logo files so the bulk of Alaveteli’s customisation can be done in our base theme’s settings partial _settings.scss.

    This allows partners to make simple changes which have a huge impact on the look and feel of the site. We’ve found that the large majority of our theme customisation can be done by just changing the variables in this file and it has sped up the theming process markedly.

    Impact

    We’ve quietly rolled out these updates to Alaveteli over the past 12 months and the impact of the changes has been encouraging.

    Internally, we’ve found our theming projects are faster and more efficient. We conservatively estimate that it takes half the time to make a new theme than it used to.

    Thanks to the self-documenting nature of BEM and the styles encapsulated in our settings partial, Alaveteli theming requires a lot less understanding of the platform’s inner workings. Our partners are able to be more self-sufficient and rarely require our intervention to assist with simple customisations.

    Maintenance and support of our themes is greatly simplified, as the structure is consistent across projects, we’re able to roll out further changes and fixes in a more predictable way.
    Overall, we’re really happy with how the changes worked out and the things we’ve learnt from this project we’ll be carrying into our others too, so look out for changes like across our work in the future.

  5. Introducing EveryPolitician’s bot

    mySociety’s EveryPolitician project aims to make data available on every politician in the world. It’s going well: we’re already sharing data on the politicians from nearly every country on the planet. That’s over 68,652 people and 2.9 million individual pieces of data, numbers which will be out of date almost as soon as you’ve read them. Naturally, the width and depth of that data varies from country to country, depending on the sources available — but that’s a topic for another blog post.

    Today the EveryPolitician team would like to introduce you to its busiest member, who is blogging at EveryPolitician bot. A bot is an automated agent — a robot, no less, albeit one crafted entirely in software.

    First, some background on why we need our little bot.

    Because there’s so much to do

    One of the obvious challenges of such a big mission is keeping on top of it all. We’re constantly adding and updating the data; it’s in no way a static dataset. Here are examples — by no means exhaustive — of circumstances that can lead to data changes:

    • Legislatures change en masse, because of elections, etc.
      We try to know when countries’ governments are due to change because that’s the kind of thing we’re interested in anyway (remember mySociety helps run websites for parliamentary monitoring organisations, such as Mzalendo in Kenya). But even anticipated changes are rarely straightforward, not least because there’s always a lag between a legislature changing and the data about its new members becoming available, especially from official national sources.
    • Legislatures change en masse, unexpectedly
      Not all sweeping changes are planned. There are coups and revolutions and other unscheduled or premature ends-of-term.
    • Politicians retire
      Or die, or change their names or titles, or switch party or faction.
    • New parties emerge
      Or the existing ones change their names, or form coalitions.
    • Areas change
      There are good reasons (better representation) and bad reasons (gerrymandering) why the areas in constituency-based systems often change. By way of a timely example, our UK readers probably know that the wards have changed for the forthcoming elections, and that mySociety built a handy tool that tells you what ward you’re in.
    • Existing data gets refined
      Played Gender Balance recently? Behind that is a dataset that keeps being updated (whenever there are new politicians) but which is itself a source of constantly-updating data for us.
    • Someone in Russia updates the wikipedia page about a politician in Japan
      Wikidata is the database underlying projects like Wikipedia, so by monitoring all the politicians we have that are also in there, we get a constant stream of updates. For example, within a few hours of someone adding it, we knew that the Russian transliteration of 安倍晋三’s name was Синдзо Абэ — that’s Shinzo Abe, in case you can’t read kanji or Cyrillic script. (If you’re wondering, whenever our sources conflict, we moderate in favour of local context.)
    • New data sources become available
      Our data comes from an ever-increasing number of sources, commonly more than one for any given legislature (the politicians’ twitter handles are often found in a different online place from their dates of birth, for example). We always welcome more contributions — if you think you’ve got new sources for the country you live in, please let us know.
    • New old data becomes available
      We collect historic data too — not just the politicians in the current term. For some countries we’ve already got data going back decades. Sources for data like this can sometimes be hard to find; slowly but surely new ones keeping turning up.

    So, with all this sort of thing going on, it’s too much to expect a small team of humans to manage it all. Which is where our bot comes in.

    Hello bot

    portrait of bot

    To be honest with you, the bot doesn’t really look like this because, being software, it’s entirely non-corporeal. Sorry.

    We’ve automated many of our processes: scraping, collecting, checking changes, submitting them for inclusion — so the humans can concentrate on what they do best (which is understanding things, and making informed decisions). In technical terms, our bot handles most things in an event-driven way. It springs into action when triggered by a notification. Often that will be a webhook (for example, a scraper finishes getting data so it issues a webhook to let the bot know), although the bot also follows a schedule of regular tasks too. Computers are great for running repetitive tasks and making quantitative comparisons, and a lot of the work that needs to be done with our ever-changing data fits such a description.

    The interconnectedness of all the different tasks the bot performs is complex. We originally thought we’d document that in one go — there’s a beautiful diagram waiting to be drawn, that’s for sure — but it soon became clear this was going to be a big job. Too big. Not only is the bot’s total activity complicated because there are a lot of interdependencies, but it’s always changing: the developers are frequently adding to the variety of tasks the bot is doing for us.

    So in the end we realised we should just let the bot speak for itself, and describe task-by-task some of the things it does. Broken down like this it’s easier to follow.

    We know not everybody will be interested, which is fine: the EveryPolitician data is useful for all sorts of people — journalists, researchers, parliamentary monitors, activists, parliamentarians themselves, and many more — and if you’re such a person you don’t need to know about how we’re making it happen. But if you’re technically-minded — and especially if you’re a developer who uses GitHub but hasn’t yet used the GitHub API as thoroughly as we’ve needed to, or are looking for ways to manage always-shifting data sets like ours — then we hope you’ll find what the bot says both informative and useful.

    everypoliticianbot-small
    The bot is already a few days into blogging — its first post was “I am a busy bot”, but you can see all the others on its own Medium page. You can also follow it on Twitter as @everypolitician. Of course, its true home, where all the real work is done, is the everypoliticianbot account on GitHub.


    Images: CC-BY-SA from the EveryPolitician bot’s very own scrapbook.

  6. Unpicking code, for the good of the project

    Last year, when we were helping to develop YourNextMP, the candidate-crowdsourcing platform for the General Election, we made what seemed like an obvious decision.

    We decided to use PopIt as the site’s datastore — the place from which it could draw information about representatives: their names, positions, et cetera. We’d been developing PopIt as a solution for parliamentary monitoring sites, but we reckoned it would also be a good fit for YourNextMP.

    That turned out to be the wrong choice.

    YourNextMP was up and running in time for the election, but at the cost of many hours of intensive development as we tried to make PopIt do what was needed for the site.

    Once you’ve got an established site in production, changing the database it uses isn’t something you do lightly. But on returning to the codebase to develop it for international reuse, we had to admit that, in the words of mySociety developer Mark Longair, PopIt was “actually causing more problems than it was solving”. It was time to unpick the code and take a different approach.

    Mark explains just what it took to decide to change course in this way, over on his own blog.

    The post contains quite a bit of technical detail, but it’s also an interesting read for anyone who’s interested in when, and why, it’s sometimes best to question the decisions you’ve made.

     

    Image: Michelle (CC)

  7. Apply for support and development help

    There are websites built on mySociety code in many countries across the world.

    If your country doesn’t already have one, perhaps you’re thinking of setting up a FixMyStreet site for your area, or maybe a Freedom of Information site run on Alaveteli?

    Possibly you’re looking at WriteInPublic or YourNextRepresentative.

    Whatever the site you’re planning, you’ll find it a lot easier with our support and development help.

    Our quarterly call for applications closes on October 30, so make sure you have yours in soon. Want to know exactly what’s involved? Start here.

     

    Image: Damian Gadal (cc)