1. How we got a 90-plus PageSpeed score on our 2020 annual report

    In this technical deep-dive, our designer Zarino Zappia explores how we used the latest web performance techniques to make our 2020 annual report quick and easy to download — no matter what kind of connection or device you’re using.


    Each year, mySociety produces an online annual report sharing a round-up of our achievements, a thank you to our supporters, funders, and volunteers, and our vision for the immediate future.

    For the last two years, that annual report has taken the form of a single, multi-section web page.

    This year, with more news than ever to cover across both mySociety and our commercial subsidiary SocietyWorks, we knew it was super important to get that single page running as smoothly and efficiently as possible in visitors’ web browsers.

    Our annual report is often shared as an introduction to the work mySociety does. It’s crucial that it sets a good first impression, and performance is a big part of that. It’s also an embodiment of what sets us apart from the crowd – we sweat the small stuff, to give users the best possible experience.

    What’s a PageSpeed score anyway?

    Google has done a lot of work championing fast loading websites, and their PageSpeed Insights tool is often used as a benchmark of web performance. Developers aim to achieve a mobile performance score of 90 or more – but often this is easier said than done!

    In this fairly technical blog post, I’m going to go through a few of ways we really optimised the performance of our 2020 annual report, to achieve a consistent 90-plus PageSpeed score. Through a combination of common sense, technological restraint, and a little extra effort, we manage to serve 6,500 words (a 33% increase over last year), 90 images, and two videos, over nine sections of a single page, in the blink of an eye.

    Here’s how.

    Keep it simple, stupid

    Like most mySociety products, the annual report starts with a very boring, very unfashionable, but very fast base – plain old HTML – in this case generated by Jekyll. Jekyll helps to keep our work maintainable by letting us refactor common elements out into their own includes, but since everything compiles to plain HTML, we get lightning fast page renders from our Nginx web server. We use static HTML sites a lot at mySociety – for example, for our documentation, our prototypes, and even some of our user-facing products. Sometimes simple is just better!

    Where we want to improve the user experience through interaction – things like hiding and showing the menu, and updating the URL as you scroll down the page – then we add small amounts of progressive enhancement via JavaScript. Again, like most mySociety products, the annual report treats JavaScript as a nice-to-have – everything would work without it, which means the browser can render the page quickly and our users spend less time looking at a blank white loading screen.

    On the styling side, we use Bootstrap to help us quickly pull together the frontend design work on the annual report in about a week at the beginning of December. We build our own slimmed down version of Bootstrap from the Sass source, with just the modules we need. Libraries like Bootstrap can sometimes be a bit of a sledgehammer to crack a nut, but by cherry-picking just the modules you need, you can dramatically reduce the size of your compiled CSS, without losing the flexibility and fast prototyping that a library like Bootstrap provides. The annual report makes heavy use of Bootstrap’s built-in components and utility functions, and then anything truly bespoke is handled by a few of our own custom styles laid on top.

    Use the tools at your disposal

    Performance tweaks that were cutting edge a few years ago have quickly gained widespread browser support, so if you’re not using them, you’re missing out. Most of them take pretty much no effort to set up, and will dramatically improve perceived loading time for your users.

    As a single, big, long page with almost 100 image tags, it’s imperative that we use lazy loading, to prevent the browser requesting all of those images at once on initial page load. Doing this is as easy as adding loading="lazy" to the image tags, but we also add a JavaScript fallback for browsers that don’t support the loading attribute, just to be safe. We even wrap it up in a nice Jekyll include, to make it super easy when managing the page content.

    We minify our CSS and JavaScript files, and the server is set to gzip them in transit. These things are really easy to forget, but can make a big difference to load time of a page full of static files.

    We also make an effort to efficiently encode our images – serving images at roughly the size they’re actually displayed, to reduce wasted bandwidth, and using WEBP compression with JPEG fallbacks, for the best balance of quality and filesize.

    Reduce run-time dependencies

    A big part of webpage performance is all the stuff that happens after the browser starts rendering the page. Loading external JavaScript libraries, tracking services, logging services – it all imposes a cost on your users. For some larger projects, you might decide the benefits for your team and the health of the product outweigh the network and performance cost for your users. But for the annual report we did the typically mySociety thing of reducing, reducing, reducing, to the simplest possible outcome. For example…

    We don’t load any social media JavaScript. Data is patchy on whether people actually use social sharing buttons anyway, but just in case, where we do display sharing buttons, we use old-fashioned sharing URLs that just open in a new tab. Rather than embedding tweets using Twitter’s JavaScript embed API, we build our own tweets out of HTML and CSS. Not only does this mean we’re avoiding having to load megabytes of third-party JavaScript on our page, but it also helps protect the privacy of our visitors.

    Sometimes you can’t avoid third-party JavaScript, though. For example, YouTube embeds. What you can do is defer loading the third-party JavaScript until right before it’s needed. In the case of YouTube embeds on the annual report, we use a little bit of our own JavaScript to delay loading any YouTube code until you click on a video thumbnail. To the end user, the entire thing happens instantaneously. But we know we’ve saved a whole lot of time not loading those iframes until they were really needed.

    Unusually for a mySociety project, we don’t even load jQuery on the annual report. While jQuery still makes sense for our bigger products, on something as simple as our annual report, it’s just not worth the filesize overhead. Instead we write modern, efficient JavaScript, and include small polyfills for features that might not be present in older browsers. It makes development slightly slower, but it’s worth it for a faster, smoother experience for our visitors.

    Nothing’s perfect!

    Overall, some sensible tech decisions early on, and a lot of hard work from the design team (including our new hire, Clive) resulted in a 2020 annual report I think we can be really proud of. I hope it’s also a taste of things to come, as we start introducing the same techniques into our other, bigger products.

    That said, there’s still a few things we could improve.

    We could achieve almost instant rendering on first pageload by extracting the CSS required to display the first few hundred pixels of the page, and inlining it, so it renders before the rest of the styles have been downloaded. We already do this on fixmystreet.com, for example, using Penthouse.

    There’s also an external dependency we could drop – Google Fonts. We’re currently using it to serve our corporate typeface, Source Sans Pro. The advantage of loading the typeface off Google Fonts is that most of our other sites also load the same typeface, so chances are the user will already have the font files in their browser cache. Self-hosting the fonts is unlikely to be much faster than loading from Google, but it would be a win for user privacy, since it’s one less external resource being hit for each pageload. Something to consider for future!

     

     

    Photo by Indira Tjokorda on Unsplash

  2. Alaveteli Releases 0.33 and 0.34

    We’ve just released two new versions of Alaveteli, our open source platform for running Freedom Of Information sites — one which we’ve packed all of bug fixes, new features and performance improvements into, and one with some important technical updates. Here are some of the highlights.

    Making your first request easier

    mySociety’s designer Zarino has subtly redesigned the authority search page – which is the first thing most users will see when making a request – to make it easier to find the help links at a glance, making the process feel much more approachable.

    Before:

    Find authority page before new design

    After:

    Find authority page after new design

    Encouraging better quality requests

    Part of our advice to users of WhatDoTheyKnow is to keep requests concise and focused. This new feature adds a visual reminder when the request text is approaching the point of being too long for an FOI officer to answer it efficiently:

    Request text getting long warning message

    With a second, stronger warning if the request becomes longer still:

    Request text now very long warning message

    We’ve also added an experimental feature on WhatDoTheyKnow to discourage users from making requests for personal information. (As well as not being valid FOI requests, publishing the responses would result in revealing confidential information about the requester.) By asking the user to clarify what they want to ask before sending a request to authorities which attract a high level of personal information requests, we can try to redirect them to the correct official channel instead:

    Checking whether user is request personal information

    One-click unsubscribe from notifications

    An improvement for site owners and end users: it is now possible to unsubscribe from email notifications about requests that you’re following right from your inbox! (Previously – and still the case with other sorts of email that the site sends – you would have to visit the site and log in to change your email preferences.)

    Allowing people to unsubscribe with ease – if they’ve forgotten that they signed up for notifications or have created a new track by mistake – should help to cut down on the number of messages being marked as spam which should in turn help improve the site’s mail sender reputation. (It also allows admins – if they receive feedback loop notification emails from email providers – to unsubscribe users who’ve marked these emails as spam, preventing further unwelcome emails being sent.)

    Technical changes

    In version 0.33, as well as our our features and fixes, we’ve added support for Ubuntu releases Xenial and Bionic and withdrawn support for Debian Jessie. The full list of highlights and upgrade notes for this release is in the changelog.

    Version 0.34 contains our Rails 5.0 upgrade work which we’ve released separately to allow reusers time to adapt to new minimum requirements for operating systems and Ruby versions.
    We’re no longer supporting Ubuntu Trusty and have also dropped Ruby versions older than 2.3 as that’s the minimum requirement for Rails 5. Upgrade notes are available in the changelog and, as ever, we recommend updating one version at a time to make sure everything’s working smoothly and reduce the risk of missing essential upgrade steps.

    Moving to Rails 5.0 will allow us to retain support for major security issues when Rails 6 is released and dropping older Ruby versions removes some key technical barriers to modernising the Alaveteli codebase and allows us to focus on improving Alaveteli Pro so that it can be reused more widely.

    Thanks again to everyone who’s contributed! Special thanks to Nigel Jones and Laurent Savaëte who contributed bug fixes for version 0.33!

     

    Image: Landscape by mike138 (CC by-nd/2.0)

  3. Alaveteli Release 0.32

    We’ve just released version 0.32 of Alaveteli, our open source platform for running Freedom Of Information sites. Here are some of the highlights.

    Making correspondence threads easier to navigate

    Thanks to our designers, it’s now possible to collapse individual messages in a correspondence thread in order to focus on just the parts you’re trying to read. Plus you can quickly collapse (or expand) all the messages in the thread using the “Collapse all” and “Expand all” links from the “Actions” menu.

    Alaveteli Pro users gain the additional benefit of a redesigned sidebar which allows for easier navigation of lengthy correspondence and avoids having to scroll to the top of the request thread to update its status. See Martin’s full explanation here.

    Better password security

    We’ve started enforcing stricter password length constraints wherever a password is set or updated to help users keep their accounts secure. And we’re also using a stronger encryption method for storing password data, using bcrypt rather than the older SHA1 algorithm to obscure the actual password. (Be sure to run the rake task documented in the release upgrade notes to upgrade secure password storage for all existing users.)

    You can read more about what this does and why it’s important if you’re interested in the technical details behind this upgrade.

    Authorities not subject to FOI law

    We’ve adopted WhatDoTheyKnow’s foi_no tag for authorities to indicate that although the authority is listed on the site, it is not legally subject to FOI law. This could be for advocacy purposes – if it’s felt an authority should be covered by legislation – or where the authority has agreed to respond on a voluntary basis.

    Adding the foi_no tag now causes an extra message to appear under the authority’s name on their page and on all related requests, and removes language about legal responsibilities to reply from the messages sent to users.

    To improve the UI, we’ve made a similar change for authorities with the eir_only tag to make it clearer that such authorities are only accepting requests about the environment.

    (Don’t worry admins, you don’t need to remember all this – we’ve updated the documentation on the edit page to reflect the new functionality!)

    Improvements for site admins

    We’ve made it easier for admins to ban users who sign up to post spam links in their profile. There’s now a “Ban for spamming” button which is available on the user edit page or as soon as you expand the user’s details in the listing rather than having to manually edit user metadata.

    We’ve also made it harder to leave requests flagged as vexatious (or “not_foi”) in an inconsistent state. Previously the site just assumed that vexatious requests would always be hidden. Now the admin interface enforces the hiding of vexatious requests by showing warnings when a request is set as vexatious while it’s visible on the site, and prevents the updated request from being saved until a valid state is selected.

    Announcements

    And last but not least – introducing the new Announcements feature!

    Easier popup banner management

    Site admins will be relieved to hear that they can now update the popup banner message on the site without needing to schedule developer time.

    This feature supports multi-language sites so if you set the announcement for your main (default) language, it will appear across all language versions that you have not added a specific translation for.

    Admin-only announcements

    You can set announcements that will only be seen by fellow administrators when they visit the summary page. (If you’re running a Pro site, you can also have announcements that will only be seen by your Pro admins.)

    Pro announcements

    Announcements for Pro users appear as a carousel at the top of their dashboard. So far we’ve used it on WhatDoTheyKnow Pro to publicise new features, offer discount codes, and encourage people to share their published stories with us.


    The full list of highlights and upgrade notes for this release is in the changelog.

    Thanks again to everyone who’s contributed!

  4. Seeing spots with FixMyStreet Pro

    If you visit FixMyStreet and suddenly start seeing spots, don’t rush to your optician: it’s just another feature to help you, and the council, when you make a report.

    In our last two blog posts we announced Buckinghamshire and Bath & NE Somerset councils’ adoption of FixMyStreet Pro, and looked at how this integrated with existing council software. It’s the latter which has brought on this sudden rash.

    At the moment, you’ll only see such dots in areas where the council has adopted FixMyStreet Pro, and gone for the ‘asset locations’ option: take a look at the Bath & NE Somerset installation to see them in action.

    What is an asset?

    mySociety developer Struan explains all.

    Councils refer to ‘assets’; in layman’s language these are things like roads, streetlights, grit bins, dog poo bins and trees. These assets are normally stored in an asset management system that tracks problems, and once hooked up, FixMyStreet Pro can deposit users’ reports directly into that system.

    Most asset management systems will have an entry for each asset and probably some location data for them too. This means that we can plot them on a map, and we can also include details about the asset.

    When you make a report, for example a broken streetlight, you’ll be able to quickly and easily specify that precise light on the map, making things a faster for you. And there’s no need for the average citizen to ever know this, but we can then include the council’s internal ID for the streetlight in the report, which then also speeds things up for the council.

    Map layers

    So, how do we get these assets on to the map? Here’s the technical part:

    The council will either have a map server with a set of asset layers on it that we can use, or they’ll provide us with files containing details of the assets and we’ll host them on our own map server.

    The map server then lets you ask for all the streetlights in an area and sends back some XML with a location for each streetlight and any associated data, such as the lamppost number. Each collection of objects is called a layer, mostly because that’s how mapping software uses them. It has a layer for the map and then adds any other features on top of this in layers.

    Will these dots clutter up the map for users who are trying to make a report about something else?

    Not at all.

    With a bit of configuration in FixMyStreet, we associate report categories with asset layers so we only show the assets on the map when the relevant category is selected.

    We can also snap problem reports to any nearby asset which is handy for things like street lights as it doesn’t make sense to send a report about a broken street light with no associated light.

    Watch this space

    And what’s coming up?

    We’re working to add data from roadworks.org, so that when a user clicks on a road we’ll be able to tell them if roadworks are happening in the near future, which might have a bearing on whether they want to report the problem — for example there’s no point in reporting a pothole if the whole road is due to be resurfaced the next week.

    Then we’ll also be looking at roads overseen by TfL. The issue with these is that while they are always within a council area, the council doesn’t have the responsibility of maintaining them, so we want to change where the report is going rather than just adding in more data. There’s also the added complication of things like “what if the issue is being reported on a council-maintained bridge that goes over a TFL road”.

    There’s always something to keep the FixMyStreet developers busy… we’ll make sure we keep you updated as these new innovations are added.

    From a council and interested in knowing more? Visit the FixMyStreet Pro website

  5. Training an AI to generate FixMyStreet reports

    Artificial intelligence and machine learning seem to be everywhere at the moment – every day there’s a new story about the latest smart assistant, self-driving car or the impending take over of the world by robots. With FixMyStreet having recently reached one million reports, I started wondering what kind of fun things could be done with that dataset.

    Inspired by a recent post that generated UK place names using a neural network, I thought I’d dip my toes in the deep learning sea and apply the same technique to FixMyStreet reports. Predictably enough the results are a bit weird.

    I took the titles from all the public reports on fixmystreet.com as the training data, and left the training process to run overnight. The number crunching was pretty slow and the calculations had barely reached 5% in the morning. I suspect the training set was a bit too large, at over 1M entries, but end result still gives enough to work with.

    The training process produces checkpoints along the way, which you can use to see how the learning is progressing. After 1000 iterations the model was starting to be aware that it should use words, but didn’t really know how to spell them:

    Mertricolbes
    Ice does thrown campryings
    Sunky riking proper, badger verwappefing cars off uping is!
    Finst Knmp
    Lyghimes Jn fence
    Moadle bridge is one descemjop
    

    After 15000 iterations it’s starting to get the hang of real words, though still struggling to form coherent sentences.

    Untaxed cacistance.
    Broken Surface in ARRUIGARDUR. Widdy movering
    Cracked already nail some house height avenue.
    Light not worky
    I large pot hole
    Dumped shood road nod at street.
    Grim
    Dog man
    Ongorently obstructing sofas. This birgs.
    Serious Dirches
    

    After 68000 iterations there seems to be enough confusion in the training data that things start to go south again with the default parameters:

    Urgely councille at jnc swept arobley men.
    They whention to public bend to street? For traffic light not working
    

    Tweaking the ‘temperature’ of the sampling process produces increasingly sensible results:

    Large crumbling on pavement
    Potholes all overgrown for deep pothole
    Very van causing the road
    Very deep potholes on pavement
    Weeds on the pavement
    Several potholes in the road
    Rubbish Dumped on the road markings
    Potholes on three away surface blocking my peride garden of the pavement
    Potholes and rubbish bags on pavement
    Poor road sign damaged
    Poor street lights not working
    Dog mess in can on road bollard on pavement
    A large potholes and street light post in middle of road
    

    As well as plenty of variations on the most popular titles:

    Pot hole
    Pot hole on pavement
    Pot holes and pavement around
    Pot holes needings to path
    Pothole
    Pothole dark
    Pothole in road
    Pothole/Damaged to to weeks
    Potholes
    Potholes all overgrown for deep pothole
    Potholes in Cavation Close
    Potholes in lamp post Out
    Potholes in right stop lines sign
    Potholes on Knothendabout
    Street Light
    Street Lighting
    Street light
    Street light fence the entranch to Parver close
    Street light not working
    Street light not working develter
    Street light out opposite 82/00 Tood
    Street lights
    Street lights not working in manham wall post
    Street lights on path
    Street lights out
    

    It also seems to do quite well at making up road names that don’t exist in any of the original reports (or in reality):

    Street Light Out - 605 Ridington Road
    Signs left on qualing Road, Leave SE2234
    4 Phiphest Park Road Hasnyleys Rd
    Apton flytipping on Willour Lane
    The road U6!
    

    Here are a few of my favourites for their sheer absurdity:

    Huge pothole signs
    Lack of rubbish
    Wheelie car
    Keep Potholes
    Mattress left on cars
    Ant flat in the middle of road
    Flytipping goon!
    Pothole on the trees
    Abandoned rubbish in lane approaching badger toward Way ockgatton trees
    Overgrown bush Is broken - life of the road.
    Poo car
    Road missing
    Missing dog fouling - under traffic lights
    

    Aside from perhaps generating realistic-looking reports for demo/development sites I don’t know if this has any practical application for FixMyStreet, but it was fun to see what kind of thing is possible with not much work.


    Your donations keep our sites running.
    Donate now


    Image: Scott Lynch (CC by/2.0)

  6. Alaveteli Release 0.26

    We’ve just released Alaveteli 0.26! Here are some of the highlights.

    Request page design update

    After some research in to where people enter the site we decided to revamp the request pages to give a better first impression.

    Alaveteli 0.26 request page redesign

    We’ve used the “action bar” pattern from the authority pages to move the request actions to a neater drop-down menu. We’ve also promoted the “follow” button to help other types of users interact with the site.

    Alaveteli release 0.26 request page action menu

    Since lots of users are entering an Alaveteli on the request pages, it might not be obvious that they too can ask for information. We’ve now made an obvious link to the new request flow from the sidebar of the request pages to emphasise this.

    The correspondence bubbles have had a bit of a makeover too. Its now a lot more obvious how to link to a particular piece of correspondence, and we’ve tidied the header so that its a little clearer who’s saying what.

    The listing of similar requests in the request page sidebar has been improved after observing they were useful to users.

    Also in design-world we’ve added the more modern request status icons, made the search interfaces more consistent and helped prevent blank searches on the “Find an authority” page.

    Admin UI Improvements

    As an Alaveteli grows it can get trickier to keep an eye on everything that’s happening on the site.

    We’ve now added a new comments list so that admins can catch offensive or spam comments sooner.

    Alaveteli Release 0.26 admin comments page

    For the same reasons, we’ve added sorting to the users list and made banned users more obvious.

    Alaveteli Release 0.26 admin users list

    The CSV import page layout and inline documentation has also been updated.

    More stats!

    The wonderful team at the OpenAustralia Foundation contributed an epic pull request to revamp the statistics page.

    The new statistics page adds contributor leaderboards to help admins identify users as potential volunteers, as well as a graph showing when site admins hide things to improve the transparency of the site.

    Alaveteli Release 0.26 leaderboard stats Alaveteli Release 0.26 hide events

    Extra search powers

    As if that wasn’t enough, the the OpenAustralia Foundation team also added a new advanced search term to be able to find requests to public authorities with a given tag.

    Conversion tracking improvements

    We’re constantly asking how we can improve Alaveteli. In order to answer that question meaningfully, we need good tracking of what happens when people make requests on the site. When we tried to track down whether people coming from social media sites were more likely to make a request than those coming from other sources, we found a problem with the way we tracked ‘conversions’ – the process of getting all the way through making a request. We were using a particular URL to mark the end of that process in Google Analytics. The issue was that sometimes, requesters would share that URL with other people, causing us to record the same request being made multiple times. We’re now using a little bit of javascript to make sure we only record the conversion when a new request is actually being made.

    The full list of highlights and upgrade notes for this release is in the changelog.

    Thanks again to everyone who’s contributed!

  7. Better-looking emails for FixMyStreet

    If you’ve used FixMyStreet recently — either to make a report, or as a member of a council who receives the reports — you might have noticed that the site’s automated emails are looking a lot more swish.

    Where previously those emails were plain text, we’ve now upgraded to HTML, with all the design possibilities that this implies.

    It’s all part of the improvements ushered in by FixMyStreet Version 2.0, which we listed, in full, in our recent blog post. If you’d like a little more technical detail about some of the thought and solutions that went into this switch to HTML, Matthew has obliged with in a blog post over on FixMyStreet.org.

  8. Introducing EveryPolitician’s bot

    mySociety’s EveryPolitician project aims to make data available on every politician in the world. It’s going well: we’re already sharing data on the politicians from nearly every country on the planet. That’s over 68,652 people and 2.9 million individual pieces of data, numbers which will be out of date almost as soon as you’ve read them. Naturally, the width and depth of that data varies from country to country, depending on the sources available — but that’s a topic for another blog post.

    Today the EveryPolitician team would like to introduce you to its busiest member, who is blogging at EveryPolitician bot. A bot is an automated agent — a robot, no less, albeit one crafted entirely in software.

    First, some background on why we need our little bot.

    Because there’s so much to do

    One of the obvious challenges of such a big mission is keeping on top of it all. We’re constantly adding and updating the data; it’s in no way a static dataset. Here are examples — by no means exhaustive — of circumstances that can lead to data changes:

    • Legislatures change en masse, because of elections, etc.
      We try to know when countries’ governments are due to change because that’s the kind of thing we’re interested in anyway (remember mySociety helps run websites for parliamentary monitoring organisations, such as Mzalendo in Kenya). But even anticipated changes are rarely straightforward, not least because there’s always a lag between a legislature changing and the data about its new members becoming available, especially from official national sources.
    • Legislatures change en masse, unexpectedly
      Not all sweeping changes are planned. There are coups and revolutions and other unscheduled or premature ends-of-term.
    • Politicians retire
      Or die, or change their names or titles, or switch party or faction.
    • New parties emerge
      Or the existing ones change their names, or form coalitions.
    • Areas change
      There are good reasons (better representation) and bad reasons (gerrymandering) why the areas in constituency-based systems often change. By way of a timely example, our UK readers probably know that the wards have changed for the forthcoming elections, and that mySociety built a handy tool that tells you what ward you’re in.
    • Existing data gets refined
      Played Gender Balance recently? Behind that is a dataset that keeps being updated (whenever there are new politicians) but which is itself a source of constantly-updating data for us.
    • Someone in Russia updates the wikipedia page about a politician in Japan
      Wikidata is the database underlying projects like Wikipedia, so by monitoring all the politicians we have that are also in there, we get a constant stream of updates. For example, within a few hours of someone adding it, we knew that the Russian transliteration of 安倍晋三’s name was Синдзо Абэ — that’s Shinzo Abe, in case you can’t read kanji or Cyrillic script. (If you’re wondering, whenever our sources conflict, we moderate in favour of local context.)
    • New data sources become available
      Our data comes from an ever-increasing number of sources, commonly more than one for any given legislature (the politicians’ twitter handles are often found in a different online place from their dates of birth, for example). We always welcome more contributions — if you think you’ve got new sources for the country you live in, please let us know.
    • New old data becomes available
      We collect historic data too — not just the politicians in the current term. For some countries we’ve already got data going back decades. Sources for data like this can sometimes be hard to find; slowly but surely new ones keeping turning up.

    So, with all this sort of thing going on, it’s too much to expect a small team of humans to manage it all. Which is where our bot comes in.

    Hello bot

    portrait of bot

    To be honest with you, the bot doesn’t really look like this because, being software, it’s entirely non-corporeal. Sorry.

    We’ve automated many of our processes: scraping, collecting, checking changes, submitting them for inclusion — so the humans can concentrate on what they do best (which is understanding things, and making informed decisions). In technical terms, our bot handles most things in an event-driven way. It springs into action when triggered by a notification. Often that will be a webhook (for example, a scraper finishes getting data so it issues a webhook to let the bot know), although the bot also follows a schedule of regular tasks too. Computers are great for running repetitive tasks and making quantitative comparisons, and a lot of the work that needs to be done with our ever-changing data fits such a description.

    The interconnectedness of all the different tasks the bot performs is complex. We originally thought we’d document that in one go — there’s a beautiful diagram waiting to be drawn, that’s for sure — but it soon became clear this was going to be a big job. Too big. Not only is the bot’s total activity complicated because there are a lot of interdependencies, but it’s always changing: the developers are frequently adding to the variety of tasks the bot is doing for us.

    So in the end we realised we should just let the bot speak for itself, and describe task-by-task some of the things it does. Broken down like this it’s easier to follow.

    We know not everybody will be interested, which is fine: the EveryPolitician data is useful for all sorts of people — journalists, researchers, parliamentary monitors, activists, parliamentarians themselves, and many more — and if you’re such a person you don’t need to know about how we’re making it happen. But if you’re technically-minded — and especially if you’re a developer who uses GitHub but hasn’t yet used the GitHub API as thoroughly as we’ve needed to, or are looking for ways to manage always-shifting data sets like ours — then we hope you’ll find what the bot says both informative and useful.

    everypoliticianbot-small
    The bot is already a few days into blogging — its first post was “I am a busy bot”, but you can see all the others on its own Medium page. You can also follow it on Twitter as @everypolitician. Of course, its true home, where all the real work is done, is the everypoliticianbot account on GitHub.


    Images: CC-BY-SA from the EveryPolitician bot’s very own scrapbook.

  9. Unpicking code, for the good of the project

    Last year, when we were helping to develop YourNextMP, the candidate-crowdsourcing platform for the General Election, we made what seemed like an obvious decision.

    We decided to use PopIt as the site’s datastore — the place from which it could draw information about representatives: their names, positions, et cetera. We’d been developing PopIt as a solution for parliamentary monitoring sites, but we reckoned it would also be a good fit for YourNextMP.

    That turned out to be the wrong choice.

    YourNextMP was up and running in time for the election, but at the cost of many hours of intensive development as we tried to make PopIt do what was needed for the site.

    Once you’ve got an established site in production, changing the database it uses isn’t something you do lightly. But on returning to the codebase to develop it for international reuse, we had to admit that, in the words of mySociety developer Mark Longair, PopIt was “actually causing more problems than it was solving”. It was time to unpick the code and take a different approach.

    Mark explains just what it took to decide to change course in this way, over on his own blog.

    The post contains quite a bit of technical detail, but it’s also an interesting read for anyone who’s interested in when, and why, it’s sometimes best to question the decisions you’ve made.

     

    Image: Michelle (CC)

  10. Photo upload and progressive enhancement for FixMyStreet

    FixMyStreet has been around for nearly nine years, letting people report things and optionally include a photo; the upshot of which is we currently have a 143GB collection of photographs of potholes, graffiti, dog poo, and much more. 🙂

    For almost all that time, attaching a photo has been through HTML’s standard file input form; it works, but that’s about all you can say for it – it’s quite ugly and unfriendly.

    We have always wanted to improve this situation – we have a ticket in our ticketing system, Display thumbnail of photo before submitting it, that says it dates from 2012, and it was probably in our previous system even before that – but it never quite made it above other priorities, or when it was looked at, browser support just made it too tricky to consider.

    Here’s a short animation of FixMyStreet’s new photo upload, which also allows you to upload multiple photos:

    For the user, the only difference from the current interface is that the photo field has been moved higher up the form, so that photos can be uploading while you are filling out the rest of the form.

    Personally, I think this benefit is the largest one, above the ability to add multiple photos at once, or the preview function. Some of our users are on slow connections – looking at the logs I see some uploads taking nearly a minute – so being able to put that process into the background hopefully speeds up the submission and makes the whole thing much nicer to use.

    Progressive enhancement

    When creating a new report, it can sometimes happen that you fill in the form, include a photo, and submit, only for the server to reject your report for some reason not caught client-side. When that happens, the form needs to be shown again, with everything the user has already entered prefilled.

    There are various reasons why this might happen; perhaps your browser doesn’t support the HTML5 required attribute (thanks Safari, though actually we do work around that); perhaps you’ve provided an incorrect password.

    However, browsers don’t remember file inputs, and as we’ve seen, photo upload can take some time. From FixMyStreet’s beginnings, we recognised that re-uploading is a pain, so we’ve always had a mechanism whereby an uploaded photo would be stored server side, even if the form had errors, and only an ID for the photo was passed back to the browser so that the user could hopefully resubmit much more quickly.

    This also helped with reports coming in from external sources like mobile phone apps or Flickr, which might come with a photo already attached but still need other information, such as location.

    Back in 2011, I wrote about how FixMyStreet maps are progressively enhanced, starting with a base of HTML image maps and layering on the normal slippy map experience on top. This has always been the way I have worked, and adding a snazzy JavaScript photo upload was no different.

    mySociety designer Zarino used dropzonejs to supply the “pop”™, and this works in a nicely easy-to-progressively-enhance way, hiding existing file input(s) and providing fallbacks. And with the behaviour the site has had since 2007, adding the server side element of this new photo upload was actually very straightforward – receive a photo and return its ID for a snippet of JavaScript to insert into the hidden form field of photo ID that has always been there in case of form error. No need to worry about how to match up the out-of-band photos with the main form submission, it’s all already taken care of. If the JavaScript doesn’t or can’t work for whatever reason, the old behaviour is still there, using the same mechanisms.

    Of course there were edge cases and things to tidy up along the way, but if the form hadn’t taken into account the user experience of error edge cases from the start, or worse, had assumed all client checks were enough, then nine years down the line my job would have been a lot harder.

    Anyway, long story short, adding photos to your FixMyStreet reports is now a smoother process, and you should try it out.