1. Unpicking code, for the good of the project

    Last year, when we were helping to develop YourNextMP, the candidate-crowdsourcing platform for the General Election, we made what seemed like an obvious decision.

    We decided to use PopIt as the site’s datastore — the place from which it could draw information about representatives: their names, positions, et cetera. We’d been developing PopIt as a solution for parliamentary monitoring sites, but we reckoned it would also be a good fit for YourNextMP.

    That turned out to be the wrong choice.

    YourNextMP was up and running in time for the election, but at the cost of many hours of intensive development as we tried to make PopIt do what was needed for the site.

    Once you’ve got an established site in production, changing the database it uses isn’t something you do lightly. But on returning to the codebase to develop it for international reuse, we had to admit that, in the words of mySociety developer Mark Longair, PopIt was “actually causing more problems than it was solving”. It was time to unpick the code and take a different approach.

    Mark explains just what it took to decide to change course in this way, over on his own blog.

    The post contains quite a bit of technical detail, but it’s also an interesting read for anyone who’s interested in when, and why, it’s sometimes best to question the decisions you’ve made.

     

    Image: Michelle (CC)

  2. Photo upload and progressive enhancement for FixMyStreet

    FixMyStreet has been around for nearly nine years, letting people report things and optionally include a photo; the upshot of which is we currently have a 143GB collection of photographs of potholes, graffiti, dog poo, and much more. :)

    For almost all that time, attaching a photo has been through HTML’s standard file input form; it works, but that’s about all you can say for it – it’s quite ugly and unfriendly.

    We have always wanted to improve this situation – we have a ticket in our ticketing system, Display thumbnail of photo before submitting it, that says it dates from 2012, and it was probably in our previous system even before that – but it never quite made it above other priorities, or when it was looked at, browser support just made it too tricky to consider.

    Here’s a short animation of FixMyStreet’s new photo upload, which also allows you to upload multiple photos:

    For the user, the only difference from the current interface is that the photo field has been moved higher up the form, so that photos can be uploading while you are filling out the rest of the form.

    Personally, I think this benefit is the largest one, above the ability to add multiple photos at once, or the preview function. Some of our users are on slow connections – looking at the logs I see some uploads taking nearly a minute – so being able to put that process into the background hopefully speeds up the submission and makes the whole thing much nicer to use.

    Progressive enhancement

    When creating a new report, it can sometimes happen that you fill in the form, include a photo, and submit, only for the server to reject your report for some reason not caught client-side. When that happens, the form needs to be shown again, with everything the user has already entered prefilled.

    There are various reasons why this might happen; perhaps your browser doesn’t support the HTML5 required attribute (thanks Safari, though actually we do work around that); perhaps you’ve provided an incorrect password.

    However, browsers don’t remember file inputs, and as we’ve seen, photo upload can take some time. From FixMyStreet’s beginnings, we recognised that re-uploading is a pain, so we’ve always had a mechanism whereby an uploaded photo would be stored server side, even if the form had errors, and only an ID for the photo was passed back to the browser so that the user could hopefully resubmit much more quickly.

    This also helped with reports coming in from external sources like mobile phone apps or Flickr, which might come with a photo already attached but still need other information, such as location.

    Back in 2011, I wrote about how FixMyStreet maps are progressively enhanced, starting with a base of HTML image maps and layering on the normal slippy map experience on top. This has always been the way I have worked, and adding a snazzy JavaScript photo upload was no different.

    mySociety designer Zarino used dropzonejs to supply the “pop”™, and this works in a nicely easy-to-progressively-enhance way, hiding existing file input(s) and providing fallbacks. And with the behaviour the site has had since 2007, adding the server side element of this new photo upload was actually very straightforward – receive a photo and return its ID for a snippet of JavaScript to insert into the hidden form field of photo ID that has always been there in case of form error. No need to worry about how to match up the out-of-band photos with the main form submission, it’s all already taken care of. If the JavaScript doesn’t or can’t work for whatever reason, the old behaviour is still there, using the same mechanisms.

    Of course there were edge cases and things to tidy up along the way, but if the form hadn’t taken into account the user experience of error edge cases from the start, or worse, had assumed all client checks were enough, then nine years down the line my job would have been a lot harder.

    Anyway, long story short, adding photos to your FixMyStreet reports is now a smoother process, and you should try it out.

  3. Now you can reverse hasty decisions—on Gender Balance, at least

    As players were quick to notice, decisions made on our politician-sorting game Gender Balance were final. Thanks to volunteer coder Andy Lulham, that’s now been rectified with an ‘undo’ button.

    Gender Balance is our answer to the fact that there’s no one source of gender information across the world’s legislatures—read more about its launch here. It serves up a series of politicians’ names and images, and asks you to identify the gender for each. Your responses, along with those of other players, helps compile a set of open data that will be available to all.

    Many early players told us, however, that it’s all too easy to accidentally click the wrong button. (The reasons for this may be various, but we can’t help thinking that it’s often because there are so many males in a row that the next female comes as a bit of a surprise…)

    In fact, this shouldn’t matter too much, because every legislature is served up to multiple players, and over time any anomalies will be ironed out of the data. That doesn’t stop the fact that it’s an upset to the user, though, and in the site’s first month of existence, an undo button has been the most-requested feature.

    genderbalance-undo-1

    Thanks to the wonders of open source, anyone can take the code and make modifications or improvements, and that’s just what Andy did in this case. He submitted this pull request (if you look at that, you can see the discussion that followed with our own developers and our designer Zarino). We’ve merged his contribution back into the main code so all players will now have the luxury of being able to reverse a hasty decision. Thanks, Andy!

     

     

    Photo credit: Head in Hands CC BY-NC 2.0 by Alex Proimos

  4. We’ve analysed every vote since the general election, and why that’s good news for you

    360 degree panorama by Eye of QvoxSince the 2010 election, there have been 1,085 votes in the House of Commons.

    Our team member Richard has now analysed every single one of those votes, and his findings have been added to each MP’s information on TheyWorkForYou.

    We hope that’s great news for users: it means that we can now present a really full picture of how your MP voted on key topics.

    It’s also potentially useful for developers, eDemocracy hackers and campaign groups, who can pick up our data and use it as they please.

    So what exactly is the data?

    Often MPs vote on motions which are, at first glance, rather incomprehensible and cryptic. They might vote for example on a motion to accept:

    Amendments (a) to (d) proposed in lieu of Lords amendments 1 to 4 and 6.

    We’ve done the research to determine what MPs were actually voting on in each case, and turned their archaic language into plain English.

    For every vote we’ve written a sentence to describe the effect of voting either “aye” or “no”. In relation to one MP’s vote on the evening of the 9th of July 2014 we write:

    Mark Pawsey MP, Rugby voted for a residence test as an eligibility criteria for civil legal aid; subject to exceptions for refugees and those who have sought asylum.

    In addition to describing every vote, we have decided whether it should be considered relevant to the topics we list on each MP’s page (see an example MP here, or check your own MP by inputting your postcode on the homepage, then clicking ‘voting record’ on your MP’s page).

    If a vote was relevant to one of the statements we show on TheyWorkForYou, we then determined whether voting ‘aye’ or ‘no’ was a vote for or against the statement and if the vote was very important, or less important. By clicking on the green ‘details’ button beside each statement on an MP’s voting record you can see exactly which individual votes contributed to it as well as how we calculated which wording such as “moderately for” or “strongly against” to apply in each case.

    Matters MPs have voted on since the 2010 general election have ranged from bankers’ bonuses to same sex marriage; from food banks to the “bedroom tax” (all of which have contributed to statements we show on TheyWorkForYou); from daylight saving to the regulation of hairdressers (neither included) – and plenty more. (We’ve written previously about how we select which topics to show on TheyWorkForYou.)

    Of course, Parliament continues to hold votes, and we’ll be continuing to analyse the results as they come in – but it is good to know that we are bang up to date.

    How can this data be used?

    We have plenty of ideas ourselves, and we want to hear yours, too. With the forthcoming general election, one obvious use is for ‘who should I vote for?’ tools, which match users’ opinions with those of each party.

    There’s also potential for comparisons between what constituents believe and what their elected representative has voted for.

    No doubt there are many other ideas that haven’t even occurred to us yet – please do get in touch if you have ideas and you’d like to use this data.

    Image: Eye of Qvox (cc)

     

  5. Open Source: trying to make it more open

    Hopeful Helping Hands by Matt Katzenberger

    This month we released a new version of FixMyStreet. Amongst the new features, fixes, and thingamajigs were some small improvements added by two volunteers, Andrew and Andy.

    Even though these are not core pieces of functionality — in fact, precisely because they are not — we want to draw your attention to why they were included, and why this is a Very Good Thing. And perhaps, if you’re a coder who wants to put something into an Open Source project but hasn’t quite found a way in, Andrew and Andy’s work will nudge you into becoming a contributor too.

    One of the axioms of Open Source is that, because anyone can read the source code, in theory anyone can contribute to it. In practice, though, it’s not really as simple as that. Both ends of the “anyone can contribute” idea require effort:

    • Before contributing to a project of any complexity (as we hope you want to do), there’s often a lot to learn, or figure out, before any work can even begin;
    • Before accepting contributions to such a project (keen as we are to do so), there’s an overhead of testing, checking, and managing the incoming code.

    The ugly real world

    The basic issue here is that software is complex — no matter how well-written, tested and documented program code is, if the problem it’s solving is in the real world, it’s not going to be simple.

    This is especially true of anything used by the public, because often you can only make things seem simple at the front (such as a clean web interface or “user journey”) by working hard behind the scenes with data structures and processes that handle the underlying complexity. It’s inevitably true of any projects which have been developed over time — programmers like to use the term “legacy code” to describe anything that wasn’t written then way they’d choose to write it now.

    Often the problems that software is solving are not quite as obvious as they first appear. At mySociety we’ve got a wealth of experience and actual usage data that ultimately changes the way we build, and develop, our platforms. We understand the fields we work in well (technically, the nerds like to call these the “problem domain”), whether it’s governmental practice or civic user behaviour, and that’s often knowledge that’s not encapsulated anywhere in the program code.

    Furthermore, any established platform must protect against the risk that new changes break old behaviour — something that regression testing is designed to catch. This is especially important on platforms like FixMyStreet or Alaveteli where the software is already running in multiple installations.

    This is why we have a team of full-time, experienced, and (thanks to our rigorous recruitment process) talented programmers who can invest the time and effort to be familiar with all these things when they set to work coding.

    But this builds up to an impediment: sensitivity to any of these issues is enough to make anyone think twice about simply forking our code and starting to hack on it for us.

    How it sometimes works

    In practice, then, how does anything get contributed? How come it doesn’t all get written by our own coders?

    The answer is, of course, we do work with major contributors outside our own team (have a look at the activity on our github repos to see them) — but it always requires a period of support and on-line discussion both before and during the process. There’s also the business of testing, and sometimes politely pushing back on, pull requests (which is how code contributions are submitted). But the fact of the matter is that this is only possible for people who are willing to spend time familiarising themselves with the specific code, technologies, and practices that we’re using on that project. These tend to be hard-code devs, and — here’s the point — they’re always experienced Open-Sourcers: this will never be the first time they’ve worked on such a project.

    Which is where the little features come in.

    The joy of small

    We noticed this problem — that contributing code to our projects is simply not easy for us or for contributors. Importantly, it’s not just us: it’s Open Source everywhere. But we can’t simply dismiss the opportunity for contribution. We want to encourage coders to do this, because we believe that Open Source is intrinsically a good thing.

    We do two things to make it easier to contribute:

    • We identify small features that a coder can approach without a full understanding of the code and the problem domain;
    • We help people (like you!) get started by opening up a laptop at our weekly meetups.

    The first of these seems obvious now: when we add issues (an idea for a new feature, or maybe a bugfix) to our github repos, if we think they’re candidates for manageable, isolated work, we tag them with the label: Suitable for volunteers (like this).

    Often these turn out to be “nice-to-haves” that one of our full-time devs can’t be pulled off more pressing problems to add just now. (Case in point: Andrew added a date-picker to the FixMyStreet admin stats page, and three of our own staff had stumbled upon and applauded the difference it had made within a week of it going live).

    It means it’s much easier for you to get involved, because often it’s a little, isolated piece of code. And it’s much more manageable for us, because the change you’ll be submitting is also isolated.

    So if you’re looking for something to tackle, pick one of those issues, and let us know (just to check nobody else has baggsied it already). Fork the repo, cut the code, write the tests, submit a pull request!

    But wait — if that last paragraph made you gulp, here’s the second thing we do: meetups. Of course, this is less helpful if you can’t make it to London on Wednesdays, but the concept is sound. Put simply, if there is a barrier to entry to diving in, and if one-on-one time with a dev, and some pizza, is what it takes to overcome that, it’s time well spent for you to come and see us.

    Not 100% confident with git? Not sure when db/schema.sql gets used or how we like to handle migrations? No problem: we’re happy to guide you.

    You?

    If this has struck a chord with you — you’d love to be an Open Source contributor one day, and you think mySociety projects make the world a better place — perhaps you should take a poke in our repos, or come along to a meetup. Start small, but do start.

    Oh, and Andrew and Andy — thanks guys 😉

    More about volunteering for mySociety

    Photo by Matt Katzenberger (CC)

  6. What we’re doing on TheyWorkForYou

    Image by William WarbyLast week we asked what  improvements you’d like to see on TheyWorkForYou. Thanks so much for all the comments on that post (do keep them coming). They’ve all been carefully documented on our development list.

    Our standard way of working on a project like this is in ‘sprints’ – short periods of activity after which we can spend some time reflecting on what went well, and what could have gone better.

    This system is great for ensuring that we don’t get involved in a large piece of work, only to realise that it doesn’t do what was intended, or hasn’t had the desired effect. So, for example, if we’ve added a new feature, we might be asking ourselves, ‘Is anyone using it?’, ‘Have there been any bug reports?’, and ‘Has it fulfilled our original aim?’. We’re striving to be as analytical and methodical as possible about these assessments, so part of the process has also been figuring out which types of metrics to collect, and how.

    That said, what have we already done?

    It’s easier to find a specific representative

    Where previously our pages listing all MPs, all MSPs and all MLAs just contained one very long list of names that you had to search or scroll through, there’s now an A-Z navigation at the top. We also added the ability to find your own MP from this page.
    AZ navigation on TheyWorkForYou

    Why? This is an example of a small usability tweak which should make a difference to a large number of people – not everyone knows how to search a web page with Ctrl+F. It’s also a fix that’s been on our to-do list for two years!

    The addition of the ‘find your MP’ box helps to serve one of our core aims: to make democracy easy to understand for the uninitiated.

    We’ve added ‘like’ and ‘follow’ buttons

    social media buttons on TheyWorkForYouWe thought you might not notice these discreet additions to our page footers – but we’ve certainly seen an upturn on the rate at which people are ‘liking’ our Facebook page. Whereas Twitter – not so much. Maybe TheyWorkForYou users are just more Facebook-inclined?

    Why? In part, this addition is for our own benefit – we welcome the opportunity that social media gives our users to spread the word. As a small organisation with no advertising budget, this kind of grass roots promotion is invaluable. Then, we are hoping that it will help us to understand our users. Clicking that ‘like’ button can be seen as a form of positive affirmation and enagement that it’s very hard to quantify by other means.

    We are still considering the addition of  buttons which would allow you to share specific debates with your social circles.

    We have noted the comments on our last post which made it clear that some of our users do not welcome integration with social media. That’s fine – we’ll never do anything that excludes you from the core activities of the site, whether you use Facebook and Twitter or not – our intention is simply to provide the functionality for those who want it.

    Those comments have been a useful reminder to us that we should continue to consult our users, because we can’t always predict what you might object to!

    You can change your email address

    If you have an account, now you can change your email address yourself.

    Why? This was identified as a common request that often puzzled users, and took up support time on our side.

    MPs’ pages will look better

    You can’t see these yet, because they’re still in progress. Due to some quirks of the code in which the site was originally built, the new design for the MPs’ pages has taken longer to implement than we’d anticipated. But we’re getting there.

    Why?  MPs’ pages contain an awful lot of information, from voting history to recent appearances, and more. The redesign will help us present all this information more clearly, making the page just as easy to read on a mobile device as it is on a desktop, and simply bringing the (frankly, dated) pages a more current look.

    Bullets are bullets

    alert for kittens on TheyWorkForYouThis is almost ridiculous, but we think it was worth attending to. In recent user tests, we noticed some confusion, caused by the fact that our bullet points were in the form of small squares – they were frequently mistaken for check boxes.

    Why? Just to rid the world of that one small piece of frustration that occurs when you try to tick a box that is not, in fact, a checkbox.

    What now?

    As I say, we are still actively collecting and working on your feedback, so please do keep it coming. Comment below this post, or drop us a line on hello@mysociety.org. I’ll be reporting back after our next sprint.

    Photo by William Warby (CC)

  7. mySociety at MozFest 2013

    mozfest from the 9th floor

    We’re good friends with the people at Mozilla. Every Wednesday, they welcome us into their London Moz space for our weekly meet-ups. They are champions of empowering possibilities of the web through Open Source software (a world we’re part of too). And they’re all so smart and lovely. So of course we’d been looking forward to this year’s Mozilla Festival for some time.

    We had a table at the “Science Fair” on Friday night, where we literally had buckets of sweets (OK, they were little plastic buckets). Tom, our director, and Dave, from our international team, talked about mySociety’s work with anyone who came close. Perhaps people were drawn in by those sweets, or the FixMyStreet demo on the monitor, or even the (new!) stickers we had to give away… but regardless of the lure, we think they all learned a little bit more about how our platforms help empower people’s civic lives: from something as simple as reporting a flickering streetlight, to holding a public authority to account, to monitoring a whole parliament. (That’s FixMyStreet, Alaveteli, and Pombola, if you were wondering).

    O2 through Ravensbourne windows

    The Mozilla Festival’s venue was, once again, London’s astonishing Ravensbourne, right next to the O2 Millenium Dome. The setting magnifies the wonder of the event. Those big round windows make it feel like being in a spaceship made of Swiss cheese. The place is so open, and so vertical, that the activity and enthusiasm doesn’t just spread out, it spreads up. There is making and teaching, learning and sharing, going on across nine floors, and it’s easy to drift up and down from one themed space to another.

    We met old friends. We got to hang out some more with our Chilean brothers-in-code from Ciudadano Inteligente, and the excellent Gaba from Uruguay’s DATA, together with the good people from the OKFN. We made lots of new friends too. And all this didn’t just happen at the sessions. A lot of serendipitous encounters took place by the Alchemy coffee stations. Or on the stairs (khun Toy and khun Hui — hi!). Or in the Alphabet City party venue, afterwards.

    So a big “thank you” to that Fiery Fox, and an enthusiastic high five (yes, there was an unLondonlike amount of enthusiasm on show — possibly because quite a few of the attendees were over from the USA — which it is impossible not to be caught up by) to all the people we met at the event. Dave grinned his way through a wonderful Scratch tutorial from Code Club, met a whole array of cool people, got answers to some nerdy coding questions, and learnt about the awesome Hive learning networks… and lots more things besides. That already describes a great weekend. But beyond that, we hope we might see a few new mySociety-powered sites spring up elsewhere in the world due to sparks that were sparked at mozfest last weekend.

  8. Hacknight report

    hack night - Instagram image by Jen Bramley

    It was great to see everyone at the hacknight on Wednesday. This time, ideas ranged from using the FixMyStreet Platform for reporting of Trading Standards transgressions, to installing a Mzalendo for Greece. We hope that these ideas will flourish, and many others will arise at our future hacknights.

    Talking of which, don’t forget to reserve a place if you’d like to come along. Hacknights are weekly. All the details are here.

  9. mySociety Weekly Hack Nights – join in the conversation

    Mountain Rescue Team by BeingFocal
    Thanks to everyone who came to the inaugural mySociety Hack Night – and thanks too to our hosts, the Open Data Institute for such a great space to work in.

    Topics ranged from community-building in post-conflict societies, to mountain rescue in Wales, via an extended front-end for WriteToThem which would put campaigns in context. It really showed what a lot of exciting ideas there are, just waiting for someone to launch into them.

    We’ll be running these nights every Wednesday: we’re currently booking for the following dates, 6:00 -9:00 pm.

    Places are restricted, so drop us a line on hello@mysociety.org if you’d like to be sure of getting in. All you need is a little coding experience and a laptop.

    We’d also like to start a conversation in the comments below, so that like-minded folk can think about hacking together. If you’re looking for people to help you with an idea, or if you see something you like the look of, leave a note below and try to synchronise which nights you’ll be attending.

    Photo by Being Focal (CC)

  10. Google Summer of Code

    A Day Spent Lying in the Grass, Watching the Clouds, by Elaine Millan

    Ah, summer: walks in the park, lazing in the long grass, and the sound of chirping crickets – all overlaid with the clatter of a thousand keyboards.

    That may not be your idea of summer, but it’s certainly the ways ours is shaping up. We’re participating in Google’s Summer of Code, which aims to put bright young programmers in touch with Open Source organisations, for mutual benefit.

    What do the students get from it? Apart from a small stipend, they have a mentored project to get their teeth into over the long summer hols, and hopefully learn a lot in the process. We, of course, see our code being used, improved and adapted – and a whole new perspective on our own work.

    Candidates come from all over the world – they’re mentored remotely – so for an organisation like mySociety, this offers a great chance to get insight into the background, politics and technical landscape of another culture. Ideas for projects that may seem startlingly obvious in, say, Latin America or India would simply never have occurred to our UK-based team.

    This year, mySociety were one of the 180 organisations participating. We had almost 100 enquiries, from countries including Lithuania, India, Peru, Georgia, and many other places. It’s a shame that we were only able to take on a couple of the many excellent applicants.

    We made suggestions for several possible projects to whet the applicants’ appetite. Mobile apps were popular, in particular an app for FixMyTransport. Reworking WriteToThem, and creating components to complement MapIt and PopIt also ranked highly.

    It was exciting to see so many ideas, and of course, hard to narrow them down.

    In the end we chose two people who wanted to help improve our nascent PopIt service. PopIt will allow people to very quickly create a public database of politicians or other figures. No technical knowledge will be needed – where in the past our code has been “Just add developers”, this one is “Just add data”. We’ll host the sites for others to build on.

    Our two successful applicants both had ideas for new websites that would use PopIt for their datastore, exactly the sort of advanced usage we hope to encourage. As well as making sure that PopIt actually works by using it they’ll both be creating transparency sites that will continue after their placements ends. They’ll also have the knowledge of how to set up such a site, and in our opinion that is a very good thing.

    We hope to bring you more details as their projects progress, throughout the long, hot (or indeed short and wet) summer.

    PS: There is a separate micro-blog where we’re currently noting some of the nitty gritty thoughts and decisions that go into building something like PopIt. If you want to see how the project goes please do subscribe!

    Google Summer of Code

    Top image by Elaine Millan, used with thanks under the Creative Commons licence.