mySociety’s Components Strategy – Our take on Small Pieces Loosely Joined

As you may already be aware, mySociety is putting considerable effort into making it super-easy to set up versions of our websites FixMyStreet and WhatDoTheyKnow in other countries.

These ‘websites in a box’ are a key part of our strategy to help people develop more successful civic and democratic websites around the world, but they are only the first half of our plan. Today I wanted to talk about the other half.

The Challenge

There are some use-cases for software in which most people are entirely happy to take some software off the shelf, press ‘Go’, and start using it. WordPress is a good example, and so is Microsoft Office.

However, there are some kinds of social issues that vary so much between different countries and regions that we believe one-size-fits-all tools for attacking them are impracticable.

This problem is particularly acute in the arena of sites and apps that allow people to track the activities of politicians. In this area there are several dozen different sites globally, almost all of which are powered by software that was written bespoke for that particular usage.

What drives this pattern of people re-writing every site from scratch is that people in different places care about different aspects of politics. In some countries what really counts is how politicians vote, in others the crux is campaign finance contributions, in others it is information on who has criminal records, and in others still it is whether public money has been vanishing suspiciously.

To build an off-the-shelf software platform that could handle all this data equally well in every country would be an immense coding task. And more important than that, we believe that it would create a codebase so huge and complex that most potential reusers would run away screaming. Or at least ignore it and start from scratch.

In short – we don’t believe there can be a WordPress for sites that monitor politicians, nor for a variety of other purposes that relate to good governance and stronger democracies.

The Waste

We believe that the wrong answer to this challenge is to just say “Well then, everyone should build their own sites from scratch.” Over the years we at mySociety have been witness to the truly sad sight of people and organisations around the world wearing themselves out and blowing their budgets just trying to get the first version of a transparency website out the door. All too often they fail to create popular, long lasting sites because the birthing process is just so exhausting and resource-consuming that there’s nothing left to drive the sites to success. Often they don’t even get to launch.

A painful aspect of this problem is that the people who work on such sites are genuine altruists who are trying to solve serious problems in their part of the world; too much of their passion and energy is used up on building tools, when there’s still so much work beyond that that’s needed to make such sites successful.  However, as we pointed out above, giving them a complete package on a plate isn’t an option.  So what can we do?

Our Proposed Answer – The Components

We start from the following observation: coders and non-coders like simple, minimal, attractive tools that help them achieve bigger goals. Simple tools don’t make anyone run away screaming – they encourage exploration and deliver little sparkles of satisfaction almost immediately. But simple tools have to be highly interoperable and reliable to form the foundation of complex systems.

Our plan is to collaborate with international friends to build a series of components that deliver quite narrow little pieces of the functionality that make up bigger websites. These include:

  • PopIt – A Component to store and share the names of politicians, and the jobs they have.
  • MapIt – A Component to store and share information on the locations of administrative boundaries, like counties, regions or cities.
  • SayIt – A Component to store and share information on the words that public figures say or put out in writing.
There will be more, possibly many more. Our goal is to radically collapse the time it takes to build new civic and democratic (and possibly governmental) websites and apps, without putting constraints on creativity.

Characteristics of each Component

There are some crucial architecture decisions that have been baked into the Components, to truly make them ‘small pieces loosely joined’.

  • Each Component is fundamentally a tool for storing and sharing one or two kinds of common data – they’re intentionally minimalist.
  • As a developer, you just use the Components that make sense for your goals – you simply don’t have to look at or learn about the Components that contain functionality that doesn’t matter to you.
  • You don’t have to install anything to get started – you can always begin by playing with a hosted Component.
  • We won’t impose our taste in programming languages on you. You can code your website in whatever language you want. The Components are not ‘modules’  - they don’t plug into some overarching framework like Drupal or WordPress. They are stand-alone tools which just present you data over REST APIs, and which you can write data into using REST APIs.
  • Each Component’s data structures will offer as much flexibility as makes sense given the goal of keeping each Component really good at one or two tasks. We’ll listen to feedback carefully to get this right.
  • Each Component has a clean, simple web front end so you can explore the data held in a store without having to write lots of SQL queries. Often you will be able to edit the data this way, too.
  • Get started in seconds – each Component offers at least some functionality which is available inside a minute after getting involved.
  • Non coders are welcome – we are building the Components so that non-coders can start gathering,  editing and sharing data straight away, possibly long before they are in a position to launch a ‘real site’.
  • Data can be added to the Components both through write APIs and through manual editing interfaces, suitable for non-coders.
  • Learn from our mistakes – it is really easy to get the wrong data structure for civic, democratic or governmental data. Good practice data structures are baked into the Components, to save you pain later.
  • Use our hosted versions, or install open source code locally. It will normally be quicker to get started in using the Components in a hosted environment, but if you want to run them locally, you’re entirely welcome. The code will be open source, and we’ll work hard to make sure it’s attractive and easy to install.
  • The Components will talk to each other, and to the rest of the web using simple open schemas which will evolve as they are built. Where possible we’ll pick up popular data standards and re-use those, rather than building anything ourselves.

What the Components Aren’t

Sometimes in life it can be easier to describe things by what they aren’t:

  • The Components are definitively not modules in a framework or platform. Each one is totally independent, and they will frequently be written in different languages – partly to force us to ensure that the APIs are truly excellent.
  • The Components aren’t either Hosted or Local, they’re both. We’ll always offer a hosted version and a downloadable version, and you’ll always be able to move any data you have stored on the hosted versions down to your local copies.
  • The Components aren’t all about mySociety. We’re planning to build the first ones in conjunction with some friends, and we’ll be announcing more about this soon. We want the family of Components to be jointly owned by a group of loving parents.

 When can I see some of the Components in Action?

We’ll be blogging more about that tomorrow…

 

Footnote – To see the provenance of the extremely useful  ’small pieces loosely joined’ concept, see this.

2 Comments

  1. Great to see that your thinking on this has evolved following that discussion at the MySociety meetup in Euston 6 months ago.

    We’ll be very interested to see how you get on, not least as we’re going through a similar process of trying to make what we do much more modular.

  2. [...] They want to do something about that. The extreme end of their spectrum, a single local government domain, was felt to be a bit of an impossible dream but the room was taken with the idea of local government components that could be shared, deployed and maintained across the country for the standard things (this is an excellent, and timely, blogpost from MySociety about exactly this approach on an internat…). [...]