We recently made some changes to our Freedom of Information software, Alaveteli, that makes it twice as easy for partners to change the look and feel of their site. This post goes quite in-depth, so it’s probably best suited for designers, developers and other technically-minded readers
This year we embarked on a ground-up redesign of Alaveteli’s theming capabilities to make customising Alaveteli easier and faster. We used a mixture of techniques to try to achieve this and since releasing the update we’re happy to have seen a decrease in the complexity of our themes, and the time we spend making them.
Alaveteli, our Freedom of Information platform is one of our most deployed software packages and each deployment requires some customisation, from simple branding changes to complex functionality to support a country’s FOI process. This is either done internally by our own developers, or our partners take this on themselves.
An Alaveteli deployment consists of two things, the core: the guts of Alaveteli with no customisations, and the theme: the aesthetic and functional customisations made to each site. We ship Alaveteli with a fully-featured base theme that gives partners a good starting point to work from, but they can also start their own theme from scratch.
After talking to our partners and the Alaveteli team we felt that the process of theming was too complicated and involved. It required a lot of knowledge of Alaveteli’s inner workings, was coded in a way that made it hard to understand, and almost always required the intervention of a front-end expert, even to make trivial changes.
Our vision for how Alaveteli’s theming should work was:
- Intuitive Front-end developers and designers should be able to follow and understand the templates without needing expert knowledge about Alaveteli’s inner-workings.
- Flexible Overriding Alaveteli’s core code should be easy, so a theme should use low-specificity, componentised code to enable this.
- Supportive Should a partner decide to only change a few colours, or embark on a large-scale redesign Alaveteli, the theming process should never get in the way.
How we did it
Using the tools we have
Alaveteli is built on Ruby on Rails, so our first goal was to make better use of Ruby on Rail’s template inheritance.
Firstly, we split all of the Sass files into logical components.
Each component has two associated Sass files, a layout.scss and a style.scss. layout.scss describes only layout of a component, e.g. the physical dimensions and position on a page. style.scss describes the styles, for example typefaces, colours, borders and other presentational properties.
Should a partner wish to totally redesign or replace a component, they can replace both files with their own. Alternatively they can just choose to replace one part of the component, for example keeping the layout, but replacing the styles.
Ensuring the lowest possible specificity
When customising Alaveteli we want to be sure that a partner never has to fight with the base theme to achieve their goals. The most important part of this is to ensure CSS styles are easy to override.
Firstly we removed all ID selector’s from Alaveteli’s stylesheets and replaced them with class selectors, giving us a good jumping off point as class selectors have much lower specificity than IDs. We then rewrote and refactored Alaveteli’s CSS using the Block Element Modifier methodology (BEM).
BEM helped us in two ways: it’s written in a way that makes CSS self-describing, for example the code snippet
.header__navigation__listtells us that this list is a child of the navigation element and a grandchild of the header element.
As well as this, using a single class to define the styles for this element means that using BEM gives our styles the lowest possible specificity, so partners can override them with little resistance.
Using a settings partial
All of our theme’s styles use the variables defined in _settings.scss when referencing colours, typefaces, spacing, and even logo files so the bulk of Alaveteli’s customisation can be done in our base theme’s settings partial _settings.scss.
This allows partners to make simple changes which have a huge impact on the look and feel of the site. We’ve found that the large majority of our theme customisation can be done by just changing the variables in this file and it has sped up the theming process markedly.
We’ve quietly rolled out these updates to Alaveteli over the past 12 months and the impact of the changes has been encouraging.
Internally, we’ve found our theming projects are faster and more efficient. We conservatively estimate that it takes half the time to make a new theme than it used to.
Thanks to the self-documenting nature of BEM and the styles encapsulated in our settings partial, Alaveteli theming requires a lot less understanding of the platform’s inner workings. Our partners are able to be more self-sufficient and rarely require our intervention to assist with simple customisations.
Maintenance and support of our themes is greatly simplified, as the structure is consistent across projects, we’re able to roll out further changes and fixes in a more predictable way.
Overall, we’re really happy with how the changes worked out and the things we’ve learnt from this project we’ll be carrying into our others too, so look out for changes like across our work in the future.
mySociety’s EveryPolitician project aims to make data available on every politician in the world. It’s going well: we’re already sharing data on the politicians from nearly every country on the planet. That’s over 68,652 people and 2.9 million individual pieces of data, numbers which will be out of date almost as soon as you’ve read them. Naturally, the width and depth of that data varies from country to country, depending on the sources available — but that’s a topic for another blog post.
Today the EveryPolitician team would like to introduce you to its busiest member, who is blogging at EveryPolitician bot. A bot is an automated agent — a robot, no less, albeit one crafted entirely in software.
First, some background on why we need our little bot.
Because there’s so much to do
One of the obvious challenges of such a big mission is keeping on top of it all. We’re constantly adding and updating the data; it’s in no way a static dataset. Here are examples — by no means exhaustive — of circumstances that can lead to data changes:
- Legislatures change en masse, because of elections, etc.
We try to know when countries’ governments are due to change because that’s the kind of thing we’re interested in anyway (remember mySociety helps run websites for parliamentary monitoring organisations, such as Mzalendo in Kenya). But even anticipated changes are rarely straightforward, not least because there’s always a lag between a legislature changing and the data about its new members becoming available, especially from official national sources.
- Legislatures change en masse, unexpectedly
Not all sweeping changes are planned. There are coups and revolutions and other unscheduled or premature ends-of-term.
- Politicians retire
Or die, or change their names or titles, or switch party or faction.
- New parties emerge
Or the existing ones change their names, or form coalitions.
- Areas change
There are good reasons (better representation) and bad reasons (gerrymandering) why the areas in constituency-based systems often change. By way of a timely example, our UK readers probably know that the wards have changed for the forthcoming elections, and that mySociety built a handy tool that tells you what ward you’re in.
- Existing data gets refined
Played Gender Balance recently? Behind that is a dataset that keeps being updated (whenever there are new politicians) but which is itself a source of constantly-updating data for us.
- Someone in Russia updates the wikipedia page about a politician in Japan
Wikidata is the database underlying projects like Wikipedia, so by monitoring all the politicians we have that are also in there, we get a constant stream of updates. For example, within a few hours of someone adding it, we knew that the Russian transliteration of 安倍晋三’s name was Синдзо Абэ — that’s Shinzo Abe, in case you can’t read kanji or Cyrillic script. (If you’re wondering, whenever our sources conflict, we moderate in favour of local context.)
- New data sources become available
Our data comes from an ever-increasing number of sources, commonly more than one for any given legislature (the politicians’ twitter handles are often found in a different online place from their dates of birth, for example). We always welcome more contributions — if you think you’ve got new sources for the country you live in, please let us know.
- New old data becomes available
We collect historic data too — not just the politicians in the current term. For some countries we’ve already got data going back decades. Sources for data like this can sometimes be hard to find; slowly but surely new ones keeping turning up.
So, with all this sort of thing going on, it’s too much to expect a small team of humans to manage it all. Which is where our bot comes in.
We’ve automated many of our processes: scraping, collecting, checking changes, submitting them for inclusion — so the humans can concentrate on what they do best (which is understanding things, and making informed decisions). In technical terms, our bot handles most things in an event-driven way. It springs into action when triggered by a notification. Often that will be a webhook (for example, a scraper finishes getting data so it issues a webhook to let the bot know), although the bot also follows a schedule of regular tasks too. Computers are great for running repetitive tasks and making quantitative comparisons, and a lot of the work that needs to be done with our ever-changing data fits such a description.
The interconnectedness of all the different tasks the bot performs is complex. We originally thought we’d document that in one go — there’s a beautiful diagram waiting to be drawn, that’s for sure — but it soon became clear this was going to be a big job. Too big. Not only is the bot’s total activity complicated because there are a lot of interdependencies, but it’s always changing: the developers are frequently adding to the variety of tasks the bot is doing for us.
So in the end we realised we should just let the bot speak for itself, and describe task-by-task some of the things it does. Broken down like this it’s easier to follow.
We know not everybody will be interested, which is fine: the EveryPolitician data is useful for all sorts of people — journalists, researchers, parliamentary monitors, activists, parliamentarians themselves, and many more — and if you’re such a person you don’t need to know about how we’re making it happen. But if you’re technically-minded — and especially if you’re a developer who uses GitHub but hasn’t yet used the GitHub API as thoroughly as we’ve needed to, or are looking for ways to manage always-shifting data sets like ours — then we hope you’ll find what the bot says both informative and useful.
The bot is already a few days into blogging — its first post was “I am a busy bot”, but you can see all the others on its own Medium page. You can also follow it on Twitter as @everypolitician. Of course, its true home, where all the real work is done, is the everypoliticianbot account on GitHub.
Images: CC-BY-SA from the EveryPolitician bot’s very own scrapbook.
- Legislatures change en masse, because of elections, etc.
Last year, when we were helping to develop YourNextMP, the candidate-crowdsourcing platform for the General Election, we made what seemed like an obvious decision.
We decided to use PopIt as the site’s datastore — the place from which it could draw information about representatives: their names, positions, et cetera. We’d been developing PopIt as a solution for parliamentary monitoring sites, but we reckoned it would also be a good fit for YourNextMP.
That turned out to be the wrong choice.
YourNextMP was up and running in time for the election, but at the cost of many hours of intensive development as we tried to make PopIt do what was needed for the site.
Once you’ve got an established site in production, changing the database it uses isn’t something you do lightly. But on returning to the codebase to develop it for international reuse, we had to admit that, in the words of mySociety developer Mark Longair, PopIt was “actually causing more problems than it was solving”. It was time to unpick the code and take a different approach.
Mark explains just what it took to decide to change course in this way, over on his own blog.
The post contains quite a bit of technical detail, but it’s also an interesting read for anyone who’s interested in when, and why, it’s sometimes best to question the decisions you’ve made.
There are websites built on mySociety code in many countries across the world.
Whatever the site you’re planning, you’ll find it a lot easier with our support and development help.
Our quarterly call for applications closes on October 30, so make sure you have yours in soon. Want to know exactly what’s involved? Start here.