[The below is a summary text, might not match delivery]

Hi, I’m Matthew Somerville, a developer at mySociety, a charitable project that helps people be active citizens, and the organiser of this conference. In this talk I will be looking at some of mySociety’s long-running projects, and the variety of unique issues that can come up that wouldn’t necessarily be in a brand new project.

mySociety has been around in its current form for over 15 years, I’ve been with it nearly all that time. I’ll just quickly run through, as instructed by the King of Hearts, a brief history of some of the projects which I’ll be covering through this talk.

Our first site to launch was WriteToThem in early 2005, a project that lets people contact their elected representatives, from local councillors to national MPs. It built upon previous work of FaxYourMP and Stand as far back as 1998. It is still running now. Also that year, we launched both PledgeBank, a collective action website, and HearFromYourMP - a constituency-based mailing list for MPs. Those two both closed in 2015.

TheyWorkForYou, a site that tells you how your MP has voted and what they’ve said in Parliament, which had actually begun in 2004 as a non-mySociety project, was made one in 2006. That year, we also started running the national petitions website for Downing Street (the British PM) (until the 2010 election).

FixMyStreet, a nationwide website for reporting street problems, launched in 2007, and WhatDoTheyKnow, a freedom of information requesting service, in 2008.

Lastly, MapIt, our internal service for mapping postcodes and points to polygons, was opened up to the public in 2010 thanks to Ordnance Survey releasing the relevant datasets as open data.

With all of these, and all our other projects, we of course try and automate as much as possible. But you can’t automate everything. Running a website, no matter how small and self-sufficient it is, requires some investment, in terms of maintenance, user support, and updating.

First, there are the issues that can arise with any long running project. It makes you feel like the Red Queen.

Updates: The framework you have written your project in, if any, may release updates; if you’re running your own servers, they will need security updates. Any APIs you use may change, break, be removed. Not sure how true it is now but years ago Facebook would change their API without notice or caring whether you could keep up to date.

Redesigns: People’s expectations change, you may need to redesign your site to keep people interested. FixMyStreet and WriteToThem have both really only had one large scale visual redesign; TheyWorkForYou and WhatDoTheyKnow have had a couple. Designing progressively and for resilience, both front and back end, can certainly help reduce the burden. For example, when we eventually had the time to improve FixMyStreet’s photo upload [we have a lot of dog poo photos], the fact we’d always written it to deal well with errors, remembering the uploaded photo to save time, made it easier to work with. FixMyStreet maps have always worked if JavaScript is unavailable, something Google Maps seems unable to manage.

Technical debt: With any long running code base, you can build up technical debt. Choices that were fine at the time no longer making sense, or things have to be done in the time available, that kind of thing. For example, with the Number 10 Petitions site we needed something that would deal with large scale operating of petition signing and confirmation, so ended up writing our own small multiplexed email sending daemon. We no longer run the national petitions site, but do still run some local council sites based on the same code, and it’s therefore more of a chore if it goes wrong/behaves differently to the norm. But it works fine, so no real need/desire/time to rewrite.

Okay, so that’s general maintenance. Now I want to talk about problems caused by relying upon external data and sources, especially in a civic technology context, what we’ll call managing flamingos.

TheyWorkForYou began as a scraper that fetched HTML from the official Parliamentary website, parsed it into XML and into a database for display on the website. The HTML was produced via a manual process in parts, leading to frequent issues with its fragility when e.g. not all a speaker’s name was in bold, or the structure was just slightly unparseable by the current code. We built in a ‘patcher’ that would let us apply patches to the source in order to get it to parse which in many cases was easier than complicating the parser yet further.

One advantage of putting all this data into a database and automating parts of it meant it could perform checks. So one morning, we received an email from the parser saying that when trying to import, it had discovered a name in the voting list that wasn’t a current MP. We assumed the parser had a bug, as usual, but upon investigation discovered that actually an MP who had recently died had been listed as having voted in the Official Report. We got in touch with Parliament to point out their mistake, and they were - very grateful :)

Nothing stays the same, however, and part of our aim is of course to encourage the official sites to improve. So we can hardly complain if they do so, even if it creates a massive amount of work for us! In this case, Parliament launched a new website for Hansard, which completely broke our scraper. However, we were also given access to the source XML that was being used to generate the HTML. We still had to write an entirely new parser, but one that is hopefully a bit more stable than the old HTML, at least until that changes too. And even within the XML, it is still a manual process (and a complicated and complex process, to be sure!) so issues can still arise, where e.g. we don’t know if member tags are wrapped in bold tags, or vice-versa. And they are changing too - for example, the House of Commons recently added proxy voting support at very little notice (after discussing it many times, it took a representative delaying giving birth in order to vote to speed it along). Hansard added a bit of text to the voting line to signify this (not much else they could do at such short notice!), and we have to adjust the parser to spot it.

Also, the person identifiers they use in their XML turn out to not be per-person, but per-role (so e.g. two Bishops of London share the same ID, as two Marquesses of Salisbury – don’t ask why the UK has unelected religious and hereditaries in our legislature…). So recently in a reverse situation to years ago, TheyWorkForYou briefly had a deceased Lord voting because we’d misunderstood and used the identifier of the previous incumbent of the title.

TheyWorkForYou also covers Scotland, and there have been large chunks of time – 2011–2013, and 2014–2015 – where we haven’t been able to update the site with new data, due to changes in their website and no resource to update the parser to cope. Thankfully, the last time this happened, they were willing to keep outputting their data in the old format to enable us to get up and running again.

So things can break all on their own, and can break due to changes made to things that you are relying on.

Then there are things not breaking, but just getting out of date unless you keep up the maintenance.

In our work on civic technology, that includes a number of obvious things. Representative details as mentioned previously - these will change at elections, between elections, plus contact details or and other information can change at any time. When WriteToThem launched (and before), we had a computer connected to a fax machine (on an unlimited fax call plan) so that it could send faxes to those MPs who didn’t yet have email. We no longer have this, as only a handful of MPs now won’t accept email, and it’s not worth it. WhatDoTheyKnow needs to maintain a list of all public bodies contactable under the UK’s FOI Act. TheyWorkForYou voting policies (describing votes, and grouping votes by category) require manual work.

Talking of elections, at the UK 2005 general election, I happened to be in Canada for work – for 3 days – which made updating TheyWorkForYou as results came in quite easy given the timezone difference! 2010 was harder going, but by 2015/2017 we also had other sites and projects such as YourNextMP and DemocracyClub, providing human-created, machine-readable data to automatically import results.

Another civic tech staple is boundary data. Whilst Brexit may take most of the coverage, a few days later a more local change is happening. Some of the councils (municipalities) in England are merging together (the old two-tier structure now feels outdated in some places, but it’s also to try and save money, I imagine), and this last happened in 2009. We need up-to-date boundary data for e.g. WriteToThem and FixMyStreet, to know which postcode or point maps to which area, but official boundaries do not get released until or just before the elections in May. So in order to be accurate, we have to manually construct boundaries from other data. Which we’ve done. This is an advanced - data-wise - country, so you can imagine how hard it is in some places to establish boundaries.

I’ve done a lot of technical talking, but I must also mention the elephant, or the dodo, in the room – funding.

Quite commonly, people and organisations seem to want to fund new things, not help keep existing things going and improve, and such funding naturally always come with conditions. mySociety has actually been very lucky to get some unrestricted funding over the years. Mzalendo, a Kenyan parliamentary monitoring site we helped relaunch in 2012, was developed jointly with us, but now funders would prefer to fund local development, which is great but a problem for us!

So we’ve been funding our charitable work through commercial services - e.g. “Pro”fication, FixMyStreet selling to councils, helping add features that help them and users. For example, we worked with Channel 4 to build a website to go with their series of programmes on the Great British Property Scandal, which was based upon the FixMyStreet code – this enabled us to add support for Northern Ireland to the main fixmystreet.com site. Good things can arise!

Another issue that can arise is our sites have been around so long, people assuming they are just "part of the furniture”.

Like the Cheshire Cat, we can fade into the background, leaving only a grin.

No thought into what's behind it, how they are funded, or the effort involved. Recently the BBC drama Bodyguard created a mock-up of TheyWorkForYou to lend background to their story, rather than the official Parliament site!

WhatDoTheyKnow has also met with some challenges, with the government (never the biggest FOI fan) initially trying to refuse public responses, or trying to introduce limitiations, thankfully all rebuffed, But viligence is needed, and that requires effort.

Then of course, one issue that can arise with any project is that you can’t maintain everything to a level that keeps it useful and functional for users, and then you have to make the hard decision to shut something down. Or something happens that makes your offering no longer necessary. Some examples we have had in our history include:

PledgeBank. A collective action project, predating Kickstarter, Unbound, Groupon, and similar pledging concepts. But it did not have the resources, scale, or commerciality to be taken further, with the initial core idea that it should run itself. Perhaps in a looking-glass world…

The second example was HearFromYourMP. At the time, the internet was still novel to many MPs, nowadays, most MPs have their own channels for contacting constituents online.

Finally, our petitions service for Number 10 was actually taken in-house after the 2010 election and are now managed directly by Parliament, obtaining a debate if you get enough signatures, which is great to see.

Lastly, there are the issues that come with your service becoming more popular, things you might want to try and foresee (or remember forwards?).

Firstly, having more users is not necessarily a good thing - especially if they’re not paying for anything! Needing more support, technical issues, more storage, more enticing to spammers even.

For FixMyStreet, growing more popular brought up UX issues, to do with e.g. the display of so many reports on a map, or to do with making report creation easier.

We spent some time making our FixMyStreet and Alaveteli (WDTK) codebases customisable and installable, and are now installed in dozens of countries around the world. But that also takes support too. Certainly with FixMyStreet, also as part of our work with UK councils, we’ve tried to make it so different users can have their own things without affecting other users; not something we did at the start, not worth doing, but something to consider if you look to expand.

Thanks for listening!