Project description

In the days ahead of the first House of Commons \”Meaningful Vote\” to ratify Theresa May’s deal for UK\’s withdrawal from the EU, there was much speculation about whether she would be defeated, and by what margin. The FT sought to produce an accurate estimate showing the range of possible outcomes of the vote based on a manually-compiled dataset containing the best available signals about the Brexit position of all 650 members of Parliament.

The difficulty we had faced in producing graphics to support analysis of the shifting coalitions within the House of Commons in late 2018 had demonstrated that the readily-available data on this subject was insufficient and that we would need to manually assemble and maintain a much richer orignal dataset to continuously classify individual MPs\’ position in the rapidly-evolving debate.

After proving its value by enabling an initial story that included a highly-accurate projection of the likely record scale of the prime minister\’s first defeat, the dataset has been developed iteratively as the subsequent Parliamentary impasse developed, allowing us to feed crucial detail into the daily reporting of the FT\’s Parliamentary reporters, powering further daily stories on the parliamentary arithmetic ahead of key votes (including a precise projection of the prime minister\’s second defeat), and enabling the FT to rapidly deploy interactive visualisations of the outcome of each new set of votes alongside details of each MPs\’ longer-term voting records on Brexit.

Over several months, we have logged MPs’ voting records in the House of Commons, along with their public statements on social media and local news interviews, the signatories to joint letters stating policy positions, and even the membership lists of chat groups used by various factions, allowing us to build up a detailed map of the various Brexit \”tribes\” defining the debate in the House of Commons.

Beyond the initial question of whether or not particular members of Parliament would support the Prime Minister\’s position on Brexit, we began analysing MPs’ voting behaviour in order to visualise the factionalisation of the House of Commons over the issue of Brexit. This was done using analysis of the co-voting networks of all possible pairs of MPs, which was published as part of our “Graphical Insight” series of print graphics, as well as on social media.

The database has developed iteratively, in reaction to the shifts in the story following its initial use in mid-January 2019, and continues to be in use at the date of the DJA awards deadline.

What makes this project innovative?

This ongoing project is an unusually sustained effort to apply structured data reporting methods to a fast-moving political story of enormous national interest, combining original reporting, manual curation of publicly-available data and automated methods of data collection to produce frequent detailed analysis and interactive graphics for a running story.

What was the impact of your project? How did you measure it?

Regardless of any external metrics, our primary aim was to enable more precise reporting on fast-moving stories and the application of data journalism methods to the challenging conditions of a fast-moving running political story. These efforts were successful, because they allowed us to: • Twice successfully accurately predict the scale of the prime minister's defeats on her Brexit deal • Better identify members the significant factions within the parties, such as the European Research Group * Iteratively develop an interactive roll-call page that could be rapidly deployed for any significant vote in the House of Commons

Source and methodology

* division ("roll-call") voting data on previous Brexit-related votes obtained from the House of Commons website and API * manually-coded statements from systematically-collected social media posts (Twitter and Facebook) and local news reports * membership lists of various formal and informal factions or groups (based on original reporting by our parliamentary reporters) * co-signatory lists of various public letters indicating support for particular policy positions * publicly-available electoral data from the 2017 general election and the 2016 EU referendum

Technologies Used

* Google Sheets was the dominant tool used in this project. This allowed many journalists to access and contribute to the core dataset. Many simple calculations, like cross-tabulations of MPs' positions could also be done directly in Sheets using standard spreadsheet functions, filters and pivot tables. * A Google Apps script was initially used to pull roll-call voting data into the Sheet from the Parliament API following each division in the House of Commons. This was later replaced with a faster method based on a scraper written as a Bash script. * A scraper, written in Python, was used to harvest relevant statements from MPs' social media accounts before manually coding them. * D3-based graphics appeared on the roll-call vote pages * A MySQL database, Gephi, Python and D3 were used to generate the co-voting anlaysis ad visualisations * R was used in some of the data analysis and graphics production

Project members

Martin Stabe, John Burn-Murdoch, Joanna Kao, Cale Tilford, David Blood, Ændrew Rininsland, Max Harlow, Aleksandra Wisniewska, Alan Smith, Sebastian Payne, Henry Mance, Jim Pickard, and James Blitz


Additional links


Click Follow to keep up with the evolution of this project:
you will receive a notification anytime the project leader updates the project page.