Project description

The Data Journalism Team is a computational investigations unit working cross-title for The Times and the Sunday Times newspapers. We specialise in using data and technology combined with more traditional approaches to find and report rich, impactful stories.

Gagging clauses (6 November 2018)
The Times revealed the existence of gagging clauses in government contracts which banned charities and other firms from criticising a Minister of State over payments to welfare claimants. Further analysis by the data team of 38,000 contracts across government departments showed over 40 charities and hundreds of companies with contracts valued at over £25bn, were subject to similar gagging orders – including advisors to the inquiry on the Grenfell Fire tragedy in which 72 people lost their lives.

Tax havens and British billionaires (7 March 2019)
A three-day series of data-led stories showing a third of British billionaires have moved to tax havens amid an exodus over the past decade. The findings show wealthy Brits controlling over 12,000 UK firms from low-tax jurisdictions, many of whom have made sizable donations to British political parties. The team also created an alternative tax haven ‘rich list’.

Peers on the Russian payroll (24 October 2018)
In the wake of the Skripal poisonings, concerns were raised over Russian influence and lobbying in the British parliament. Using ministers’ declarations and recently released ultimate ownership registers, the data team identified a number of lords on the payroll of Russian state-backed firms, who had sought ministerial meetings or asked questions in the house that would be beneficial to their clients.

Pupil off-rolling (28 August 2018)
Schools will often go to great lengths to boost league table rankings but a collaboration between the investigations and data teams found thousands of students being removed from schools prior to exams every year. So-called ‘off-rolling’ saw many lower performing pupils sent to pupil referral units, often a target for gang recruitment, or simply disappearing from records entirely.

Profiting from poor care (5 November 2018)
The data team worked with the social affairs editor to find private companies and charities making millions of pounds from the NHS for psychiatric care – often providing substandard or unsafe care. It follows the news that a young woman with autism was held in seclusion for months at a time in a failing health facility.

Veteran suicides (11 November 2018)
Support for former soldiers seemed inadequate as reports of veterans taking their own lives seemed to be on the up – one month saw five suicides in as many days. No official government body was monitoring the situation so we stepped in to create the missing database, prompting the Ministry of Defence to agree to formally monitor and address the issue.

Transport series (30 April 2018)
Not a day goes by without stories of disgruntled road and rail users. This three-day data-led series showed motorists being unfairly caught out by ‘smart cameras’ and variable speed limits on motorways and train companies making significant profits – £100m a year in fact – from delays that cause daily misery for commuters.

What makes this project innovative?

Gagging clauses At the core of this investigation was the ability to run text analysis over thousands of government contracts - which was achieved using a bespoke database built by the data team to find and quantify the extent of the problem. Tax havens and British billionaires We started with the recently released ultimate ownership register for companies in the UK to identify every British business owner who gave an address in a known tax haven. This, programmatically matched up to other sources such as rich lists, property ownership and political donations registers allowed to narrow down the list to the most notable individuals to report on. Peers on the Russian payroll What would otherwise have required a lengthy process of manual trawling through the declarations of 850 members of the House of Lords, the data team was able to complete in just a few hours using scraping and combining datasets to swiftly identify peers of interest which formed the basis of the story. Pupil off-rolling The team analysed more than 30,000 Department for Education records for the past three years to identify schools with wide gaps between the number of pupils registered as starting GCSEs and those whose results then counted towards league tables. Profiting from poor care The data team was able to analyse huge datasets from the regulator to identify not only those that were failing but those who used solitary confinement and took a deep dive into the accounts to find fat cat perks and profits siphoned to offshore trusts. Soldier suicides After spotting a worrying trend with veteran suicides seemingly on the up, but with no official records to check this, our team used data-driven OSINT techniques to find and record the details of every suicide of a former soldier that year. Transport series Data obtained via Freedom of Information requests revealed that tens of thousands of motorists were being caught out by a new ‘smart motorways’ system of varying speed limits without offering clear guidance to drivers. Over on the rail network, public funds are used to compensate private rail operators for such things as a tree falling on the line causing delays. For the first time, we compared the compensation payments with payouts to commuters - and found a sizeable £100m profit being made by private firms from delays.

What was the impact of your project? How did you measure it?

The Data Journalism Team strives to produce stories that lead to meaningful change, such as changes in law, remedying injustices or informing current public debate. We use data and technology to reach the stories our rivals can’t. The gagging clauses investigation provoked uproar in the third sector, with charity bosses accusing the government of stifling legitimate criticism. Theresa May promised a review of the policy in an embarrassing climb down. Our investigation into Lords on the Russian payroll sparked an inquiry by the Parliamentary Commissioner for Standards into the conduct of the peers we had reported on. Our bespoke database into veteran suicides prompted the Ministry of Defence to pledge to formally monitor and record veterans who take their own lives - with suicides and open verdicts recorded by coroners to be passed on to the MoD. In response to the transport series, Highways England who run and maintain motorways in the UK, said a “comprehensive” review of variable limits would be launched after admitting that lower limits were not always correctly set. Every story was the splash on their respective papers - promoting public debate and often pick up from other news outlets.

Source and methodology

Gagging clauses Tip-offs from the charity sector alerted our reporters to the existence of gagging clauses in government contracts. Thousands of contracts were scraped from government websites. Our data team created a bespoke database and analysed over 38,000 of these documents. Tax havens and British billionaires We used the ultimate ownership register to look for Brits that had registered their address in known tax havens as a starting point to identify the billionaires of interest. We used cross matching tools to programmatically compare against rich lists and other sources to find and report the extent of their wealth and influence on British public life. Peers on the Russian payroll To find the number of Lords on the Russian payroll, we scraped the list of members interests and combined it with the recently released ultimate ownership register of companies in the UK. We were swiftly able to locate those with Russian ultimate owners, which formed the basis for the resulting stories. Pupil off-rolling Department of Education records were used to identify schools with a consistent pattern of ‘off-rolling’. Profiting from poor care The data team began with a comprehensive database of all CQC inspections in recent years, and filtered those who provided psychiatric care including the use of solitary confinement that had been rated inadequate or unsafe in recent years. We then calculated income from public funds (NHS) and profits made despite continued substandard care. Soldier suicides After spotting a worrying trend with veteran suicides seemingly on the up, but with no official records to check this, our team used data-driven OSINT techniques to find and record the details of every suicide of a former soldier that year. Transport series The Times analysed net incomes to private rail operators from the schedule 8 compensation scheme (payouts for unplanned disruptions) over the past five years and compared it to sums paid out to passengers for disruption under the government’s “delay repay” scheme. We found very little compensation was passed on to commuters effectively giving the companies sizeable profits from delays.

Technologies Used

The Data Team uses a variety of technologies and carefully selects the best tools and techniques on a project by project basis. Commonly used tools include R, Pandas, database software including Neo4j for graphs and PostgreSQL as well as numerous command line tools - such as CSV Kit and XSV. The team also teaches spreadsheets and Open Refine for cleaning and reconciling data across the newsroom to empower our reporters and improve data literacy.

Project members

The data team consists of Leila Haddou (Data Journalism Editor), Louis Goddard (Data Journalist), Krystina Shveda (Data Journalist) and George Greenwood (Data Journalist). Gagging clauses Paul Morgan-Bentley (Head of Investigations, The Times) Louis Goddard (Data Team) Tax havens and British billionaires Paul Morgan-Bentley (Head of Investigations, The Times) Billy Kenber (Investigations Reporter, The Times) Louis Goddard (Data team) Peers on the Russian payroll Lucy Fisher (Defence Correspondent, The Times) George Greenwood (Data team) Pupil off-rolling Paul Morgan-Bentley (Head of Investigations, The Times) Louis Goddard (Data team) Felix Thompson (Intern) Profiting from poor care Greg Hurst (Social Affairs Editor, The Sunday Times) George Greenwood (Data team) Soldier suicides Mark Hookham (Defence/Transport Correspondent) Richard Kerbaj (Defence Correspondent, The Sunday Times) Caroline Wheeler (Deputy Political Editor, The Sunday Times) Krystina Shveda (Data team) Transport series Graeme Paton (Transport Correspondent, The Times) Leila Haddou (Data Journalism Editor)


Additional links


Click Follow to keep up with the evolution of this project:
you will receive a notification anytime the project leader updates the project page.