Project description

This project has its roots in a debate that was aired on BBC Radio 4’s Today programme early in 2018 discussing the state of the UK’s broadband infrastructure. The BBC presenter, replying to fears expressed by an interviewee that the country’s rural areas were being left behind, sought to balance the argument by suggesting it was unreasonable to expect rural areas to ever be as fast as central London.

This epitomised a widely-held public view that access to broadband – and the speed that might be delivered in an area – is simply a function of urban/rural status. We decided to question this received wisdom by delving deep into new open data released by industry regulator Ofcom, linking it with other data sources. We decided to see what over 1.3 million postcodes worth of data — a previously unseen level of geographic detail — could say about the state of the UK’s broadband infrastructure.

We approached the project with an open mind. We were quite prepared to accept that the widely held intuition outlined above was true — and if that were the case, we likely would not have spent time on producing a lavish interactive project. But our investigation revealed huge surprises about the state of UK broadband, resulting in not just a ‘numbers story’, but amazing human ones too.

Timing was important. Implicit to much of the ongoing Brexit debate is how well prepared the UK is to compete internationally after its anticipated departure from the EU. In this context, broadband lines up alongside transport and housing as key elements of the national infrastructure.

The Financial Times operates a subscription model for access to its content — we wanted to produce premium content and analysis that justified our subscribers’ faith in their FT subscription, rather than aiming for simple page view metrics. The feedback/engagement from our informed audience on the quality and depth of our journalism was what mattered above all else.

What makes this project innovative?

Behind the delivery of the project were two important innovative elements: a GIS (geographic information system) analysis of Ofcom data — the methodology of which we included visually in the article itself; and a ground-breaking use of Mapbox to deliver the results of this analysis in a visual and personal way for the reader, even on a mobile phone. Our project fused data journalism with more traditional reporting techniques in a mature way that combined the best of both worlds. We used open source QGIS and R software to merge the Ofcom data with UK-wide urban/rural definitions from the Consumer Data Research Centre and new open source data from Ordnance Survey to produce broadband speed estimates at building block level. Our initial analysis found that 41 of the top 44 fastest postcodes in GB are in rural areas, mainly in Cumbria and Lancashire. This was surprising. Further investigation revealed that these postcodes’ internet services were provided by a small rural startup founded by a farmer’s wife and rural volunteer community. Our telecoms correspondent Nic Fildes travelled north to interview them and found amazing human stories: the teacher who had to give up teaching after a cancer diagnosis and now ran an internet business on Etsy from her rural home over its new gigabit internet connection; the owner of a local pub who delighted in seeing “gobsmacked” tourists logging on to their country pub’s wifi. Conversely, we also found the UK’s city centres — including London, Birmingham, Manchester and Glasgow — to be running slow, many because they are either costly to transform or mired in planning restrictions. Just as with the “fast” parts of the country, an important part of the project was to relate this to the human experience. For example, we found a professional photographer in Rotherhithe who had to move home because trying to upload photos from there was like “trying to milk a donkey”.

What was the impact of your project? How did you measure it?

Overall page views were pleasing at around 50,000 — but the most important metric that reflected the success of our project was the average page view time of 3 minutes 29 seconds, well over 3 times the duration of a regular FT article. The project clearly engaged readers and uptake of the social media functionality we built into the app reflected this too. We encouraged readers to share speeds for their own broadband area to promote the debate: https://twitter.com/search?f=tweets&vertical=default&q=CheckhowyourscomparesontheFTsmap&src=typd Comments from famously hard-to-impress FT readers were also revealing: “Thank you, my life makes more sense now 😉 I live on the Bermondsey/Rotherhithe border and have struggled to get a decent provider. I’m currently using Hyperoptic but some nights, I can’t even make a google search, let alone get Netflix started. Apparently my broadband is slower than 95% of London postcodes! I don’t know how the other 5% do it!” “Quite an eye opener! I would have definitely assumed that denser urban populations would get higher quality internet but suppose the difficulty in digging up central streets prevents that. My speed is apparently lower than in 70% of London postcodes :(“

Source and methodology

* Open source data on broadband speed metrics released by Ofcom (~1.3 million postcodes, provided by service providers) were the starting point — the first independent, large scale broadband speed dataset available from the industry regulator. * This was matched using RStudio to urban/rural postcode data produced by Consumer Data Research Centre to explore how speeds varied across urban and rural areas. An advantage of the CDRC data set is it allows comparisons across the UK (regular urban/rural classifications are only at country level for England/Scotland). * Exploratory visualisations were created in RStudio to identify key patterns in the data, i.e. fast rural outliers, slow city centres (bimodal distributions in urban areas). * At this stage, our analysis informed our telecoms correspondent Nic Fildes on where to look for human stories on the impact of fast/slow broadband speeds. * Broadband speed estimates were then generated at street level by using the Inverse Distance Weighting technique in QGIS to map the 1.3m postcodes’ data to the building blocks in Ordnance Survey’s VectorMap District detailed mapping product (2.7m polygons). * This was then used to produce the interactive map and graphics, as described in more detail below.

Technologies Used

We overcame numerous technical challenges on this project, many of which are addressed in an detailed article we wrote for the data journalism community website Source (https://source.opennews.org/articles/how-we-made-our-broadband-map-using-mapbox/). The map itself was the product of a long and complex data transformation process, as detailed in the methodology note appended to the article (https://ig.ft.com/gb-broadband-speed-map/#methodology). The Ofcom fixed broadband data set contains broadband speed data for 1.3m British postcodes; in lieu of the prohibitively expensive postcode polygons data set, we plotted each postcode’s centroid using QGIS (https://qgis.org/en/site/), then used inverse distance weighting to generate a 250 sq m grid wherein each square was assigned an average broadband speed based on the postcode centroids within it. These grid values were used to shade the building polygons from the Ordnance Survey VectorMap District product (https://www.ordnancesurvey.co.uk/business-and-government/products/vectormap-district.html). Finally, we converted the resulting shapefile into vector tiles optimised for the web using the command-line tools GDAL (https://www.gdal.org/) and Tippecanoe (https://github.com/mapbox/tippecanoe). The core of the app is an interactive map of Great Britain powered by Mapbox GL (https://docs.mapbox.com/mapbox-gl-js/api/), a JavaScript library for rendering maps using WebGL. Given the large amounts of data involved, it was necessary to ensure that the app was highly optimised for performance. To achieve this, its component-based architecture was built using React (https://reactjs.org/), an open-source framework for building web apps. To improve the developer experience of integrating Mapbox GL with React, we used a tool called react-map-gl (https://uber.github.io/react-map-gl/) created by the Uber visualisation team. The full codebase for the app is available on GitHub: https://github.com/ft-interactive/uk-broadband-map.

Project members

Alan Smith, Nic Fildes, David Blood, Ændrew Rininsland, Caroline Nevitt, Max Harlow

Video

Link

Additional links

Followers

Click Follow to keep up with the evolution of this project:
you will receive a notification anytime the project leader updates the project page.