Project description

The St. Louis Crime Tracker is a news app which allows residents of St. Louis city and county to browse maps of crime incidents, examine crime trends by neighborhood or police department, and make year-over-year comparisons of crime totals.

We created the first version of the Crime Tracker in 2016, but it was limited to only the city of St. Louis. The city is served by a single police department, and it puts its monthly incident reports online in one place. Each month we download those files, parse them, and analyze the numbers to create the graphics and maps for the Crime Tracker.

Readers responded. They particularly liked the crime maps. But most of our readers live in St. Louis County. They wanted to see similar reports for their own municipalities and police departments.

The problem is that St. Louis County suffers from extreme fragmentation. There are 88 municipalities in the county right now, policed by more than 50 different police departments. Some of these departments are very large and professional. Others are small and have difficulty staying in compliance with Missouri\’s statutes on reporting of crime statistics.

Revamping the Crime Tracker to include St. Louis County took more than a year of development, during which we also sought reliable sources from which we could get the data to make incident maps and charts of statistics.

It required us to account for fluid policing arrangements in the county. A municipality may decide to stop running its own police department and sign a patrol contract with a neighbor. A city council might decide to switch their patrol contract to a different department to save money. Consolidation and change occurred frequently even as we were doing our development work.

We believe our Crime Tracker offers the most complete picture of crime available in St. Louis right now. Readers responded immediately to it. However, it must be said that fragmentation has resulted in limitations in the app.

Because we obtain incident-level data from the St. Louis County Police, we can only show incident maps for those departments which have contracts with St. Louis County Police (roughly half of all the departments). Similarly, many departments in the county are slow to submit their monthly UCR crime reports to the Missouri Highway Patrol. These delays lead to some lag time in how quickly the county section of the app updates versus the St. Louis city section.

What makes this project innovative?

As far as we can tell, the St. Louis Crime Tracker is the first data portal to show monthly crime totals, year-over-year comparisons, and monthly incident maps for the majority of the region. The app is powered by numerous Python scripts. Some scripts watch websites for monthly crime reports and download the data when it's available. Others combine and analyze the reports, splitting the numbers by department and accounting for the various patrol arrangements throughout St. Louis County. They output JSON files which are pushed to our server. We have also adapted the Python analysis scripts for use in our crime reporting, particularly to examine St. Louis city's statistics for chief John Hayden's "rectangle" policing strategy. We can draw a polygon in QGIS, export it as a GEOJSON file, and then generate statistical reports for that particular area. The news app itself is written in HTML and Javascript using a number of libraries: * On the front page of the app, we use Chart Tool to produce the combined trend line chart. * On the St. Louis County and St. Louis City overview pages, we use D3.js to create the year-over-year change maps. * On the neighborhood/jurisdiction report pages, we use Leaflet to produce the monthly incident maps and Highcharts to produce the various bar charts. The news app includes a geolocation feature so that users within St. Louis city or county can immediately go to the report page for the neighborhood or jurisdiction in which they are located.

What was the impact of your project? How did you measure it?

As we faced the fragmentation problem, we realize that reader engagement would be important. We designed the app to display a prominent notice on any jurisdiction's report page if we lacked incident data for that jurisdiction (and therefore could not display a monthly incident map). We included a call to action in the notice, asking readers to let us know if they wanted to see an incident map for that jurisdiction. Since we launched the app, we have received more than 300 such requests to add an incident map. We have heard anecdotally that readers were contacting police departments and city councilmembers directly to ask why their city didn't have an incident map. In the months since we relaunched the Crime Tracker, St. Louis County Police signed agreements or new contracts with around 20 municipalities to allow them to publish their monthly incident data. While the Post-Dispatch has know way of knowing how much credit we can claim for that, we do believe that the publication of the Crime Tracker motivated departments to sign up. The Crime Tracker's first weekend was particularly strong, with more than 10,000 users and more than 177,000 page views. Since then traffic has leveled off, but it bounces whenever we notify readers that the monthly numbers have been updated. In terms of other impacts, the Crime Tracker led me to write a couple related stories. The first looked at how some area police departments failed to comply with state law on submitting monthly crime statistics. The second was an explainer for readers about how it was possible to have a negative number of crimes in special circumstances. The Crime Tracker also prompted a discussion of crime statistics, policing, and transparency on our podcast, "Inside the Post-Dispatch".

Source and methodology

Our crime data is sourced comes from the following agencies: * Missouri Highway Patrol's Criminal Justice Information Services Division * St. Louis Metropolitan Police * St. Louis County Police's Bureau of Research and Analysis We use population data and geographic boundaries obtained from the U.S. Census Bureau, as well as geographic boundaries obtained from the GIS departments of St. Louis city and St. Louis County. Because of the fragmentation in St. Louis County, it was also necessary to report and compile a list of all police departments, their policing contracts, and the start and end dates for contract changes. We engaged with several people at St. Louis County Police as we developed our Crime Tracker. They were simultaneously developing their own crime map, and were very responsive to our feedback. We wrote scripts to analyze and check the quality of the data we are receiving. As we developed the Crime Tracker and wrote our parsers, we found many problems with the county's data. Sometimes the issues were small, such as inconsistencies in column labeling. At other times the issues were significant, such as disparities between UCR index totals we calculated from the county's incident data, compared with the index totals that had been submitted the Missouri Highway Patrol. The Crime Tracker has also allowed us to identify a couple times when police departments have revised previously-submitted UCR crime totals. When we notice these updates, we reach out to the department to ask why the revisions were necessary. Though we have not uncovered any malfeasance, we feel like this is a useful watchdog practice. Finally, as mentioned above, we have adapted our Crime Tracker scripts to allow us to generate crime statistics and reports for any geographical boundary or polygon. This has allowed for useful analysis in reporting recent stories about St. Louis police chief John Hayden's "rectangle" policing strategy.

Technologies Used

Applications: * QGIS * Microsoft Excel Python libraries: * NPR app template * Agate * Beautiful Soup * Shapely * Omgeo Javascript libraries: * Chart Tool * D3 * Leaflet HTML / CSS: * jQuery * Geolocation * LESS There are thousands of lines of custom Python code in the various scrapers, parsers, and analysis scripts. We scrape incident-level spreadsheets from St. Louis city. In St. Louis County, we obtain incident-level JSON data from an API. In both situations, the data is parsed, combined and cleaned before further analysis. We also try to do minimal correcting for certain geographic oddities, such as missing coordinates, or coordinates that are set to a different neighborhood than the neighborhood to which the crime was classified. For St. Louis City, we calculate UCR index totals from the incident-level data. But for St. Louis County, where we only have incident-level data for some police departments, we instead use UCR index totals that have been submitted to the Missouri Highway Patrol. The Missouri Highway Patrol reports are in a strange Excel format which we never successfully parsed using any existing Python libraries. Instead we process the reports using Beautiful Soup because the files' data structure is essentially HTML. We use the Agate library and our own custom Python code to calculate year-over-year comparisons, per-capita crime rates and other statistics from the raw index totals.

Project members

Jean Buchanan (Projects editor) Walker Moskop (Former data reporter)



Additional links


Click Contribute to request to join the project.
You will be sending a direct message to the project leader, once they have validated your request you will be added to the Contributors list, and can start working on the project.


Click Follow to keep up with the evolution of this project:
you will receive a notification anytime the project leader updates the project page.