Data Potluck: 7 Million Rows of Data

There were a lot of people at this week’s data potluck

Data Potluck is a monthly event occurring the last Tuesday of every month at 6:00pm inside 1871.  Like the OpenGov Hack nights, these events focus on how open data and civic apps can help improve the citizen experience. However, these events have a more non-profit focus to them. Data Potluck was inspired by last year’s DataKind Data Drive which helped gather data for the Chicago area Red Cross. In order to keep the effort moving forward, Young-Jin Kim, Matt Gee and Nicholas Mader started the DataPotluck Meetup group.

DataPotluck’s other advantage? People bring food.

Rayid Ghani, Chief Scientist for Obama for America

At this month’s Data Potluck we had two presentations. The first was from  Rayid Ghani, former Chief Scientist for the Obama for America 2012 Campaign. Rayid explained how the Obama for America campaign used the power of predictive analysis and social media to help win the election. 

Rayid announced that the same model that made the Obama team so effective at their outreach efforts would be made available to non-profits.

Historical Traffic Congestion Data

The second presentation was by the City of Chicago’s Chief Analytics officer to announce the release of a seven million row dataset. Chicago has just released data on traffic congestion by segment.


To get an idea of just how big this data set is, a traffic segment is about a half mile. The city has 300 miles of road that the city keeps real-time traffic data for. The city refreshes the database that lives on the portal every ten minutes.

The city first released the real-time data in December, but civic developers wanted to take a look at historical data.

So, the city worked with Socrata to enable the city’s data portal to be able to handle such a massive volume of data. Now, civic developers can dig into all of the Chicago’s traffic data.

To help developers dig into the data, they’ve created a very well documented API.

This documentation includes code samples in multiple languages on how to access the data as well as definitions on all the different fields in the data set and the possible errors you could get.

Now that this data has been released, we’re excited to see what cool, useful, and interesting things that people will do with this data.

If you want to work with civic data:

For people who are interested in working with civic data, there are two opportunities that they should look into.

The first is the Chicago Data Science Fellowship. The University of Chicago and Argonne National Laboratory are recruiting people with statistics, programming, and data skills to work with real world data to make an impact on social issues.

The second is that the City of Chicago is hiring a data scientist to help ensure that Chicago becomes the very best civic data team in the world. The City of Chicago is looking to hire a new data scientist to join their team. If you are interested, you should apply on the city’s website.