Month: September 2013

Smart Chicago Collaborative and the City’s Technology Plan

Smart Chicago

Earlier this month, Chicago Chief Technology Officer John Tolva unveiled the city’s very first technology plan. The plan was a result of a year-long process of research, brainstorming, and thinking about how to make all of Chicago competitive in the new digital economy.

This plan is a comprehensive framework for growing Chicago’s technology sector , getting broadband connectivity for everyone, and  ensuring that Chicago remains a leader in open government data .

The plan also highlights the work that the civic technology community has been doing in Chicago. From the weekly OpenGov Hack Nights, the Smart Communities Program, and youth STEM programs; Chicago already enjoys a strong set of technology strengths and this plan will enable the city to advance even further.

Smart Chicago Collaborative is proud to have a key role in many of these initiatives and is dedicated to  implementing this plan. Here’s a look at our role in the plan and the aspects of our existing work in this context.

Continue reading →

Building Chicago Works For You

Chris Gansen

Chicago Works For You (CWFY) is comprised of three components: a AngularJS/Jekyll front-end, a custom Go API and worker layer, and a PostgreSQL/PostGIS database. This post will dive into the mechanics of the system and explain why it works the way it does.

CWFY started with only a handful of requirements. It had to pull service requests from Chicago’s Open311 API, save them to a database, do some math, and expose the results to a pretty user interface.

The first sketch of the system was pretty rudimentary, but it matches very closely the end result. We implemented the worker and API in Go, a powerful and fun language that is well-suited for this type of work. We are hosting the Go applications and database on a m1.medium EC2 instance on the Smart Chicago Apps infrastructure.

The worker application finds the timestamp of the most recently updated service request in the CWFY database and asks the Open311 API endpoint for all service requests created or updated since that point. This repeats every thirty seconds. That interval is short enough that we’re not clobbering the Open311 API, the number of requests in the response is small (usually less than a dozen), and we still have near-real-time data on our site. We’ve found that the Open311 API is very reliable — and fast — and can stand sustained load, as it did when we backfilled our database.

The database schema is very simple. There is a single service_requests table with more than three million rows. That’s all service data stretching back to January 2008. This table is very similar to the service_requests schema used by Code for America’s 311.fm application. In fact, we used that schema to bootstrap this application.

There are two tables of ward boundary data, one for the current ward boundaries, and one for the new ward boundaries, to be instated in 2015. But the city’s service request records only include the ward the request iscurrently in. Using the popular PostGIS PostgreSQL extension, we’re able to use the latitude and longitude associated with each service request to geo-locate it within its 2015 ward-to-be and determine if it’s in one the 233 areas of the city that are changing from one ward to another. These calculations allow CWFY to compute service delivery information for current and future wards, as well as individual transition areas.

Finally, there’s a daily_counts table, populated with counts of service requests opened for each day, ward, and service type combination. This table is populated automatically by a Postgres trigger, and allows CWFY to perform quick aggregate calculations.

The API layer is a collection of HTTP endpoints that generate JSON. Instead of building an API and then building a client, we built the API to explicitly support the needs of the user interface. We would sketch out ideas for how we’d like to present the data, then work backwards from that to determine what endpoints were needed, what data they must expose, and how to format that data. Each endpoint allows for JSONP output, making use by JavaScript applications very simple. The API sits behind a Varnish cache server, significantly reducing the load on the database during periods of high traffic.

The user-facing application is powered by HTML and AngularJS. We use the Jekyll static site tool to automatically generate pages for each ward and service type, and to manage packaging the SCSS and Javascript assets. AngularJS is a powerful framework for building API-backed JavaScript applications. Angular manages all of the communication between the front-end and the API.

The site is hosted in a Amazon Web Service S3 bucket, making it very inexpensive to host, highly available, and simple to update.

We relied heavily on third-party libraries to speed our development. We acknowledge the following and offer our thanks:

  • Leaflet, Cloudmade, and OpenStreetMap: powering our beautiful maps
  • Gorilla: Golang tools
  • SCSS: the only way to write CSS
  • PostgreSQL and PostGIS: simple data munging
  • lib/pq: Golang + PostgreSQL
  • CapistranoRb: simple deployments
  • Jekyll: smart static-site creation
  • Jim Pravetz: Jekyll page generator
  • AngularJS: easy dynamic webapps
  • Bootstrap: simple grid framework
  • Highcharts: beautiful charts and graphs
  • Underscore: better Javascript

The Launch of Chicago Works For You

Smart Chicago

Today we’re launching a new project, Chicago Works For You, a citywide dashboard with ward-by-ward views of service delivery in Chicago.

Chicago Works For You The citywide dashboard with ward by ward views of service delivery in Chicago on Launch Day, September 19, 2013This site is going to be a central tool for Smart Chicago as we continue our work around internet access, digital skills, and civic apps. It allows us to engage residents, government, and developers around shared data so that we can all work together to make Chicago work better for all.

We’re conducting CUTGroup tests with residents, offering training and outreach to Alderman, and maintaining a full API of the data that drives our site.

This site is a result of years of work by many. It would not be possible without the vision of the MacArthur Foundation and the Chicago Community Trust to fund the Open311 project and the determination of the City of Chicago to make that implementation the largest and most comprehensive one in the world.

Much more to come. Write me at [email protected] with questions or comments.

Meanwhile, here’s a look at what this site does, taken from the About page:

Home

The homepage is a citywide map with a daily summary of all service requests submitted, by service type and ward.

Dark lines under and up-arrows next to a request type means there were more requests of that type on that date than average. The longer the line, the higher above average. Highest above average is highlighted on the map as default.

Click any service request type to see the raw numbers and averages. The legend in the lake shows you the number ranges for each type in each ward. Click any service type to see those numbers for any day.

Go back and forth in time, it will show numbers for that service request, updating the web address so that you can share any view via Twitter, Facebook, or email.

For advanced users who like to hack URLs, you can type in any date you want, going back to January 1, 2008, the first day for which we have data from the City.

Services

Choose any service request type from the Services menu to see weekly views of service delivery by ward.

The bars show all requests opened in each ward, displayed by the day of the week it was opened. Hover over any bar to see exact numbers of opened requests. The check mark is how many requests of that type were closed in each ward.

Click any day to see that service type on a map.

Move backward and forward in time to see the weeks go by.

Each service has details on the work it takes to close requests. Again, you can share any view and click links to view raw data or submit requests of this type.

Wards

Choose any ward to see the week in review– all open and closed requests. You can choose any request or view all of them at the same time.

The Time-to-close ranking shows the average time from open to close, per request, in context with all other wards.

The Days with the most requests is a raw count for each service type for this ward, going back to January 1, 2008.

The alderman name links to their ward website, and the Chicago start links to their contact information on the City website. See also Facebook account, Twitter stream, and legislative history (as compiled by Councilmatic) links.

Photos

See the last 500 photos submits with service requests. You can sort by service request and see raw counts. Clicking any image takes you to the detail page on the City’s Service Tracker website to see details and current status.

Annotations

All of the data, maps, and charts on this site are just mute representations of a 14-type portion of the services delivered by the city for the city. We use our Tumblr blog to make observations and solicit yours.

Update on Our Smart Health Centers Project

Kyla Williams

Smart Health Center Day with Chicago Health Corps at the Chicago Community TrustThe Smart Health Center project places trained health information specialists in low-income clinics to assist patients in connecting to their own medical records and to find information about their health online. The project is a joint effort led by the Alliance of Chicago in collaboration with the Otho S. A. Sprague Memorial Institute and the Smart Chicago Collaborative. We’d like to share a few updates about how the program is doing.

Continue reading →