Posts tagged "google docs"

Google Docs + Miso-Powered Apps: a note on collaborative workflow

Working both collaboratively and efficiently on deadline, needless to say, is important. For our fastest-paced interactives and news apps, we’ve come to use a combination of Google Spreadsheets and Dataset.js from the Miso Project in a way that we’ve come to really quite like. We figured we’d share it with you here.

Essentially, we use a Google Doc as a temporary database for our text or numeric data. When we go live, we download that data to a local CSV, change the path, and deploy.

We’ve used this workflow on the Obama Haters Book Club, the sidebar to our election coverage page (which doesn’t exist on our site anymore but which looked like this), our Meet Romney’s Bundlers interactive, a variant of it for HavingTroubleVoting.com (where we had a Google Form power a Google Spreadsheet) and the Olympic Medal Tally among others. 

Note: This post is a bit more technical than usual.

The first step is getting everyone to put the information they gather into a structured spreadsheet. Here’s what the Obama bundlers spreadsheet looked like.

From there, we “Publish the spreadsheet” from the File menu — the crucial step when running an app off of a Google Spreadsheet — and import the data into our app. The Dataset.js documentation has a very simple tutorial on that step: http://misoproject.com/dataset/tutorials/googlespreadsheets

With this setup, reporters can be writing text and the photography department can be adding links to images at the same time the interactive is being built. What’s particularly great is an editor can edit the text in the spreadsheet and it will be piped into the interactive without his or her having to edit any HTML. The order of the rows can also be swapped if some elements need to appear before others on the page.

Mother Jones’s Tasneem Raja has written about this workflow as well, using Tabletop.js. Her post is definitely worth a read. 

So Why Dataset over Tabletop?

I’ve come to like Dataset.js over Tabletop.js for two main reasons. The first is that powering a live news app from a Google Spreadsheet can cause a lot of problems—so we download our data from the cloud onto our servers before going live. Dataset.js makes this transition really simple.

Jeremy Singer-Vine at the Wall Street Journal has written about the dangers of running live apps from Google Docs and he has a strong argument: 1) Google can rate-limit you, killing your live app; 2) If Google changes its API in the future your app may go offline; 3) Google can add extra data to your API request that can slow down your readers; 4) Sometimes you want to make a bunch of changes all at once, like add a complete row of information — with Google, incremental changes cell-by-cell are immediately reflected in the app, which will create broken elements until you’ve finished a full row; 5) Security and data privacy — sometimes you have columns in your data that you want to use internally but shouldn’t be public facing. We ran into this with HavingTroubleVoting.com where we collected ready contact information to later verify reports but we didn’t want to make that info public to the world. 

By making a local CSV whose columns we’re picking for public view, we solve a lot of these problems. We get the benefit of working collaboratively in development and can switch to something more archivable and robust for production. I have some other scripts that could be improved that will download a spreadsheet automatically and turn it into a clean CSV. If I can clean them up a bit we can post them as well. 

The second reason is Dataset.js gives your data a lot of functionality, such as the ability to group, query, and turn your columns into arrays (which is particularly useful for hooking it up to libraries like HighCharts). In fact, being able to hook up a Google Spreadsheet to your app is just an extra feature of Dataset.js — as the name suggests, it’s built to work with data and it can do some fancy things regardless of whether you’re working with a local CSV or a spreadsheet in the cloud. 

If you have any suggestions, I’m at @mhkeller.

-michael

HavingTroubleVoting.com is one of our latest projects to come out of our election coverage. It was born when fellow reporter, Allison Yarrow, whom I (Michael) sit across from, was talking about how we should do something to make voting issues fascinatingly interesting and digital. A Google Form, some reading about issues facing voters, and some CSS media calls later, we had a mobile-friendly submission form tied to a Leaflet map. 
We partnered with Mother Jones who had a similar project since the task of verifying so many reports (at last count ~700) is quite a process. As you can see, we took a different approach with this one: the site has a separate URL and we built it completely outside of our CMS. We took inspiration from the question-URL-based Guardian US project IsBarackObamaThePresident.com. I’m a fan because it lends itself to be more creative and have a longer shelf-life than a story that goes into a regular CMS, has a timestamp and all that. It also gives itself a catchy url that’s easy to remember. 
Under the hood
We ran into some issues being rate-limited using the results of our  Google Form as a data source so I wrote in some middleware with R that pulls down the form, does the geocoding for new entries (to avoid geocoding rate-limits) and let me put it on S3 easily.
Originally we were geo-locating the submissions based on browser location (since the main thinking was for mobile) but the percentage of successful tags was lower than expected. We also decided we would rather geocode based on the address people wrote in as opposed to their browser. I used a version of this geocoder to grab new entries every hour or so and make a new file to upload. 
We’re still going through the results and collecting more, if you have any, let us know! We’re pretty much always online at newsbeastlabs@gmail.com.
-michael

HavingTroubleVoting.com is one of our latest projects to come out of our election coverage. It was born when fellow reporter, Allison Yarrow, whom I (Michael) sit across from, was talking about how we should do something to make voting issues fascinatingly interesting and digital. A Google Form, some reading about issues facing voters, and some CSS media calls later, we had a mobile-friendly submission form tied to a Leaflet map. 

We partnered with Mother Jones who had a similar project since the task of verifying so many reports (at last count ~700) is quite a process. As you can see, we took a different approach with this one: the site has a separate URL and we built it completely outside of our CMS. We took inspiration from the question-URL-based Guardian US project IsBarackObamaThePresident.com. I’m a fan because it lends itself to be more creative and have a longer shelf-life than a story that goes into a regular CMS, has a timestamp and all that. It also gives itself a catchy url that’s easy to remember. 

Under the hood

We ran into some issues being rate-limited using the results of our  Google Form as a data source so I wrote in some middleware with R that pulls down the form, does the geocoding for new entries (to avoid geocoding rate-limits) and let me put it on S3 easily.

Originally we were geo-locating the submissions based on browser location (since the main thinking was for mobile) but the percentage of successful tags was lower than expected. We also decided we would rather geocode based on the address people wrote in as opposed to their browser. I used a version of this geocoder to grab new entries every hour or so and make a new file to upload. 

We’re still going through the results and collecting more, if you have any, let us know! We’re pretty much always online at newsbeastlabs@gmail.com.

-michael

Notes and images from an ever-growing digital newsroom.

Newsweek & The Daily Beast

Contributors:
Brian Ries & Sam Schlinkert

Formerly:
Michael Keller, Andrew Sprouse, Lynn Maharas, & Clarisa Diaz

view archive