Planet MoJo (Mozilla + Journalism)

Cole Gillespie: filtering through re:publica 2012

cole — Wed, 09 May 2012 14:55:44 +0000

The German blogosphere came together last week at re:publica to kick start Berlin web week. A few weeks before this Kai Bennerman approached me with an idea about tracking this conference as it was being talked about in real time via twitter. He had a few ideas about some different key words that he wanted to follow that were being tagged with the #re12 hash. My approach to building this widget first started with the streaming API that twitter provides the only downfall of this is that using the streaming api would require some server pieces to work properly and the assets were not available to build such a service in the time frame that we had to complete this project. I decided to use twitters public search api instead as my platform to build a jQuery plugin on top of.

The first idea was simple. Filter tweets that contain the following words: ’session’,'track’, ‘vortrag’, ‘talk’, ‘ and panel’ that were included in tweets that contained the rp12 hashtag. At first you would think that you would just include these words inside of the query string that we send to the twitter api but if you do that the results seem to get over filtered and raise the complexity of the query string which increases your chance of getting rate limited by the search api. Another problem with adding the items to the query string meant that twitter would try and return these tweets to you ONLY if all words were matched, I need an OR scenario not an AND.

My way around this was to query the search for only the hashtag #rp12 and then use javascript to search for the requested strings myself vs having twitter do it. As I searched through all the tweets I would keep track of how many I found that met the proper search criteria and once I had enough items I would cut off the query to twitter. While I am doing this checking as soon as I found a match I would create the DOM element and append it right away so it would be able let it scroll on the page and be visible to the user.

You can find the code on the openNews github.

Dan Sinker: Work at OpenNews: We're Hiring a Community Manager

Mon, 07 May 2012 16:23:00 +0000

As the Knight-Mozilla OpenNews program reaches its 2012 stride, it’s time to grow the team to help achieve the scale we’re looking for. So we’re hiring a Community Manager.

The entire job listing is here on the Mozilla Jobvite page, but I wanted to give a little background on the person we’re looking for that doesn’t break down as easily into bullet points.

At its core, the OpenNews program is about community: specifically around growing and strengthening the community of people making code for journalism. That’s what our fellowship program is about: sending individuals into newsrooms for ten months to create kick-ass journalism code. That’s what our hackdays are about: building momentum around code through events. That’s what our upcoming website Source is about: creating a central place to learn about the code being made and the motivations behind it. All of these pieces (with more to come soon) add up to helping foster a vibrant, growing community. But that’s a big goal, and it’s not going to happen without some help.

The Community Manager is what it says on the box: a person who can help to manage, foster, and grow the community around open-source code in journalism. It’s a unique person: Someone who’s outgoing, both in the world and online; someone who’s familiar with journalism, sure, but even more so someone who understands open-source communities and can communicate effectively with them; someone who’s well organized, and able to keep an eye on a lot of different, disparate balls at once; and finally, someone who’s able to collaborate effectively and efficiently.

Maybe you’re that person? If so, you should totally apply. We’re trying to fill this position as soon as possible, so no better time to get your application in than right now.

Oh, and you don’t have to move: we love remote workers.

Mark Boas: Drupal and Al Jazeera

Wed, 25 Apr 2012 08:33:00 +0000

As you can probably gather from the title, part of my remit as Knight-Mozilla OpenNews Fellow is to investigate and help implement open solutions. Solutions that will not only benefit Al Jazeera English but also other enlightened news organisations. Luckily AJE is already somewhere down the path to openness having released much of their content under a Creative Commons license. This license allows others to use this content without having to worry about royalties - this is of particular interest to me, planning as I am to build tools that will allow anybody to create new content by mashing up existing stuff.

A related approach currently being taken by AJE is to use the open source Drupal Content Management System to power the blogs section of their website. They’ve also had the vision to employ one of the key contributors to the Drupal codebase and community Dick Olsson. Significant customisation of the new CMS will be required in order to meet the requirements of a newsroom and so bringing in someone already active within the Drupal community makes good sense. I spoke to Olsson and his colleague Alaa Batayneh about how they saw this all panning out. We discussed architecture, testing and crucially the release cycle. It seems they already have an agile methodology in place and seeming like two very capable developers I imagine they will be able to push custom code pretty quickly. This is of course essential - as needs of the newsrooms evolve, so must the supporting framework. There will be no quicker evolution than at the very start but no system of this kind is ever finished. This will be real reactive user driven development.

The real beauty of using a popular open source library like Drupal (now on its 7th major iteration) is not only the fact that you start from a solid base but importantly that any modifications you make can be released back out into the ‘wild’ for others to test, enhance and benefit from. The modular approach is key - it means you don’t need to modify the base of the framework and so in theory your modules can be applied to new versions of the base as it is released. Further, by plugging together modules to make solutions that work in a certain environments — in this case news — you can create distributions that address those specific areas of industry. So although the modules are not necessarily news-centric, the distribution they make up is and it is this that AJE and Olsson are keen to release.

Assuming then that Drupal is properly modular and can be easily and comprehensively customised, I would expect an open strategy to yield very real benefits. This is an opportunity to take advantage of an eager and vibrant community of Drupal volunteers working on the continual improvement of the core of the product while initial customisation takes place in-house. Actually I am thinking of creating a couple of modules myself just for the fun of it. When something is both cool and open it draws curious developers to it like bees to honey.

Sometimes for large and established organisations to adopt new ways forward it just needs someone to lead the way, so here’s hoping Al Jazeera will do with systems what they did with content - that way we all win.

Mark Boas

Erin Kissane: Functional spec --> lo-fi wireframes

Tue, 24 Apr 2012 15:11:00 +0000

Posting in-progress lo-fi wireframes for Source based on our evolving functional spec. I haven’t broken down the code index or represented a filtering/search scheme yet, and will want advice from our designer/dev on that, but here are some starting points.

Nicola Hughes: The MISO Project: Making Interactive Story-Telling Open

Nicola Hughes — Tue, 24 Apr 2012 14:10:16 +0000

Friday saw the release of the keystone for a JavaScript Open Source data visualization library called Miso. It is a joint project between The Guardian and Bocoup, supported by The Guardian Global Development desk, who are funded by the Bill & Melinda Gates foundation.

The foundation of any data project lies in cleaning, sorting and connecting the data through all the steps needed for analysis and presentation. This is the lesson I am learning at the moment looking at a dataset of the order of 10,000-50,000 rows. There are no clear steps involved. There is no methodology. There is no “10 Steps to Understanding Your Data” guide.

What I have are pipes and a work flow system. These need to be constructed for each data project as they are dependent on the type, structure and cleanliness of the original data. Each node could be a parser, a refining tool or a database query. With each iteration of the data exploring process the data needs to be piped through various parts of this circuit. I talked about this in my previous post and will draw up my network flow and post my scripts to GitHub with every project I complete. A neat trick I’ve just learnt is to use command line scripts to pipe your data though individual parsing steps using standard in and out so as to avoid clunky paths.

Being able to see this circuit and having control over the flows make recursive journalism possible and the Miso release, Dataset, is built with this sentiment in mind. Alex Graul, Guardian developer behind the project, explains:

One of the most common patterns we’ve found while building JavaScript-based interactive content is the need to handle a variety of data sources such as JSON files, CSVs, remote APIs and Google Spreadsheets. Dataset simplifies this part of the process by providing a set of powerful tools to import those sources and work with the data. Once data is in a Dataset, it becomes simple to select, group, and calculate properties of, the data. Additionally, Dataset makes it easy to work with real-time and changing data, which pose one of the more complex challenges to data visualization work.

I’ve seen this working in action. I’ve updated a Google Spreadsheet and the resultant chart updated automatically. I am looking forward to working with the library as a journalist and a developer to really glean the benefits of Open Source. Building a properly documented and described piece of software is a massive job which needs to be done meticulously for the Open Source community to take it on. Miso is for anyone who wishes to call themselves a ‘developer’ or a ‘journalist’. It is organisation neutral and part of a growing trend of opening up newsroom tools.

To keep up to date you can follow @themisoproject on twitter and check out the code on GitHub. The lead developers behind the project are Alex Graul and Irene Ros.

Dan Sinker: Fantastic talk by the LA Times’ Ben Welsh about...

Mon, 23 Apr 2012 21:07:08 +0000

Fantastic talk by the LA Times’ Ben Welsh about algorithms, reporting, and people.

Dan Sinker: OpenNews: A Weekend of Hacking Journalism

Tue, 17 Apr 2012 20:02:00 +0000

photo by Aspiration Tech

There’s no better example of the global scale of the Knight-Mozilla OpenNews project than the dualing hack days we sponsored this weekend, in New York City and Buenos Aires.

In New York, we gave money for travel scholarships to bring top-notch developers to town to take part in the Wall Street Journal’s Data Transparency Weekend, which brought over 100 developers and privacy experts to town to create tools to “help people see and control their personal data online.” The hackathon grew out of the Wall Street Journal’s excellent ongoing series that looks at how your online footprint is being used by corporations.

The three-day event (documented extensively here, here, and here) resulted in code for almost 30 different projects with winners in “Scanning,” “Education,” and “Control” tracks.

photo from Hacks/Hackers Buenos Aires

Five-thousand miles to the south, we sponsored the Hacks/Hackers Buenos Aires ShowTimeLine Hackathon, which brought 45 developers together to work on making new time-line based visualization tools. The OpenNews sponsorship went to hosting the hack day, as well as a small amount of seed money to keep projects going afterwards.

The team of developers and journalists in Buenos Aires took a series of different approaches to displaying data over time, from automatic data-and-date extraction from documents to translating preexisting timeline libraries into Spanish, and more.

These are exactly the kind of topic-driven code-based events that we’re looking to help sponsor at OpenNews. If you’ve got an idea brewing for a journalism hack day, we’d love to hear about it. Let’s work together to make this year the year of journalism code.

Ben Moskowitz: Frontline/PRX Hack Day: Prototyping “A Perfect Terrorist”

Ben Moskowitz — Fri, 13 Apr 2012 07:41:06 +0000

Last weekend I was fortunate to take part in the FRONTLINE / PRX colloquium on interactive storytelling, organized by Andrew Golis, Carla Borras, Jake Shapiro, Andrew Kuklewicz and Sam Bailey. The event brought forth practitioners in documentary, journalism, and interactive to discuss new digital directions for FRONTLINE, and public media generally. The highlight for me—unsurprisingly—was the PRX-led hack day that followed. Though I love a stimulating conversation, I belong to the less yak, more hack guild.

All participants prepping the day before the event.

Four projects participated:

+ A Perfect Terrorist, an investigation of “the mysterious circumstances behind David Headley’s rise from heroin dealer and U.S. government informant to plotter of the 2008 attack on Mumbai.”

+ The Wounded Platoon, the story of the “3rd Platoon, Charlie Company, 1st Battalion, 506th Infantry… a group of young men changed by war.”

+ Post Mortem, a look at how death investigations take place in “a dysfunctional system in which there are few standards, little oversight and the mistakes are literally buried.”

+ a group focused on archival applications of Frontline interviews, starting with the recent Economy series.

I was drawn to the “Perfect Terrorist” group, probably because of a fascination with the Mumbai terror attacks. When I passed through Mumbai in late 2009, the Lashkar e Taiba attack made for very interesting conversations with Indian college students.

Tom Jennings, producer of "A Perfect Terrorist" gives a whiteboard run-down of the Collaba neighborhood and the "Kill Zone."

Our group wanted to visualize the complex web of relationships in which the American terrorist David Headley is centered. Headley is the mastermind of the 2009 Mumbai terror attacks—a man who was, in turns, an informant for the DEA and a member of Lashkar e Taiba. Producer Tom Jennings calls him a “perfect terrorist.” He is a complicated figure.

The FRONTLINE piece is outstanding, and the long-form journalism style has distinct advantages. But we wanted to help Tom tell this story in ways that traditional forms can’t afford. Though a documentary runtime must have a beginning, middle, and end, a web-native documentary can let users follow story threads that lead in many directions.

FRONTLINE has generated a ton of material around this story—not just the film, but a series on in-depth blog posts and features. There’s an opportunity for viewers to more deeply explore themes and events in the story, like Headley’s motivation for planning the attack. Just how does someone grow up to become a “perfect terrorist?” How was the attack orchestrated? How can we understand the terror network of 19 accomplices? Who are the victims? What is the aftermath? What are we to make of allegations that Headley was a double agent, and how do we begin to resolve some of the bigger questions raised by the tragedy?

Ideation

We imagined a ton of UX concepts. Among the ideas discussed: a navigable map of the “Kill Zone”; making use of perspective video from surveillance cams; a navigable 3D scene of the same; an attack timeline; a map of relationships; making a viewer clear the “fog of discovery”; and challenging users to ask questions and contribute to a discussion. But though these ideas appeal to the geek in me, they must be developed in service of story, or they’re not worth doing.

After cycling through story, then UX vision, then tech (in the style on which I insist!) we arrived at a pretty interesting concept. It’s a hybrid concept that gets at many of the ideas above.

We drew inspiration from the force-directed graphs in use by Mozilla Collusion and others. Without creating a “force” graph, per se, we wanted to create a visual representation of the “spider web” of relationships that Headley spun. With Headley at the center, we wanted to show the life events that culminated in a perfect terrorist recruiting and directing the 19 attackers, and how this ultimately affected the 166 victims of the attack and their families.

But we also wanted there to be an element of time control. Users should be able to experience this project passively, just by watching—but also actively, by navigating the timeline of events and seeing how the relationships developed.

A mockup prepared by Pietro Gagliano shows our interface ideas.

To do this, we placed Headley as a central “node ” on a vertical timeline. The timeline corresponds to a condensed narration of the entire Headley story. When the user presses play, the narration begins and Headley’s node travels through time. Users can go back and forth in time by scrubbing Headley along the timeline. Because Headley’s is basically a playhead on an audio track, we’ve casually taken to calling this “Playheadley.”

But Headley is also at the center of a time-controlled visualization. As Headley comes into contact with key players in the story, “nodes” are spawned for each person—along with connecting lines of varying levels of elasticity.

By watching this play out passively, you can see Headley’s placement in both the terrorist and informant networks evolve over time. And you can pause the experience at any time to explore a node more deeply. Click on one of the terrorists and the narration will pause. You’ll have the opportunity to explore B-Roll, blog posts, and other elements of the story in a non-linear (but intuitive) fashion.

Each node on the timeline represents an element of the story.

We also discussed how this story map could link to other projects in the same form. So you can imagine a tapestry of FRONTLINE stories connected to each other in this way.

This is an interesting way of mixing time and relationships, telling the story in a way that would be impossible in the traditional form. Check out Pietro’s rapid-fire summary deck for more detail: [PDF]

Hackin' away

Prototype

A protoype was made in six hours and is up on GitHub. It’s made with some custom d3.js voodoo by Devin Chalmers, a simple jQuery powered HTML audio player, and popcorn.js. Because of the short amount of time available for hacking, the project is pretty incomplete. But the repo has semi-functional code for all the individual pieces and is hours away from assembly.

There’s interest in taking this beyond the protoype, which I find tremendously exciting. I think there’s a lot of overlap with The Tillman Story project, and its objectives: to let users explore a much more expansive “surface area” of the story; to let interest drive the user’s experience; to provide additional context to the story; to connect it to journalistic material and conversations in social media; and to accommodate a large number of story threads while keeping the main thread engaging. Both of these projects are pioneering ways of telling non-fiction stories on the web.

A high-five for HTML5

As an open video booster, one thing about the event struck me as particularly fascinating: no-one blinked an eye that this was essentially an HTML5 media hack day. Until recently, this type of event would have been impossible. But thanks to the maturation of HTML media, to libraries like Popcorn, to human readable and hackable source, and to GitHub, these types of events are way more viable. Actually, rapid protoyping with video apps is way more viable—imagine having to coordinate with team members while compiling a Flash or iOS app and merging in functional and stylistic changes from six people on the fly. Very, very difficult. With HTML, we can make changes immediately and with little friction.

HTML5 media rocks. You can also check out the repo for the Wounded Platoon group, another HTML5-based project from the event.

Nicola Hughes: Data Journalism Is Iterative, What We Want Is Recursive

Nicola Hughes — Thu, 12 Apr 2012 23:02:45 +0000

One of the first things you will notice when you begin work at a big, busy, bustling news organisations is process. The many layers, overlaps and intricate weaves. The printing press is an elegant machine made of nuts, bolts and levers. Its processes, boiled down, are quite simple. But they are quite rigid and each system depends on the next. In other words, change is difficult, evolution stunted.

What we have in newsrooms today are people as processes and software as systems. We work on the web, through the web and so of the web. The newsroom is now an evolving organism. In getting out a story there is no longer a clear cut path through the newsroom hierarchy. For those of you unfortunate enough never to have worked in a big news organisation I can tell you that whilst a lot of the content produced in newsrooms look similar in their output, the people and the process behind its creation are vastly different from the next.

When you work with data, code and people, there is no set order, process or system. And that is necessary to the endeavour! For any story to be uncovered from data there will always be a data cleaner. Someone needs to be assigned data janitor. In fact, data archeologist would be a more suitable (and desirable) term. Data has layers. And with the peeling away of each layer, data needs to be sifted, cleaned, sorted, analyzed and hypothesized. The process is iterative and at each iteration the researchers, journalists, programmers, editors, etc all have to work in tandem.

: Tools for the job (from left): R project, Google Refine, MySQL

Most importantly, their working process needs to be closer to recursion than iteration. So what do I mean by this? In the first level of exploration the base case needs to be set i.e. the surface level of what needs cohering and cleaning, what rough outline of the data picture can we obtain, what’s the story we think it is telling and how can we best tell it. When breaking through the strata, we can’t afford to do the same thing again. So we write as many scripts as possible.

At the moment I am data janitor. That’s a good thing as it’s heavier on the journalism than it is on the code. I’m using Google Refine where I can export the data or just reuse the JSON. I’m using the query language SQL and the statistical analysis package R. With these, I can write the script and rerun it at each level of iteration. But as we dig deeper and learn more, we can ask more complex questions and paint more creative pictures. I can build up my scripts. We are not re-doing anything. We are never using the same process. Each time we are building upon the previous output.

This is the beauty of freeing yourself from the shackles of software. You only need to do something once and that allows you to loop through the process many times. In that sense, team work is most important, having all hands on deck at each level. You therefore need a multi-skilled team with a lot of crossover. Right now, I’m looking to be an entry level data archeologist, one of the first in the chain. For that you need a journalistic acumen, some faceting with Refine, knowledge of a query language, and some stats with R.

This level is really important in creating a data driven story. Making each level recursive rather than iterative could be the key to a data driven newsroom.

Dan Sinker: It’s only the start of April, and already it’s been...

Mon, 09 Apr 2012 15:05:37 +0000

It’s only the start of April, and already it’s been a big year for the Knight-Mozilla Partnership: We’ve placed four fellows at the BBC, the Guardian, Zeit Online, and Al Jazeera (a fifth fellow, at the Boston Globe, will be starting a little later this spring). We’ve renamed and refocused the Partnership under the Knight-Mozilla OpenNews name. We’ve begun sponsoring hack days around the world (in fact, two are coming up this weekend!). And we’ve started having bi-weekly open conference calls with the larger journo-code community (one is happening this Wednesday). And we’re only getting started—there is a ton more to announce, starting today with the Fellowship Application.

Just a few weeks ago, we announced the addition of four new news partners for the 2012/13 Fellowship cycle. With that, we’ve now expanded our partners to eight: The New York Times, the BBC, the Guardian, Zeit Online, Spiegel Online, the Boston Globe, ProPublica, and La Nación.

And today, as the video above explains, we’re announcing the opening of the window to apply to become one of our eight 2012/13 Knight-Mozilla Fellows. Starting today, April 9, and going until August 11, you can fill out the first round application. Borrowing from friends at the Knight News Challenge and Code for America, it’s designed to be quick to fill out, but also give us a broader sense of both your talents and your ideas.

If you’re a developer or technologist interested in helping to change the way people learn about and engage with the world around them, this is an incredible opportunity. There’s a ton more detail in the fellowships section of the entirely revamped OpenNews site, so give a gander over there and then apply today!

Dan Schultz: Back from Berlin