KNC12 DATA | INVSTG8.NET: Deep data sourcing via simple data matching | Knight News Challenge 2012

One of my entries for the Knight News Challenge: Data.

INVSTG8.NET | Deep data sourcing via simple data matching

1. What do you propose to do? [20 words] aims to create a keyword/concept matching tool to help connect journalists to story-crucial peer-sourced data.

2. How will your project make data more useful? [50 words] enables journalists get access to, and use, otherwise inaccessible/unused, crucial data for stories in a highly targeted manner.

3. How is your project different from what already exists? [30 words]
The tool uses simple data in a trusted, secured framework to focus on the human connection to obtaining deep, targeted, existent but unsurfaced data.

4. Why will it work? [100 words] eliminates structural, practical, technological and financial barriers by linking journalists anywhere, who find it impossible to get data, to peers with access.
The tool will filter and match journalists seeking data with those likely to have access to it, in a highly targeted manner. will accelerate reporting on stories it would take months or years to tell, or that wouldn’t be told at all for lack of data. was inspired by an African journalist unable to obtain data crucial to his foreign mining investigation. He said it would take him months to get data I could get in minutes.

5. Who is working on it? [100 words]
Saleem Khan: Project leader, journalist [editor and reporter, ex- CBC, Metro International, Toronto Star newspapers; chairman/director, Canadian Association of Journalists]; advisor, University of Toronto ThingTank Lab [Faculty of Information]


  • K. Khan: User experience strategist and designer, OCAD University sLab advisor; leader of UXI, Canada’s largest UX professionals group
  • H. Leson: Director of community engagement, Ushahidi; open source community developer, library and information technician.
  • M. Saniga, CA: President and co-founder, near-realtime business intelligence/data insight generation software firm Quant Inc.; former finance director and manager at Cara, Dell.

Code team:
Various ad hoc/pro bono.

6. What part of the project have you already built? [100 words]
A functioning development-environment prototype of the query, receipt and request-handling mechanism that accepts open SMS has been built. Code is available at [ ].
The keyword/concept based request engine that forms the core of is still needed.

7. How would you use News Challenge funds? [50 words]
Funds would be applied toward development of the keyword/concept request-matching engine and related support. At its core, this means hiring dedicated developers to advance the project at a pace far faster than the current, incrmental rate of progress.

8. How would you sustain the project after the funding expires? [50 words]
We have encountered venture capitalists, private companies, NGOs and news organizations that have expressed interest in investing in once a working prototype is available, or in purchasing beta or release-candidate licences.
We will also sustain the project by selling consulting services and training, a freemium pricing model, and ancillary merchandise.

Requested amount: $350,000
Expected number of months to complete project: 12
Total Project Cost: $525,000
Name: Saleem Khan
Twitter: @saleemkhan
Email address [optional]:
Organization: / Technovica
City: Toronto
Country: Canada
How did you learn about the contest? I have been following and participating in the Knight News Challenge for years.

KNC12 NETWORKS | InvestigateNet: Microcollaboration for journalists | Knight News Challenge 2012

My Knight News Challenge proposal for a microcollaboration tool for journalists to eliminate barriers to newsgathering and foster collaboration among journalists everywhere.

InvestigateNet: Microcollaboration for journalists

1. What do you propose to do? [20 words]
Enable journalists to surmount structural barriers to newsgathering, through microcollaboration, by linking them with social tools and technologies.

2. Is anyone doing something like this now and how is your project different? [30 words]
Services like Quora, StackExchange, WikiAnswers, LinkedIn Answers, Facebook Questions, and journalism discussion lists/forums, are nominally similar, but address neither the South’s technology limitations, nor journalists’ structural, privacy or security needs.

3. Describe the network with which you intend to build or work. [50 words]
Investigate Net enables microcollaboration on a local to global scale.
It connects journalists around the world to peer-sourced information unavailable otherwise, through low-bandwidth communication technologies such as:
– lean/mobile Web site
– text or SMS messaging
– e-mail
It uses open source technologies [e.g. Apache, FrontlineSMS, osTicket], and connects through APIs to Twitter, Facebook, etc.

4. Why will it work? [100 words]
InvestigateNet eliminates structural, practical, technological and financial barriers by linking journalists who find it impossible to get information to peers with easy access.
Explainer video: [Long link]
InvestigateNet was inspired by an African journalist unable to identify the value and uses of mined minerals: Experts wouldn’t talk. Poor Internet access and a basic mobile phone limited his communication.
The information he sought, I could get in minutes.
Journalists at the Global Investigative Journalism Conference in Kiev in November 2012 who expressed enthusiasm to use InvestigateNet included the Forum for African Investigative Reporters, Global Investigative Journalism Network, and Investigative Reporters and Editors.

5. Who is working on it? [100 words]
Saleem Khan is a Toronto-based journalist, founded Innovate News; and journalism working group chairman at the University of Toronto ThingTank Lab.
Innovate News gathers executives, managers, journalists, designers, technologists, and academics to resolve challenges facing journalism.
ThingTank Lab is an open community lab that experiments, prototypes and makes digital/physical media, i.e., “Internet of things.”
Saleem launched, managed, edited and reported for the Canadian Broadcasting Corporation’s (CBC) technology news service, was news and global technology editor for Metro, the world’s largest international newspaper; and was an editor at Toronto Star newspapers.
He was chairman and director of the Canadian Association of Journalists for a decade.

6. What part of the project have you already built? [100 words]
We have developed the concept, produced paper-based prototypes of the system and how it works, produced a basic mock-up video, and identified open source tools and technologies that we will use to build this project.
The main obstacle to InvestigateNet is a lack of funding. The project lead is not a programmer [but is learning to code], and relies on rotating volunteer programmers and user experience designers with expertise in open source development, software design and security.
Funding would allow us to focus on building a working prototype and iterate into a viable tool.

7. How would you sustain the project after the funding expires? [50 words]
We have encountered venture capitalists, private companies, NGOs and news organizations that have expressed interest in investing in InvestigateNet once a working prototype is available, or in purchasing beta or release-candidate licences.
We will also sustain the project by selling consulting services and training, a freemium pricing model, and ancillary merchandise.

Requested amount from Knight News Challenge: $350,000
Expected amount of time required to complete project: 1 year
Total Project Cost: $350,000
Name: Saleem Khan
Twitter: @saleemkhan
Organization: Technovica
Country: Canada

Investigate Net: Microcollaboration for journalists

Investigate Net

Investigate Net MozNewsLab proposal from Saleem Khan on Vimeo. ]



Investigate Net enables journalists to surmount structural barriers to newsgathering.

It uses simple, readily available technologies to help journalists find and collaborate with colleagues anywhere.

Investigate Net home page mockup

Investigate Net home page mockup.

Investigate Net mobile mockup.

Investigate Net mobile mockup.





Investigate Net was inspired by an African journalist I met, who was unable to identify the value or uses of minerals mined in his country because the experts worked for mining interests and would not talk to him.

Research was difficult with a maximum of 30 minutes per week on a dial-up connection, and a low-end mobile phone for most of his communication.

He said these problems were widespread in Africa.

The information he sought, I could easily get in minutes. That was the seed of Investigate Net: Information one journalist found impossible to gain was easily accessible to a peer.



Journalists in one locale are forced to shelve or discontinue investigations because they cannot obtain crucial, basic information due to access, technological, financial or practical barriers.

That information is easily accessible by journalists elsewhere. Yet, it remains out of reach to isolated journalists because collaborations typically stem from personal networks and both groups are unaware of each other and their needs.



Investigate Net enables micro-collaboration among journalists on a local to global scale.

It connects journalists around the world to peer-sourced information that is otherwise unavailable, through low-bandwidth communication technologies such as:

  • a lean or mobile Web site
  • text or SMS messaging
  • e-mail



A journalist submits an information request and other pertinent information, such as keywords and privacy considerations, through a simple, mobile-friendly, Investigate Net form online, in e-mail or text message.


Investigate Net request form

Investigate Net request form mockup.


The query goes to journalists who self-identify as experts or interested in subjects noted in the request. They obtain the sought information and send it to the requesting journalist according to preferences: E-mail, text message, fax, etc.


Investigate Net e-mail inbox notification

Investigate Net notification deliveries are set by the user.


Journalists in the economic North with abundant technology or information access may also need sources outside of their networks, especially when a story leads abroad where their news outlet has no presence or contacts. It is impractical to always fly a correspondent there, so the story is shelved or killed.

Instead, Investigate Net would enable Southern journalists to become virtual field producers.



Investigate Net builds on existing technologies, tools and behaviours familiar and accessible to journalists:

  • Web forms
  • e-mail
  • text messages
  • and the reporting process.

It uses simple, form-based submissions to collect and disseminate information requests.

Investigate Net request and fulfillment flow

Journalists in the North and South can send and receive information through distributed servers. If technology limitations are a factor, requests and fulfillment can be sent and received in off-peak hour batch operations.


For journalists with limited Internet access, batch-send operations could process locally cached requests during off-peak times.

Investigate Net fits with a newsroom’s reporting workflow: Research, make calls, interview and synthesize it all into a coherent context. The outcome is information but benefits are greater.


Sketch: News story timeline

News story timeline


Fulfillment is good professional courtesy. Journalists privileged to work in the North are responsible — perhaps obliged — to help less privileged colleagues in the South.

Northern journalists realize reciprocal benefits and discover stories they may not otherwise find.

Above all, these simple information requests can help all journalists build networks and foster collaboration.



Several open source projects could be used to build Investigate Net:

  • FrontlineSMS to manage and transmit requests and answers.
  • osTicket support system as a backbone to handle and track requests.
  • Apache projects as the server and Web management platform.
  • The recently launched Investigative Dashboard portal cites collaboration as a goal and could enhance Investigate Net. (I discussed the problem Investigate Net aims to solve with Dashboard’s Paul Radu at the Global Investigative Journalism Conference in 2007.)

Journalists can easily participate by:

  • identifying gaps in their information needs
  • helping to design request and fulfillment processes
  • testing as end-users.

Developers bring key technical knowledge to build Investigate Net. Because requirements are simple by design, it will be a quick and easy build at minimal cost.

Editorial and technical collaboration should be smooth as the system’s simplicity in scope and concept minimizes miscommunication.



Journalist participation in the North in particular is not guaranteed. It may be necessary to add incentives or game mechanics. This will be revealed in iterative testing.

Privacy and security beyond the browser connection may be needed. Tools like Tor and profile management features may address them.

Authentication occurs at a device level but does not address device sharing, or human error that grants access to a user’s Investigate Net profile. This is an issue for future development.

Services like Quora, StackExchange, WikiAnswers, LinkedIn Answers and Facebook Questions, and a range of journalism discussion lists and forums, are nominal competitors but do not address the South’s technology limitations, structural, privacy or security needs.



Investigate Net will enable and extend the capabilities and impact of journalists and news organizations:

  1. Barriers to basic and critical information will be erased, letting journalists focus on developing deeper stories instead of scraping for simple facts.
  2. News organizations will be able to better allocate personnel and financial resources, save significantly on story-related costs and improve productivity.
  3. Journalists will be better linked, fostering work and co-productions.
  4. Journalists will gain information, and in the South, income, too.
  5. Stories that would not be told, or would fall short, will gain life and momentum. You cannot report on something if you cannot meaningfully identify and understand it, or if the trail leads somewhere and you have no budget, presence or network to follow it.
  6. Deeper co-productions will stem from Investigate Net collaborations, making it possible to maximize editorial impact on a local, regional, national or global scale and drive the news agenda. A story or investigation by one news organization is easy for authorities to ignore; one covered by multiple outlets is not as easily dismissed.


Investigate Net is meant to be extensible and scalable.

It could extend to amateur journalists, commerce, rich media distribution and more.

Ultimately, Investigate Net will foster collaborations and joint investigations that tell local stories in the North and South and span the globe.


[UPDATE 08.21.2011: The video at the page linked via the Investigate Net image above has been corrected. +08.15.2011: is not the correct one but I am leaving it there for historical context, at least for the moment. A blog post on this is forthcoming. The actual Investigate Net proposal video is online.

Bridging the divide

Any technology project

should solve problems  journalists are having,

even if it’s a problem they don’t know they have.

Mohamed Nanabhay
Al Jazeera

This week’s Knight Mozilla News Lab lectures — particularly those about technology in the newsroom — resonated deeply with me. They recalled challenges I have faced in my career and mission to inspire, encourage and entice my professional peers into looking at journalism from a broader perspective than that to which they are accustomed, and in building bridges, particularly with developers. They echoed much thinking and some exposition that went into my featured post last week, about the divide between news and open development cultures.

The lectures made it clear that although there has been progress, the challenges I previously identified are far from a relic of the past.

It’s something I pursued with developers in my own news organizations with varying success: Senior managers, often traditionalists, have typically put a stop to this sort of collaboration as a poor use of time, typically because they don’t understand it.

What is now changing is that there is a realization that developers have a key role to play in enabling deeper, richer, more accessible and diverse storytelling, and that everyone in a news organization — and increasingly, outside of them, too — are potential partners and collaborators.

News process flow at MozNewsLabtO

News process flow at MozNewsLabTO


The question

How does your project take into account the need to facilitate collaboration in the newsroom (whether real or virtual), while acknowledging that team members will have varying technological skill sets?

Journalists work in a competitive environment in a way that is at odds with other fields, especially open source technology and software. Reporters in particular compete not only against external rivals, but  against peers in their own organizations. Every day, they have the challenge of making their story the top story. Deep, meaningful collaboration typically occurs only in small units, if at all.


Sketch: News story timeline

News story timeline


That is why Al Jazeera’s open approach is particularly intriguing: Releasing content under a Creative Commons licence,  and beyond that, the tools it develops, as Mohamed Nanbhay said:

Open source is important to us.

It’s important that we put back into the public domain…

[and] leverage off of each other.


I strongly believe and have long argued that this is the best approach.

The biggest limitation on technology adoption in the newsroom is journalists’ willingness to use it, which is often a function of time and skill. As Mohamed Nanabhay noted:


The newsroom is fast-paced.

There’s not always time to sit back and reflect on how we use technology.

Realtime journalists: "We sometimes had almost two minutes' time before we had to publish."

The newsroom is fast-paced.


User-friendly, simple, easy and easy-to-understand collaboration is the core of my Investigate Net tool, the thin edge of a wedge to create a collaboration culture, even for technophobes. It uses familiar skills and behaviours to fill a need of journalists everywhere: Getting high-quality, reliable information, quickly, easily and cheaply — even for journalists who don’t know it’s a problem they have.

Open source, closed mind: News technology as cultural Trojan


Few developers understand the non-technical issues that go into an open-source project.

jQuery. Khan Academy


Dilbert: Another journalism major enters the workforce. Hired journalism major to walk around flapping arms to keep motion-controlled lgihts on.

What developers and engineers think of journalists .

A common theme has emerged among the Knight-Mozilla Lab lecturers: Culture, its dynamics and how we approach it will be our hardest task.

Every lecture has focused on or mentioned culture and related issues as a key point. Among them: John Resig on open source project development, management, community, and lessons from developing jQuery.

As he talked about the challenges, I couldn’t help but reflect on how starkly the open source development approach contrasts with news culture. The former is an open, accessible framework with a bottom-up, grassroots structure. The latter is a closed, restricted framework with a top-down, hierarchical structure.


Sketch: News is a reductive process

News is traditionally a reductive process that sheds people.


Although news organizations are starting to collaborate in a more open manner, even large ones may take a token approach, asking people to submit photos or to follow social media accounts.  I have little doubt that substantive collaboration and implementation  will be an issue for many — especially those who have not experienced or don’t understand news and journalist culture. I saw and experienced it where I worked. In particular, developers were the strange, mysterious wizards who made things work but journalists didn’t know how and were suspicious or indifferent to their ideas.

However, adding Resig’s principles will strengthen my project — a collaboration tool for journalists that bridges the industrialized and developing worlds — and could help it foster a more open, connected, community-driven culture by first extending the trust journalists have for professional peers.


Sketch: Software prototype development

Open source code and prototyping add people in process.



Resig raised important points, some of which are already elements of my project:

  • Understand what your users are trying to achieve. It will help you to create a better product. I talk to journalists all the time about their needs.
  • Make sign-up as painless as possible. It would require little more than a phone or fax number, or an e-mail address.
  • Treat every user as a potential contributor. The concept behind the tool is contribution, so it should be easy to migrate that behaviour to bug reports, documentation, feature requests, etc., per Resig.

Other points require further thought:

  • Make documentation as accessible as possible. I initially thought the tool would roll out to English-speakers first but now see that may limit uptake and introduce cultural bias into its design. Full documentation will have to be in multiple formats and languages.
  • Provide places for people to ask questions. My focus was so tight on my archetypal user, I didn’t consider all channels by which users might contact me. Set-up is underway.
  • Answer questions every day. You can’t be lazy about it. Now planned: Answering support questions.
  • Have an open process. Make decisions with public input from the community. I already take community input privately. This will expand as the project unfolds.

And, hopefully, change news culture.

News culture vs. sketch culture: On Aza Raskin on prototyping


The hardest part of software

— and design in general —

is neither design nor software.

It’s culture.

Aza Raskin



Aza Raskin | MozNewsLab Lecture - Prototyping and influenceAza Raskin: Prototyping and influence.


What Aza was talking about was change.

The hardest part of change is culture.

That’s what we’re trying to do: Change culture.

Why is it so hard? We’re creatures of habit.

For journalists, the rote, ritualized, status quo culture emerged because there can be serious, costly — even dire — consequences when one deviates.

It infiltrated my approach, which I didn’t realize until Aza’s talk.


Prototype culture vs. news culture


Aza Raskin | MozNewsLab Lecture - Prototyping principles

Aza Raskin on prototyping principles.


1. You are going to get it wrong the first time.

In journalism, it’s the reason a process and culture of verification emerged in news gathering, production and delivery. But there’s a premium on getting it right the first time.


2. Finish  the artifact in a day.

Journalists do this daily: We make multiple products (stories) combined into another (newspaper, broadcast, site, etc).


3. Iterate fast. Dogfood much.

News narrows vs. branching. It typically goes through many drafts until we think it’s right. We stake our careers on it.


4. It’s a sketch, don’t fill in all the lines.

This is where traditional journalism culture breaks down. Sketch culture, prototype culture and their processes are somewhat antithetical to journalism. Or, they were.

Now, a new ethic has emerged: Process journalism is an analogue to sketching. News is reported as it happens, before the full story is clear. It’s corrected and adjusted in process.


5. You will change the problems you are trying to solve.

In journalism, we answer a question: “What is the story?” It rarely changes.


6. Plan to throw it away.

This happens every day in news but not ideally. So it did in my process.


7. Steal it [visual design].

The journalistic equivalent is plagiarism. Forbidden. Career-killer.


My approach: Aza and me

After Aza’s lecture, I realized my approach was too rooted in journalistic tradition.

One of my first "sketches": Hyper detailed wireframe.

Doing it wrong: Hyper-detailed wireframe as a first "sketch."


1. I took my news challenge submissions as the answers before moving beyond ideas.

2. I hadn’t made an artifact for want of process and technical skill.

3. I hadn’t iterated beyond my original thinking. This ruled out dogfooding.

4. My sketching was more like layering: High-fidelity sketches rather than basic ones to test ideas.


MozNewsLab: Doing it wrong 2: Hyper-detailed sketch.

Doing it wrong 2: Hyper-detailed sketch for MozNewsLab


5. Instead of changing the problem I was trying to solve, I was driving toward solving the problem I convinced myself I’d defined.

6. Although I was conceptually prepared to throw everything away, in practice, I did the opposite: I kept everything as though I was constructing something defined.

7. I didn’t reach the (journalistically heretical) point of using another’s visual design, but it’s helpful to know it’s not only acceptable but encouraged.



Based on Aza Raskin’s lecture and the resultant observations on my own process, I decided that to get the most out of the lab, I should start a new project that applies the principles, lessons — and culture — gained along the way.

So, here I go:

Sketchbook and pens:  "You have to be pretty technical before you can even aspire to crudeness." Johnny Mnemonic by William Gibson.

"You have to be pretty technical before you can even aspire to crudeness." — Johnny Mnemonic /William Gibson


What has gone before will have to wait

A brief administrative note: If things are looking a little sparse around here, there’s a good reason for it.

One of the things that I’ve learned over the years is that things rarely go as planned. So, when I dutifully clicked to automatically update this WordPress installation to the latest version, I really shouldn’t have been surprised when all I got were internal server errors.

After backing up everything, creating a new MySQL database and reinstalling WordPress three times, older content (except, somewhat inexplicably, the first post to this blog) will be inaccessible for the time being. It’ll come back along with the old (or maybe new) template and plugins when I get a chunk of time to identify and repair whatever has gone wrong.

In the meantime, the new posts will have to do.

BREAK THE NEWS: Shape, expore, remix

The Knight-Mozilla News Technology Partnership is seeking ideas on how to reinvent journalism for the open Web, so they’re running a series of challenges. The first Knight-Mozilla challenge focused on “unlocking video,” the second aims to solve the news comments problem by going beyond comment threads, and the third is a blue-sky effort to find the “Open Web’s Killer App,” later narrowed to “People-Powered News.” [which I think is too narrow a focus but was already the foundation of my idea]:

What should a news website look like in 2011 and beyond?

MoJo Knight Mozilla News Technology challenge open Web's killer app logo / people-powered news

This challenge is a broad one: what would you build on the web that actually makes news better for the people who create and read it? How would you involve the public in the news making, editing or sharing process? The only constraint: your idea should be built using open technologies and languages.

We’re asking you to forget old conventions like the story form, column inches & deadlines and propose new ways to connect news producers to news readers.

Here’s my proposal:

BREAK THE NEWS: Shape, explore, remix


Break the News dismantles the traditional news collection and dissemination structures and reassembles them in a unified approach that allows the news community to shape, explore and remix.


The future of news is open and collaborative, but also evolving to incorporate a number of new technologies, modes of access and devices, some of which are not yet widespread.

The best prototype of where news is headed can be seen in Al Jazeera’s The Stream, which bridges the Web and traditional broadcast, aggregates crowdsourced story ideas, videos and other content, and engages its community in an authentic conversation.

The Break the News model is similar to this but recognizes that the principal nature of a thing changes when its scale and medium change. To this end, the new model of news resides on the open Web, assembled into objects for context-aware aggregation, curation and dissemination of visual content, conversational streams, geolocative, gestural, haptic, collaborative and algorithmic layers for the bridge to emergent devices and physical-world intelligent/enabled objects.

As commonplace objects become both displays or output devices as well as seamless input or information-gathering devices that are holistic and pervasive in both of those modes, the Break the News integrated platform will interactively and intelligently employ these channels, either automatically or with the assistance of a human operator or permission-based automated AI/expert system.

Among the Break the News digital objects is a collaboration channel or tool that enables people to securely share and manage information, documents, multimedia or other binary data for potential initiation of, or inclusion in, a “story.” [The word “story” is expressed in quotation marks to denote that the traditional narrative forms that have been widespread and commonly accepted until now is just a subset of what we will soon come to consider a story.] This content can be handled with varying degrees of security and distribution, contingent on the wishes of the source or agreement between source and journalist.

Break the News would also enable its community to remix the streams for different perspectives and relevant understanding of context.

This system would span both a high-fidelity Web experience and a low-fidelity one to account for regional disparities in network access and technology. Break the News would be a medium to convey and collaborate on information, minimizing the barriers between people and access to resources, including those among journalists. Web-based gateways could bridge low-fidelity information technologies outmoded in the North to the cutting-edge technologies that may not be commonplace in South.

Break the News is more than the sum of its parts, some of which I have already described more fully in proposals for video and commenting challenges, and much — if not all — of which is possible today. It is a platform that shatters the procedural, institutional and cultural barriers in the daily practice of journalism to instead allow the free flow of ideas and information for a more open, transparent and democratic model of news media.

Comments Holistic Open Platform: How to fix news comments

The Knight-Mozilla News Technology Partnership is seeking ideas on how to reinvent journalism for the open Web, so they’re running a series of challenges. The first Knight-Mozilla challenge focused on “unlocking video.” The second aims to solve the news comments problem by going beyond comment threads:

One of the best things about the web is that it enables many voices to be heard. Blogs, comment threads, forums, and social networks empower people to take part in new kinds of discussion, dialogue, and debate.

The best discussions around the web can be pretty isolated. Take comments, tweets, and other fragments out of their original context, and they can become meaningless. And take a look below the fold—in comment threads at news outlets, political blogs, YouTube, and elsewhere, you’ll often find that the loudest voices drown out everyone else.

Knight-Mozilla News Challenge - Beyond Comment Threads logo: AtomAt the same time, media is moving beyond the traditional “news story” as the only unit for commenting and interaction, stretching to include narrative arcs of multiple stories over periods of time, “explainers” that provide background knowledge for strings of stories, “streams” that include initial reports followed by updates and corrections, and more.

With all that activity happening across the web, how do we enable more coherent, elevated discussion? How can news organizations improve the signal-to-noise ratio in public news commentary?

Here’s my proposal:



This is a holistic approach to what I see as the five major problems facing comments today and in the near to medium term.

  1. QUALITY. The major problem in commenting on news sites today is finding a way to maintain high-quality discussions.
  2. DISCOVERY/PERSISTENCE. Commenting is fragmented and no system properly federates them.
  3. COMMENT FORMATS. No truly transmedia commenting system exists.
  4. CONTEXT AWARENESS. Commenting systems don’t address people’s needs in various device, temporal and physical contexts.
  5. EMERGENT MEDIA. There is no standard for comments in emergent media and platforms. Each element of this plan can be developed separately, or as part of a phased, holistic solution.




Maintaining high-quality discussions.

    a) Community ranking / moderation + TrustRankCommunity ranking’s fundamental flaw: It rewards popular ideas and unpopular ones are often submerged. To offset this, an algorithm could assign a TrustRank score that surfaces comments from trusted people on a sliding scale weighted by the viewer. TrustRank improves over time and volume of comments.
    b) Semantic / sentiment parsing e.g. SentimentRankCommunity or algorithmic ranking and/or parsing of meaning and sentiment would assign a score viewers could weight. SentimentRank would find commenters with a similar temperament and outlook and help determine which comments they see. This improves over time and volume of comments and participants.
    c) Crowdsourced or automated summaries of longer comment posts
    As comments accumulate, reviewing and understanding the discourse can become onerous. Volunteers or an algorithm could identify and summarize key themes for brief, headline-style summaries to help viewers discover and understand context.
    d) Aggregate summaries and sentiment scores into heatmap / graph / that surfaces key points
    Rankings can be visually displayed in a map or graph. This would help to surface key thematic comment clusters that the viewer could drill down on for finer granularity.
    e) Badges / incentives / credits or scrip toward paid services for commenting
    Commenters receive rank badges to enable viewers to quickly assess the quality of an individual comment in a historical context. This could be combined with credit or scrip system that news organizations can use to reward commenters toward paid services. The open platform would make rewards portable across outlets using it.
    f) Trust circles, connected communities on other networks, TrustRank + SentimentRank sort
    Subject to individual preferences, people see comments from friends in social networks, extended networks, then people outside their networks.
    g) Present commenter with others’ comments inconsistent with own views
    Combine factors to offer viewers comments that oppose their own, to help stimulate meaningful debate vs. a cargo cult.



Commenting is fragmented. While multiple platforms exist to federate comments, they still occur in isolated islands. To address this:

    a) Open standard that federates and categorizes comments
    Federate across services: blogs, status, chat, photo, video, text, SMS, etc. (Opt-in.)
    b) Visual / audio / tactile clustering
    Comment systems are heavily biased toward educated, literate, able-bodied individuals. Inclusive commenting systems would also assist the fully able. Standardized/automated markup would help identify and enable content federation for the multimedia and emerging sensory/haptic Web and enabled devices. People could navigate comment heatmaps parsed for subject, sentiment and trust through visuals, audio or tactile/force feedback. Similarly, content types could be toggled as comment options.
    c) Static vs. real-time
    Static comment threads are easy to hijack. Real-time commenting and discussion archived or parsed for inclusion in a historical forum/thread would enable actual discussions vs. turn-based commenting and repetitive or irrelevant crosstalk.



Transmedia comments vs. plain text.

    a) Text – Real-time commenting and discussion into archived/threaded forum enables collaboration with journalists on a story before publication-to-deadline, and fosters higher quality ongoing commentary and story development post-deadline. Granular word/sentence/paragraph level comments can be tagged.
    b) Images – Image-post comments for people with varying time, literacy and forms of self-expression. Character recognition and transmedia publication for any text within images, sentiment-parsing for facial expressions or gestures, and image summaries can be extracted and posted via community scoring or algorithm.
    c) Audio – Audio comments can be parsed as text and ranked for sentiment and trust.
    d) Video – Video comments (time delimited) can be parsed into text posted with and/or annotating video. To sort multiple video comments, they would be thumbnail stacks akin to BumpTop’s concept. Brushing a clustered stack would surface the video thumbnail on a card with the commenter’s profile, semantic TrustRank + SentimentRank score and/or graph, personality matching, and a capsule comment summary.



The type of comment a person can leave often depends on time of day or their physical location:

    Office / mobile: Likely fosters shorter comments.
    Home / tablet: Likely fosters longer comments.
    Sorting and serving up comments to viewers and commenters based on set preferences, time of day and physical contexts including Web-enabled objects could foster higher quality discussions by reducing the frustration factor for commenters and viewers.



    a) Projected/sensors: Pico-projectors, digital vision, tactile and other sensors can shift comments from a solitary experience to a shared one. Collaboration in physical space is essential.
    b) Wearable: Personal augmented reality and wearable displays demand that comment systems include the ability to interact in real-time and with geolocated and physical, Web-aware objects.

First Sources Video: A secured transparency platform for video


The Knight-Mozilla News Technology Partnership is looking for ideas on how to reinvent journalism, especially on the Web. To that end, they’re running a series of challenges. The first focuses on “unlocking video“:


Video is a central part of many people’s daily news experience. But most online video is still stuck in a boring embedded box, like “TV on a web page,” separated from the rest of the page content. This offers little in the way of context or opportunities for viewers to engage more deeply.

New open video tools make it possible to pull data from across the web right into the story. Information related to the video can literally “pop” into the page. And videos themselves can change, dynamically adapting as stories evolve. The challenge is to use these tools in ways that serve the story. How can we enrich news video through things like added context, deeper viewer engagement, and the real time web? What are the untapped possibilities inherent in many-to-many, web video?

Here is my entry:

FIRST SOURCES VIDEO: A secured, open platform for crowdsourced, trusted, pseudonymized and anonymized video.


First Sources is a secured, transparent video dissemination system that can be deployed in any locale, to any degree of granularity to free not only journalists, citizens, whistleblowers and other people of conscience to act in the public interest, but governments and other institutions as well.

First Sources will enable people and organizations to securely submit video and information anonymously or pseudonymously in real-time or asynchronously from any device, to an openly accessible platform so journalists and citizens can collaborate to surface public-interest information on-demand, or that may otherwise remain submerged.

The core of this system would be the ability to transmit anonymized or pseudonymized video securely while enabling participants to issue alerts for supply and demand of geolocated news.

Later phases of the implementation would apply either real-time machine or crowdsourced translation and subtitles, and make it possible for journalists and the public to collaborate around video objects. This collaboration would include but not be limited to real-time discussion, remixing and creating contextual narrative using other openly available online resources such as status updates, knowledge resources such as Wikipedia, online news and more.

First Sources’ initial phase or iteration would be primarily enabled by Tor or a similar technology, HTML 5 video, Popcorn and Butter.


The partnerships between established and credible news organizations and the whistleblowing documents publisher WikiLeaks have dramatically reminded us of the power of documentary evidence to enable journalists to tell stories that alert and inform the citizens in a democratic society of how their public institutions operate — and of the news they don’t see. The bulk of this material is text, with notable exceptions such as the Collateral Murder video.

Waiting days, weeks, months or years for troves of text to be released poses a problem not only for dissemination of news and information needed in the present, but for consumption, comprehension and action: Humans are visual creatures.

One need only look at the movements for change boiling up across the Arab world to see the power of information, networked communications and bearing witness in person or from afar. Video is a key part of this equation.

The risk to those who would supply this video, real-time or short-term reportage and information is great.

First Sources is a secured, transparent video dissemination system that can be deployed in any locale, to any degree of granularity to free not only journalists, citizens, whistleblowers and other people of conscience to act in the public interest, but governments and other institutions, too.

First Sources will enable people and organizations to securely submit video and information anonymously or pseudonymously in real-time or asynchronously from any device, to an openly accessible platform so journalists and citizens can collaborate to surface public-interest information on-demand, or that may otherwise remain submerged.

Similarly, enlightened governments and other institutions could use such a platform to proactively release video and information to create and sustain an atmosphere of public transparency. By doing so, citizens could anonymously or pseudonymously retrieve the released video without fear of being monitored and its potential consequences.

By combining and automating the secured identity anonymizing/pseudonymizing function within the system, it helps to ensure that journalists, witnesses, whistleblowers or users of that video receive the maximum possible identity protection and minimizes the potential for reprisals.

Once deployed, the system would be openly accessible by members of the public, or a journalist could give a source a dynamically generated invitation key. This would also provide a secure channel for sources and journalists to communicate with each other.
Alerts for supply and demand of geolocated news would make it possible for journalists and the public to collaborate around video objects in real-time discussion, remix, and contextual federated narrative.

A virtual currency or scrip exchangeable across publishers using the platform could reward the public for contributing.

First Sources would bring global and national scale video-based transparency down to the state, provincial, city or even town or community level. The same kind of transparency enabled by international and national news organizations reporting on openly available original source video would be available to anyone at any level.