Finally, the print copy of Internet Evangelism in the 21st Century is available through Amazon.com. It is $22,99 plus shipping! Also, remember to check out the print copy of Mediastrategy for Christian Witness available at the same price.

A famous New Yorker cartoon from 1993 showed two dogs at a computer, with one saying to the other, “On the Internet, nobody knows you’re a dog.”

That may no longer be true.

A new analysis of online consumer data shows that large Web companies are learning more about people than ever from what they search for and do on the Internet, gathering clues about the tastes and preferences of a typical user several hundred times a month.

These companies use that information to predict what content and advertisements people most likely want to see. They can charge steep prices for carefully tailored ads because of their high response rates.

The analysis, conducted for The New York Times by the research firm comScore, provides what advertising executives say is the first broad estimate of the amount of consumer data that is transmitted to Internet companies.

Privacy advocates have previously sounded alarms about the practices of Internet companies and provided vague estimates about the volume of data they collect, but they did not give comprehensive figures.

The Web companies are, in effect, taking the trail of crumbs people leave behind as they move around the Internet, and then analyzing them to anticipate people’s next steps. So anybody who searches for information on such disparate topics as iron supplements, airlines, hotels and soft drinks may see ads for those products and services later on.

Consumers have not complained to any great extent about data collection online. But privacy experts say that is because the collection is invisible to them. Unlike Facebook’s Beacon program, which stirred controversy last year when it broadcast its members’ purchases to their online friends, most companies do not flash a notice on the screen when they collect data about visitors to their sites.

“When you start to get into the details, it’s scarier than you might suspect,” said Marc Rotenberg, executive director of the Electronic Privacy Information Center, a privacy rights group. “We’re recording preferences, hopes, worries and fears.”

But executives from the largest Web companies say that privacy fears are misplaced, and that they have policies in place to protect consumers’ names and other personal information from advertisers. Moreover, they say, the data is a boon to consumers, because it makes the ads they see more relevant.

These companies often connect consumer data to unique codes identifying their computers, rather than their names.

“What is targeting in the long term?” said Michael Galgon, Microsoft’s chief advertising strategist. “You’re getting content about things and messaging about things that are spot-on to who you are.”

The rich troves of data at the fingertips of the biggest Internet companies are also creating a new kind of digital divide within the industry. Traditional media companies, which collect far less data about visitors to their sites, are increasingly at a disadvantage when they compete for ad dollars.

The major television networks and magazine and newspaper companies “aren’t even in the same league,” said Linda Abraham, an executive vice president at comScore. “They can’t really play in this sandbox.”

During the Internet’s short life, most people have used a yardstick from traditional media to measure success: audience size. Like magazines and newspapers, Web sites are most often ranked based on how many people visit them and how long they are there.

But on the Internet, advertisers are increasingly choosing where to place their ads based on how much sites know about Web surfers. ComScore’s analysis is a novel attempt to estimate how many times major Web companies can collect data about their users in a given month.

Web companies once could monitor the actions of consumers only on their own sites. But over the last couple of years, the Internet giants have spread their reach by acting as intermediaries that place ads on thousands of Web sites, and now can follow people’s activities on far more sites.

Large Web companies like Microsoft and Yahoo have also acquired a number of companies in the last year that have rich consumer data.

“So many of the deals are really about data,” said David Verklin, chief executive of Carat Americas, an ad agency in the Aegis Group that decides where to place ads for clients.

“Everyone feels that if we can get more data, we could put ads in front of people who are interested in them,” he said. “That’s the whole idea here: put dog food ads in front of people who have dogs.”

Web companies also can collect more data as people spend more time online. The number of searches that American Web users enter each month has nearly doubled since summer of 2006, to 14.6 billion searches in January, according to comScore.

ComScore analyzed 15 major media companies’ potential to collect online data in December. The analysis captured how many searches, display ads, videos and page views occurred on those sites and estimated the number of ads shown in their ad networks.

These actions represented “data transmission events” — times when consumer data was zapped back to the Web companies’ servers. Five large Web operations — Yahoo, Google, Microsoft, AOL and MySpace — record at least 336 billion transmission events in a month, not counting their ad networks.

The methodology was worked out with comScore and based on the advice of senior online advertising executives at two of the largest Internet companies.

“I think it’s a reasonable way to look at how many touch-points companies have with their consumers,” Jules Polonetsky, the chief privacy officer for AOL, said of the comScore findings on Friday.

But Mr. Polonetsky cautions that not all of the data at every company is used together. Much of it is stored separately.

The information transmitted might include the person’s ZIP code, a search for anything from vacation information to celebrity gossip, or a purchase of prescription drugs or other intimate items. Some types of data, like search queries, tends to be more valuable than others.

Yahoo came out with the most data collection points in a month on its own sites — about 110 billion collections, or 811 for the average user. In addition, Yahoo has about 1,700 other opportunities to collect data about the average person on partner sites like eBay, where Yahoo sells the ads.

MySpace, which is owned by the News Corporation, and AOL, a unit of Time Warner, were not far behind.

ComScore said it recorded the ad networks using different methods and that the exact ordering of these top companies might vary with a different methodology, but the overall picture would be similar.

Google also has scores of data collection events, but the company says it is unique in that it mostly uses only current information rather than past actions to select ads.

The depth of Yahoo’s database goes far in explaining why AOL is talking with Yahoo about a merger and Microsoft is willing to pay more than $41.2 billion to acquire the company.

Traditional media companies come in far behind.

Condé Nast magazine sites, for example, have only 34 data collection events for the average site visitor each month. The numbers for other traditional media companies, as generated by comScore, were 45 for The New York Times Company; 49 for another newspaper company, the McClatchy Corporation; and 64 for the Walt Disney Company.

Some companies are trying to close the gap. Walt Disney, for example, is studying how to combine data from its divisions like ESPN, Disney and ABC. The News Corporation is exploring ways to use information that MySpace members post on that site to select ads for those members when they visit other News Corporation sites.

IAC is using data from its LendingTree site to deliver ads on its other sites to people it knows are looking for mortgages.

Some advertising executives say media companies will have little choice but to outsource their ad sales to companies like Microsoft and Yahoo to benefit from their data. The Web companies may prove they can use their algorithms and consumer information to better select which ads for visitors better than media companies can.

“I think a lot of publishers are going to find they don’t have enough data,” said David W. Kenny, chief executive of Digitas, a digital advertising agency in the Publicis Groupe. “There’s only going to be a handful of big players who can manage the data.”

People who spend more time on the Internet, of course, will have more information transmitted about them. The comScore per-person figures are averages; occasional Web users have far less transmitted about them.

The comScore figures do not include the data that consumers offer voluntarily when registering for sites or e-mail services. When consumers do so, they often give sites permission to link some of their interests or searches to their user name.

The figures also do not account for information people enter on social network pages. MySpace, for example, collects billions of user actions each day in the form of blogs, comments and profile updates, said Peter Levinsohn, president of Fox Interactive Media, which owns MySpace.

Even with all the data Web companies have, they are finding ways to obtain more. The giant Internet portals have been buying ad-delivery companies like DoubleClick and Atlas, which have stockpiles of information. Atlas, for example, delivers 6 billion ads every day. The comScore figures do not capture such data.

Executives from Web companies said they had been working to inform consumers on their data practices.

These companies noted their consumer-protection policies. AOL, for example, lets users opt out of some ad targeting, Google lets users edit the search histories that are linked to their user names, Yahoo is working on a policy to obscure people’s computer identification addresses that are connected to search results, and Microsoft says it does not link any of its visitors’ behavior to their user names, even if those people are registered.

A study of California adults last year found that 85 percent thought sites should not be allowed to track their behavior around the Web to show them ads, according to the Samuelson Law, Technology & Public Policy Clinic at the University of California at Berkeley, which conducted the study.

Commentary: Understanding search engine optimization and the need to work quickly can help journalists write more effectively for the Web.

The best online newswriting differs from print newswriting; and journalism students can, and should, learn those differences. This week, I talked with my graduate online journalism class at the USC Annenberg School of Journalism about the differences in form and style between online and traditional print newswriting.Since so much news on the Web is simply repurposed from print newspaper and magazines, one might question the need to teach online as a distinct writing skill. But just because too many news organizations fail to take full advantage of the medium’s opportunities does not mean those opportunities do not exist. Online publishing offers at least four unique writing formats for journalism, and savvy online reporters ought to learn how to use write in ways that enmesh their work within the robust context of the World Wide Web. And… to do so swiftly, to remain viable in the far more competitive online publishing market. To that end, I wrapped up the class with a competitive exercise that took my students by surprise.

I don’t use textbooks in my classes (college students have enough expenses as it is), opting instead to direct students to readings available free online. To prepare for this week’s class, I asked my students to read Mindy McAdams’ guide to online newswriting, as well as OJR’s wiki on the topic. I would recommend both resources to any journalist eager to improve his/her online writing skills.

Outside the traditional print-derived story format, online journalists write in blogs, wikis and discussion forums. They are also often called upon to write heads, decks and short article summaries that fuel RSS feeds and e-mail newsletters.

Unique online writing formats

Blogs offers the closest comparison to print newswriting, particularly the column form. Not all blogs need be first-person opinion, indeed, the best blogs, like the best columns, are built upon strong original reporting. But a great blog offers a distinct voice that grabs the reader’s attention and draws them into the piece. Blog writers must draw upon their personal life experiences and sharp observation skills to put their reporting into a context that their readers will understand quickly and intuitively. And, oh yeah, it helps immensely if those bloggers can do this in minutes and several times a day.Wikis are the ultimate in online “writing by committee.” The natural comparison to the print world here lies with the copy desk, revising and clarifying the work of others. Of course, wikis should not derivative, and require writers who can blend fresh information seamlessly into the existing article. Of all formats of online writing, this might be the toughest to do well. I continue to suggest that newsrooms should make more aggressive use of wikis to bring new readers up to speed on news stories, and to draw more search engine traffic into their websites.

Discussion boards have stymied newspaper websites for years, but allowed solo web publishers to build immense audiences online. The best discussion leaders take the skills of a great interviewer and apply them to their online communities, writing with a style that acknowledges and builds upon previous comments, sustaining the momentum of threads and eliciting knowledgeable responses. Smart, personal and well-informed words help these writers make their readers feel that they cannot possibly spend even single day away from the conversation.

Feed writing takes the traditional print skill of headline writing into a new medium, where the primary goal is not communication within a defined number of spaces on a page, but writing heads and decks that elicit clicks, forwards and “Diggs” from as many readers as possible. A print headline strives to get people to keep reading the article underneath, but a feed headline faces a tougher challenge: to get the reader to click through to an article from an RSS feed or e-mail, or, better yet, to motivate the reader to forward that link to others via e-mail, instant message and/or social bookmarks.

What are some the specific skills that online writers can employ to distinguish their work in these formats? As my class suggested, traditional qualities of great newswriting still apply: active voice, clear construction and careful vocabulary. But what else?

Writing for search engines… and reaching your readers

I suggested that my students first focus on single task unknown to print journalists: search engine optimization. I am aware that the suggestion that journalists write to please algorithms at the Googleplex will infuriate some journalism pros. But when you write a piece to score highly in search engine result pages, you craft a piece that serves its readers, as well.

To place well in search engine results, an article must be sharply focused to the keywords that readers are likely to use in an effort to find the piece. To write such articles, I asked my students to put themselves in the position of their potential readers (never a bad idea for a writer!), then envision what one or two words and phrases a reader would use to search for their piece.This forces the writer to (a) figure out just what exactly their piece is about and (b) narrow that topic to one or two key ideas. It’s a great way to clarify before writing a piece. Then, I asked the students to make sure that they used their keyword or phrase in the headline and lead paragraph of their piece, then several more times in the remaining copy, to stay on focus and keep the piece from wandering.

If the story’s moving into another direction, then you need another article. Hyperlink them for context, as necessary.

And that’s the second stage in writing well for online. Hyperlinking is essential, both to make full use of the deep context and background available on the Web [see my commentary from last September], and to enmesh their work within the Web, increasing its chances to be moved up into the top pages for search engine results. In addition to keyword relevancy, an article needs inbound links from other websites to rank well in search engine results.

To increase an article’s chances for search engine (and therefore, readership) success, I told the class that they should one day learn how to configure their online publishing tool to ensure that each article appears under a single, distinct URL, so that their pieces get the full search engine benefit of all inbound links to it, without duplicate content penalties. But, I assured them, that’s a “how-to” topic for a later day.

The speed quiz

We wrapped up the class with an exercise that, I think, drives home the challenge for online journalists to write well, and quickly.I gave each student a section from a recent Los Angeles Times newspaper and told them to pick the three most important stories within that section, then to write a head and deck for each story, as they would if they were crafting an RSS feed or e-mail newsletter. They would be graded on news judgment, use of keywords, active voice and economy of language. Each head/deck combo would be worth 13 points, on a 50-point scale.

That made for 39 points. The other 11? Well, there are 11 students in my class, and the final points would be awarded on Borda Count scale. The first student to send their summaries to my e-mail in box gets 11 points, the next 10, on down to the last student to complete the assignment, who would get just one point.

Most students gasped when I told them this, but their competitiveness soon kicked in. I warned them: if you rush to get your piece out first, and make an error of fact or spelling, you’ll end up with fewer points than if you proof-read your piece and turned it in last. And from past years giving this same assignment, I’ve found that many of the faster writers getting lower grades than their slower classmates.

A “teaching moment,” indeed.

A Sacred Heart University Poll found significantly declining percentages of Americans saying they believe all or most of media news reporting. In the current national poll, just 19.6% of those surveyed could say they believe all or most news media reporting. This is down from 27.4% in 2003. Just under one-quarter, 23.9%, in 2007 said they believe little or none of reporting while 55.3% suggested they believe some media news reporting.

…The perception is growing among Americans that the news media attempts to influence public opinion – from 79.3% strongly or somewhat agreeing in 2003 to 87.6% in 2007.

And, 86.0% agreed (strongly or somewhat) that the news media attempts to influence public policies – up from 76.7% in 2003. The Centre Daily  has even more disturbing findings from this Sacred Heart poll, via Terp Mole:
“Americans are discerning, through a maze of information sources, the truth about our status in Iraq. They see more success than the media is reporting,” said Jerry C. Lindsley, director of the Sacred Heart University Polling Institute. He added, “They are especially disturbed that negative media reports damage U.S. troop morale.”

Nearly three-quarters of all Americans surveyed, 70.7%, indicated they strongly or somewhat agreed that negative media reporting damages troop morale.

SEE Sacred Heart University

COMMENT by Dan Henrich: It seems to me that we are fast approaching a news crisis in the US.  Main Stream Media journalists are seeing them demise in the very near future and are crossing that line between objectivity and opinion with opinion leading in the guise of “reporting” on the war in Iraq.

Spongecell and Facebook are teaming up. Spongecell allows marketers to further spread and track peer-to-peer communications within Facebook’s user community.Events from any Spongecell-powered calendar, blog or site can be added to a user’s Facebook profile. The calendar items on a profile will show up on the mini-feed, or can be sent as an invitation to people in your network.

Marc Guldimann, CEO of Spongecell said “Online communities are where consumers ‘live’ and connect, and it’s a natural environment for marketers to communicate with their audience more organically and efficiently. The extension of the Spongecell application to Facebook is an obvious next step as many brands are seeking to engage people online and on social networks. Our customers use our software as a way to drive people to take action – from attending an event to buying a ticket to purchasing merchandise.”

“Because our software is platform-agnostic and can be integrated with many different forms of media, we empower our customers to engage their audience anywhere including on their mobile devices, MS Outlook, blogs or any web-based venue.”

Spongecell provides real-time metrics for marketers and event promoters. With a Facebook application, marketers can track comments made about their event. By segmenting audiences, people are informed about relevant content that interests them and are more likely to spread the word about events.

“The world’s top universities have come late to the world of online education, but they’re arriving at last, creating an all-you-can eat online buffet of information.

And mostly, they are giving it away.

MIT’s initiative is the largest, but the trend is spreading. More than 100 universities worldwide, including Johns Hopkins, Tufts and Notre Dame, have joined MIT in a consortium of schools promoting their own open courseware. You no longer need a Princeton ID to hear the prominent guests who speak regularly on campus, just an Internet connection. This month, Yale announced it would make material from seven popular courses available online, with 30 more to follow”.

See CBS NEWS for more

Just after 10 am on June 7, 2007, Ryan Sit glanced at his Gmail inbox and saw the message he had been waiting nine months to receive. Sit, a 29-year-old software developer from San Diego, is the founder of Listpic, a site that used bots — automatic software-based agents — to pull images from craigslist for-sale listings and reorganize them into an easier-to-navigate, more attractive format. Instead of tediously clicking individual links to view photos, Listpic users could see them all collected onto a single page. The service was an instant success, and by early June it was pulling in more than 43,000 visitors a day and thousands of dollars a month in Google AdSense revenue.

Sit had long dared to hope that Listpic’s success might prompt craigslist to commend him, initiate a partnership, or even buy Listpic and bring him aboard. So when he saw the message from craigslist CEO Jim Buckmaster in his inbox, he thought that his dreams were about to be realized.
Then he read the subject line: “Cease and desist.”
Instead of praising Sit, Buckmaster’s email charged him with violating craigslist’s terms of use, claiming that Listpic crossed the line between homage and copyright infringement. The missive demanded he stop displaying craigslist content. It closed with a terse “Please let us know of your plans for complying.

“Sit didn’t have much of a chance to respond. Two hours after receiving the message, Sit went to Listpic and found that none of the images on his homepage were loading. When he clicked on one of the links that was supposed to lead to a specific listing, he was redirected to craigslist’s main page. Sit’s bots had been crippled. “They didn’t even talk to me about trying to work something out,” he says. “They just banned me.

“Distraught and perhaps a tad vengeful, Sit posted a message on his homepage asking Listpic fans to send protest emails to Buckmaster and craigslist founder Craig Newmark. But craigslist refused to budge. Buckmaster is unapologetic. He points to a couple of factors in craigslist’s decision: Listpic’s constant stream of data requests had slowed craigslist’s page-loading times to a crawl, and, more egregious, Listpic had run Google text ads alongside the content, an affront to craigslist’s pristine anti-advertising stance. “It sounds old-fashioned,” Buckmaster says, “but we don’t view postings by craigslist users as data to be exploited by third parties.” Within weeks, Listpic had fallen from its perch as one of the top 15,000 sites on the Web — the height of its popularity — to somewhere below 100,000th place, where it languishes still. Today, Listpic pulls data from a different listings site, called Oodle, which was itself banned from accessing craigslist data.

“My goal was to help craigslist by making the user experience better,” a despondent Sit says. “This just sucks.”

The Internet these days is supposed to be all about sharing. Thanks to a common commitment to open access and cooperation, the data mashups that have defined the Web2.0 phenomenon have exploded. Zillow pulls map information from several partners, including Navteq, GlobeXplorer, and Proxix, and combines it with real estate data from public records to estimate what a house is worth. Photosynth, a service that Microsoft is developing, merges pictures from Flickr and other sources into eye-popping 3-D models. A popular startup called Mint lets customers pull financial information from their bank accounts and reorganize it into an interface that puts Quicken to shame. And the tools to tap and manipulate all this data can be found at sites like Dapper and Kapow.

Giants like Yahoo and Google have thus far taken a mostly nonproprietary stance toward their data, typically letting outside developers access it in an attempt to curry favor with them and foster increased inbound Web traffic. Most of the largest Web companies position themselves as benign, bountiful data gardens, supplying the environment and raw materials to build inspired new products. After all, Google itself, that harbinger of the Web2.0 era, thrives on info that could be said to “belong” to others — the links, keywords, and metadata that reside on other Web sites and that Google harvests and repositions into search results.

But beneath all the kumbayas, there’s an awkward dance going on, an unregulated give-and-take of information for which the rules are still being worked out. And in many cases, some of the big guys that have been the source of that data are finding they can’t — or simply don’t want to — allow everyone to access their information, Web2.0 dogma be damned. The result: a generation of businesses that depend upon the continued good graces of a relatively small group of Internet powerhouses that philosophically agree information should be free — until suddenly it isn’t.

Scraping is such an unkind word. It refers to the act of automatically harvesting information from another site and using the results for sometimes nefarious activities. (Some scrapers, for instance, collect email addresses from public Web sites and sell them to spammers.) And so most Web 2.0 companies eschew the term, preferring words like importing to describe their own data-harvesting expeditions. But whatever you call it, it’s a pretty simple process. Scrapers write software robots using scripting languages like Perl, PHP, or Java. They direct the bots to go out (either from a Web server or a computer of their own) to the target site and, if necessary, log in. Then the bots copy and bring back the requested payload, be it images, lists of contact information, or a price catalog.

Technically, such activity violates most Web companies’ terms of use. Gmail forbids its members from using “any robot, spider, other automated device, or manual process to monitor or copy any content from the Service.” Microsoft echoes that in the terms of use for Windows Live, prohibiting “any automated process or service to access and/or use the service (such as a BOT, a spider, periodic caching of information stored by Microsoft, or meta-searching’).” The Facebook agreement directs developers not to “use automated scripts to collect information from or otherwise interact with the Service or the Site.

“But despite the fine print, many companies welcome scrapers. Bank of America, Fidelity Investments, and scores of other financial institutions let their customers use bots from Yodlee to gather their account histories and reassemble them on Web servers outside of their corporate firewalls. And eBay permits Google’s shopping service, Google Product Search, to scrape sales listings and display them on its own site. Sure, by allowing scraping, these companies are inviting a deluge of potentially cumbersome data requests. But they’re also getting more visibility and happier customers who find the scrapee’s information ever-more useful. That, it seems, is a worthwhile trade.

The mostly benign attitude toward scrapers also stems from an inconvenient truth: They can be tricky to stop. One way is to require all users to retype a series of distorted characters, those graphic forms called captchas, which bots are unable to read. But too many of these annoy — even alienate — customers. Another method, devised by Facebook to prevent wholesale copying of users’ emails, is to display addresses as image files rather than text. With a little more effort, a site can task a counterbot to identify browser sessions that have suspiciously high rates of data requests — most bots work at a pace that’s far too quick to be human — and shut off their access. But overuse of these measures can cost the data source, degrading the site’s usability or plunging it into bot warfare. If an outside scraper improves user experience and maybe even brings in a few new visitors, companies usually let the bots come and go unopposed.

Sometimes, though, a Web 2.0 upstart can improve the user experience too much for its own good. In February 2006, Ron Hornbaker created Alexaholic, a site that scraped data from Alexa, Amazon.com’s Web-traffic service, and presented it in what Hornbaker thought was a friendlier interface. Users agreed with him: Alexaholic’s traffic quickly shot up to 500,000 unique visitors a month. Then, in March 2007, Amazon began blocking browser and server requests from Alexaholic. (According to Amazon’s public statements, it blocked Alexaholic only after it had “explored an acquisition” and was rebuffed.) Hornbaker rerouted his traffic through other servers, circumventing the blockade. Then Amazon sent him a cease-and-desist letter, demanding he stop scraping Alexa’s data and profiting from its brand. Hornbaker changed his site’s name to Statsaholic but continued to scrape and remix Alexa stats. Finally, Amazon — seemingly tired of the cat-and-mouse game — served Hornbaker with a lawsuit charging that he was violating its trademarks. Hornbaker had little choice but to give up. Today, Statsaholic draws upon traffic statistics from a variety of other sources, like Quantcast and Compete. (Hornbaker and Amazon would not discuss the fracas, citing terms of their settlement. Ironically, Statsaholic is three times more popular than Hornbaker’s Alexaholic ever was.)

Such vulnerability to sudden data blackouts illustrates why some potential investors get nervous about funding scraping-dependent businesses. “Anybody who is a supplier to you has power over you,” says Allen Morgan, a venture capitalist at the Mayfield Fund who has invested in a raft of Web 2.0 companies, including Tagged, a teen social network and Slide, one of the most successful makers of Facebook applications. Morgan says that as those data providers help power more applications, they take on the role of operating systems — with a vested interest in consolidating their power. “Inevitably, they will feel compelled to compete with application developers in order to grow their business — and it’s an unfair fight.”

Investors aren’t the only ones wary of the unspoken agreements and one-sided relationships that characterize the scraping industry. Some large Web companies don’t relish the unregulated dispersal of their data and would love to find a way to monitor and control the information they dole out. That’s why many of them have begun encouraging developers to access their data through sets of application protocol interfaces, or APIs. If scraping is similar to raiding someone’s kitchen, using an API is like ordering food at a restaurant. Rather than create their own bots, developers use a piece of code provided by the data source. Then, all information requests are funneled through the API, which can tell who is tapping the data and can set parameters on how much of it can be accessed. The advantage for an outside developer is that with a formal relationship, a data source is less likely to suddenly turn off the taps.

The downside, from the remixers’ point of view, is that it gives data sources greater control over what information the remixers can access and how much of it they can harvest. With most APIs, a developer gets a unique key that lets the data supplier know when the developer is using the API. But it also lets the source block the key’s owner for any reason.

In February, Jeremy Stoppelman, the 30-year-old cofounder of the community-directory site Yelp, received a late-night phone call from one of his engineers informing him that the maps on Stoppelman’s site, compiled through a Google Maps API, were no longer working. It turns out that Yelp was generating more than the maximum number of data requests the API agreement allowed.

“It was scary,” Stoppelman says of the subsequent negotiation with Google. A few months earlier, Yelp had raised a $10 million round of funding. Paying for map data hadn’t been part of the business plan, and going into the meeting with Google, he says, “I didn’t know if we’d get priced out.” Eventually, Stoppelman cut a deal with Google to allow continued access to Google Maps for an undisclosed sum.

The promise — and the threat — of scraping is nowhere more evident than in the booming proto-industry of social networking. Social networks have thrived on scraping: Facebook, MySpace, and LinkedIn all encourage users to tap into their webmail address books as a way of inviting and connecting with their friends and coworkers. After prompting users to submit their login information, the sites unleash bots that scrape the webmail companies’ servers, pulling out friends’ addresses, checking them against the network’s roster, and letting users invite contacts who aren’t already signed up. The tactic has fueled an explosion in each site’s membership; Facebook’s stands at 54million and is growing by more than a million new users every week.

But recently, as the competition between social networks heats up, scraping has emerged as a high-stakes strategy. Microsoft announced a $240million investment in Facebook last fall, and within weeks, LinkedIn users found themselves suddenly unable to import their webmail contacts from Microsoft’s webmail services. Angus Logan, a Microsoft executive, says the restrictions are a matter of security and that the company is developing user-data APIs. “We do not advocate the practice of contacts scraping,” he says, “as we believe it poses unnecessary risks to consumers, whether it be for nefarious practices like phishing scams or more straightforward social networking activities.” But that philosophy is applied inconsistently. As of late November, Facebook members were still able to import their Microsoft webmail accounts through scraping.

In the end, says Reid Hoffman, the founding CEO of LinkedIn, it’s the users who lose out when Web companies decide to crack down on popular scrapers. After all, LinkedIn becomes much less useful if its members can’t quickly invite all of their friends; Yelp loses much of its appeal if it can’t display Google’s maps. “The question you hear,” Hoffman says, “is You’re doing all this scraping, and you’re increasing the load on our servers. What are we getting out of it?’” Hoffman’s answer: happy, connected users.

And in the process, the world is getting a better Internet, one where bright ideas become great services almost instantly and where information is easy to discover and use. Fundamentally, Hoffman adds, it’s not the place of companies like Yahoo, Microsoft, Facebook, or LinkedIn to decide who gets access to their users’ data. It should be up to the users themselves. “It’s simple,” he says. “The individual owns the data.” Even if it sits in some company’s server farm.

Source: WIRED

Outfoxed Media examines how media empires, led by Rupert Murdoch’s Fox News, have been running a “race to the bottom” in television news. This film provides an in-depth look at Fox News and the dangers of ever-enlarging corporations taking control of the public’s right to know.

LINK HERE

This video provides explains how the media fits into political and social change and specifically addresses the following questions:

* How is the Press supposed to Work?
* How and why does the Press act like an Echo Chamber?
* How is media changing?
* What does it mean that the “news is becoming a conversation”?
* Will these new media changes affect the nature of politics?
* How does The Echo Chamber Project fit into all of this?

This video explores the changes in the way we find, store, create, critique, and share information. This video was created as a conversation starter, and works especially well when brainstorming with people about the near future and the skills needed in order to harness, evaluate, and create information effectively. If you are interested in this topic, check out Clay Shirky’s work, especially: http://www.shirky.com/writings/ontolo…

Also check out David Weinberger’s Everything is Miscellaneous:
http://www.everythingismiscellaneous….