In this talk at UC Berkeley, Google's Sergey Brin confesses (at minute 1:27) that he thought Wikipedia couldn't work. Most people wouldn't contribute, he rightly assumed, and it would never reach critical mass.

He was in good company. In the classic "free rider" problem, you imagine an elementary school class with 20 students. If only two parents (10%) agree to volunteer to  help out as room parents and drive on field trips, the whole system breaks down: there aren't enough helpers and the two parents get angry at the others for not joining in. And that's exactly what Brin assumed would happen with Wikipedia.

But he was wrong, he says, because he--even he!--had underestimated the way scale can change the game. Sure, the experts say only 1% of Wikipedia's users actually contribute to making it better. Indeed, if you do the math, it's even worse than that: probably closer to 0.01% (today, Wikipedia has 75,000 active contributors out of 684 million visitors). But that 0.01% have created 10 million articles.

Most people don't contribute, just as Brin had feared, but it doesn't matter because the tiny fraction that do are a tiny fraction of an absolutely whopping number.

The lesson is that more is different. The Internet, by giving everybody access to a market of hundreds of millions of people, can work at participation rates that would be a disaster in the traditional world of non-zero marginal costs. YouTube works with just 0.1% of users uploading their own videos. Spammers can make a fortune with response rates of 0.00001%. (To give you some context, in my business of magazines, response rates of less than 2% on direct-mail subscription offers are considered a failure.)

This is the underlying logic of the Freemium business model, which uses the near-zero marginal cost of online distribution to reach the maximum possible audience, converting just a tiny fraction of them to paid users.

That's impossible for traditional products, which usually have non-zero marginal costs. You can't mail a brownie to everyone in the world on the hopes that a tiny fraction of them will come back for more. But on the Internet, it's not only possible, it's the smartest strategy.

That's why Freemium is so new (it was waiting for a zero-cost distribution method), and so counterintuitive to many people. Freemium doesn't work with the small numbers we're used to in daily life. Getting 5% of 100 people to pay for your software is no business, and in the traditional world it takes expensive marketing to reach more people than that.  But getting 5% of 100,000 people to pay for your software is a very nice business indeed, and online it costs virtually nothing to reach that many potential customers.

This is the point that everyone seems to miss: Free is not a business--it's zero-cost marketing for a business. And it works best at the largest scale: a small percentage of a big number is a big number.

Over the past few weeks there has been a flurry of reappraisals of the Long Tail, most of which center around the question of whether it creates bigger blockbusters or smaller ones (more concentrated markets or less concentrated ones). 

My predictions have always been that massive increase in variety plus massive improvements in "filters" (tools to make it easier to find new stuff that's right for you) would tend to reduce the blockbuster effect and redistribute attention over a wider range. And, indeed, that's what the data I cited in my book showed, where online markets of books, DVDs and music saw between 20% and 40% of the demand shift to products not available in traditional bricks and mortar stores. 

But there were clearly exceptions to this. One of the main ones was the irony that there was a very short Head of Long Tail aggregators: Amazon, iTunes, Google and their kin dominate their markets to a blockbuster-like degree. 

I blamed this on a still-young market and assumed that even aggregators would fall victim to the flight from one-size-fits-all someday. But new research from McKinsey (free registration req'd) suggests that this sort of radical inequality is increasingly the norm as markets get more networked. 

The past decade has seen the rise of many “mega-institutions”—companies of unprecedented scale and scope—that have steadily pulled away from their smaller competitors. What has received less attention is the striking degree of inequality in the size and performance of even the mega-institutions themselves. Plotting the distribution of net income among the global top 150 corporations in 2005, for example, doesn’t yield a common bell curve, which would imply a relatively even spread of values around a mean. The result instead is a “power curve,” which, unlike normal distributions, implies that most companies are below average.

Here's an example: the increasing gap between the top two or three financial services companies and everyone else since 1994 (the recent collapse of many financial firms and the roll-ups that have led to three "megabanks" will only accentuate this):

And it's not just companies. The Long Tail--the powerlaw created by network effects--may be creating super-celebrity, too. Here's what Google CEO Eric Schmidt (pictured above) says about the Long Tail and blockbusters (I've transcribed his interview with McKinsey):

I would like to tell you that the Internet has made such a level playing field that the Long Tail is absolutely the place to be, that there’s so much differentiation, so much diversity, so many new voices. I’d love to tell you that that’s in fact how it really works. Unfortunately, it’s not. What really happens is that we follow what’s called a powerlaw. A powerlaw has the property that a small number of things are very highly concentrated and most things have relatively little volume. And virtually all of these new network markets follow what’s called Zipfs’s law, or a powerlaw. 

It means unfortunately for our political point of view, although the tail is very interesting and we enable it, the vast majority of the revenue remains in the head. And this a lesson that businesses have to learn. While you can have a long tail strategy, you better have a head, because that’s where all the revenue is. 

And in fact it’s probable that the Internet will lead to larger blockbusters, and more more concentration of brands, which doesn’t make sense to most people, because it’s a larger distribution medium and when you get everybody together they still want to have one superstar. It’s a bigger superstar. It’s no longer a US superstar, it’s a global superstar. Global celebrities, global scandals, global politicians. And just to be clear, it’s a 90/10 model. We love the long tail, but we make most of our money in the head, just because of the math of the powerlaw. But you need both. You need the head and the tail to make the model work.

Since Google is the canonical Long Tail company, I should probably have a good response to this. But the truth is that he's no doubt right. Powerlaws do imply wildly unequal distributions of money, power, celebrity and everything else. But this is nothing new. The rich get richer, fame snowballs, and so on. Vilfredo Pareto spotted this in 1906 in Italian wealth distribution, which is why a powerlaw is more commonly known as a "Pareto distribution."

So how to square this with my own prediction of more widely distributed markets? Three ways:

1. There's a big difference between a highly concentrated market with a small number of players and one with a huge number of players. If there are ten stores in the world, having one of them with 70% market share smacks of a monopoly. If there are ten million stores, it doesn't. Amazon may be by far the largest e-tailer, but there are hundreds of thousand of other, smaller specialists. Google may be the largest online advertising company, but there are again many thousands in its wake. The Tail is Long. As Schmidt notes, these countless specialists are essential to a well-functioning market, but it's true that few of them are getting rich at it.
2. The consequence of the above is that it's very hard to measure the entire market. Schmidt rightly notes that network effects on a global scale mean that Paris Hilton is more famous today, in terms of the number of people who have heard of her, than she would have been a half-century ago. But at the same time, there are millions of microcelebrities (of which I am one, I guess) who are also more more famous than they would have been a half-century ago, for the same reasons. What is the total market of celebrity, and what is Paris Hilton's share? We have no idea--it's practically impossible to measure, to say nothing of how it has changed over the years. But I would bet that the aggregate market for microcelebrity (think: Facebook) is gaining share on traditional celebrity.
3. Finally, it's not all about money. As I've said many times, both in the book and elsewhere, most of the rewards in the Tail are non-monetary: a larger audience for producers, and more choice for consumers. Sometimes those can lead to economic benefits, but often they can't. Today, a decade into the arrival of unlimited variety online, the Long Tail is still more of a cultural force than an economic one.

I'll end by conceding a point: It's hard to make money in the Tail. As Schmidt notes, it's also hard to make money if you don't have a Tail (to satisfy minority taste, which improves the consumer experience), but the revenues are disproportionately in the Head. Perhaps that will never change, but what will change is our definition of Head. Once that was choice counted in tens or hundreds of items. Now, especially in Google's world, it's counted in tens or hundreds of thousands.  Powerlaws may indeed create bigger fish, but the Long Tail is all about the bigger pond.

In my original Wired article on Free, I described Freemium as the opposite of the traditional free sample: instead of giving away 1% of your product to sell 99%, you give away 99% of your product to sell `1%. The reason this makes sense is that for digital products, where the marginal cost is close to zero, the 99% cost you little and allow you to reach a huge market. So the 1% you convert, is 1% of a big number.

But that was just a hypothetical percentage split, to make a point. In the real world, what's the right balance? The answer varies from market to market, but some of the best data is in the games world.

In online free-to-play games, companies aim to structure their costs so they can break even if as little as 5-10% of the users pay. Anything above that is profit. Which is why these numbers from Nabeel Hyatt are so impressive:

• Club Penguin: 25% monthly uniques pay, $5/mo per paying user 
• Habbo: 10% monthly players pay, $10.30/mo per paying user 
• Runescape: 16.6% monthly uniques pay, $5/mo per paying user 
• Puzzle Pirates: 22% monthly players pay, $7.95/mo per paying user

As the blog notes, that compares very well to the 2% of the casual downloadable game market that pays, or a 3-5% that a lot of "penny gap" free trial web startups get. Estimates for the number of free Flickr users that convert to paid Flickr Pro range from 5-10%. Ning says 3% of its 500,000 social network creators pay for the premium version. And shareware software programs often see less than 0.5% of users paying up.

But others companies are able to do much better. Intuit, for instance, offers basic TurboTax Online free for federal taxes, but charges you for the state version. Company officials tell me 70% of users opt to pay for that version. That's a special case--practically everyone has to pay both federal and state taxes--but it's evidence that some very high conversion rates are possible in the Freemium model.

For the typical Web 2.0 company planning to use Freemium as its revenue model, my advice would be to set 5% as break-even, but balance the mix of free vs. paid features with the hopes of actually converting 10%. More than that, and you may be offering too little in your free version and thus not maximizing the reach that's possible with free. Less than that, and the costs of the freeloaders start to get significant, making it difficult to make money.

I was chatting last night with the CEO of one of the biggest software-as-a-service companies about how he could release a version of his product with a freemium model. The options that seemed best for him were these four:

1) Time limited (30 days free, then pay. This is the Salesforce model)

• Upside: Easy to do, low risk of cannibalization
• Downside: Many potential customers will be unwilling to commit enough to give the software a real test, since they know that if they don't pay they'll get no benefit after 30 days.

2) Feature limited (basic version free, more sophisticated version paid. This is the WordPress model)

• Upside: Best way to maximize reach. When customers convert to paid, they're doing it for the right reason (they understand the value of what they're paying for) and are likely to be more loyal and less price sensitive.
• Downside: Need to create two versions of the product. If you put too many features in the free version, not enough people will convert. If you put too few, not enough will use it long enough to convert.

3) Seat limited (can be used by up to some number of people for free, but more than that is paid. This is the Intuit QuickBooks model)

• Upside: Easy to implement. Easy to understand
• Downside: Might cannibalize the low end of the market.

4) Customer type limited (small and young companies get it free, bigger and older companies pay. This is the model used by Microsoft's BizSpark, where companies less than 3 years old and under $1 million in revenues get Microsoft's business software free.)

• Upside: Charges companies according to their ability to pay. Get fast-growing companies early.
• Downside: Complicated and hard-to-police verification process.

He hasn't decided which one to choose, but I like 3 and 4 best. They allow you to reach the largest potential market with the most useful product, and then convert the ones that are likely to be the best, most committed customers.

Over in the UK, Will Page, who works for the copyright collection society there, presented some interesting data at a music conference last week that suggests that the Long Tail's usual powerlaw shape doesn't fit the sales they're seeing.

I can't find his presentation online [UPDATE: there's more information in a long interview with him here], so I have to go with the good coverage by Yankee Group analyst Benoit Felten and the comically angry coverage of The Register. The basics are that Page and the founders of Mblox, a mobile music provider, presented research that showed that a dataset of the sales of 13 million tracks showed a lognormal distribution, rather than the usual powerlaw, and as a result the total sales in the Tail were lower than the theory would predict.

Here are the two specific datapoints that came out in the coverage

• 80% of the revenue came from the top 52,000 songs (which is the inventory in a typical music store, my definition of "head"). In my own Rhapsody research, only 60% of the downloads were in top 52,000. So the tail was half as big in their dataset as it was in mine.
• Of the 13 million available tracks on iTunes UK, 10 million don't sell at all. They don't say over what period.

And here are my three thoughts on that:

1. Page did not reveal where the data came from. Felten suspects it was iTunes UK, but given that Page's partner in this research was Mblox, a mobile music provider, it seems more likely that it was a mobile carrier or their partner. If so, this is nothing new. I've already discussed why the Long Tail doesn't work for mobile music, due to the lack of good "filters"--easy search, recommendations (social and otherwise) and sampling methods--on such limited devices.
2. If it is iTunes UK, that would be a little bit more surprising, and I'd need to look at the data to see why it didn't follow the powerlaw shape that we saw in the data from Rhapsody (another digital music provider).  It could be that the pay-per-track model discourages risk-taking and exploration of new music, which is not an issue with Rhapsody, which uses an all-you-can-eat subscription model. It could say something about iTunes' less effective music recommendation tools. Or it might just mean that the UK iTunes consumer is not like the US consumer.
3. I've discussed many times that some markets that look like Long Tails (powerlaws) may in fact be lognormals, both in the book (Chapter 8 Notes) and in speeches and here on the blog.  The two distributions look similar at first glance, and you have to plot them log-log (or fit them with a statistical package) to tell the difference.  Long Tails are "heavy-tailed" distributions, where a lot of the total volume is the tail, while lognormals are more like the classic top-heavy hit distribution. The theory (and the Rhaspody) data puts music the powerlaw camp in a marketplace with vast choice and good filters, but it would not be a surprise to find that in markets with poor filters, music demand reverts to a lognormal.

Unfortunately, Page didn't send me the data or reveal its source, so we may not be able to answer these questions. He's a good economist, so I'm sure his analysis is excellent. But without knowing where the data came from, we really have no way of knowing whether he's discovered anything about music demand broadly, or has just been reminded once again that some music markets, such as mobile, don't work very well.

Search: LT is huge

Finally, just a reminder that the Long Tail effect is seen in many markets, not just music. One of the best examples is search, and Google is usually held up as the largest company based on monetizing the Long Tail. Over at the Hitwise site, Dustin Woodard has a new post with some data (sample below) that shows why this is.

In a study of 14 million search terms, he found the following:

• Top 100 terms: 5.7% of the all search traffic
• Top 500 terms: 8.9% of the all search traffic
• Top 1,000 terms: 10.6% of the all search traffic
• Top 10,000 terms: 18.5% of the all search traffic

This means if you had a monopoly over the top 1,000 search terms across all search engines (which is impossible), you’d still be missing out on 89.4% of all search traffic. There’s so much traffic in the tail it is hard to even comprehend. To illustrate, if search were represented by a tiny lizard with a one-inch head, the tail of that lizard would stretch for 221 miles.

The truth is my research is still greatly understating the true size of the tail because:
• The Hitwise sample contains 10 million U.S. Internet users and a complete data set would uncover much larger portions of the long tail.
• The data set I used filtered out adult searches.
• I only looked at 3-months worth of data (which were some of the slower months for search engines).

In summary, the long tail aspect of the search is true, but the data tells us that there may really be no head or body. When it comes to search, virtually all traffic is long tail and the word “long” doesn’t do the length of the tail justice.

That's what an article (sub required for full version) in this month's HBR asks. As usual for business school professors, the authors don't give as satisfying an answer as one might want (and take a long time to make some pretty obvious points), but there are some interesting nuggets in the piece.

It starts like this:

Customers who pay little or nothing and are subsidized by another set of customers are essential to a vast array of businesses, including shopping malls, real estate brokerages, information technology providers, auction houses, print and online media, and employment and dating services. According to one estimate, this business model accounts for a majority of the revenues of 60 of the world’s 100 largest companies.

With the explosion in the number of free services offered on the internet, the prevalence of so-called two-sided markets is likely to grow. The rationale for this approach, of course, is that by charging one set of customers little or nothing, the business will attract the critical mass of them required to draw in large numbers of another set of customers, and the income generated by the latter will handsomely exceed the cost of acquiring and serving the former. The high-stakes challenge is figuring out the true value of each “free” customer. Although executives know that free customers matter, they tend to underestimate their significance for two reasons: First, managers naturally focus more on customers who generate the bulk of revenues, and second, they lack a rigorous method for calculating the lifetime value of free customers.

The authors base most of their analysis on one case study, a research project that they did for an undisclosed online auctions company that they call "auctions.com". Sellers paid, but buyers could use the service for free. The question was what these free buyers were worth.

The answer is this: more at the company's start than after it was a few years old. The early adopters, enticed by the free service, were more important in bringing in other buyers and sellers than later adopters. This is basic network effects, but they at least quantify it with this chart:

You can see an earlier paper from these authors that gives the models behind this here.

Last year I mentioned that after my book came out, Anheuser-Busch created a division, called Long Tail Libations, to sell niche brews. While digging around for imagery for a presentation, I happened to stumble on its website and logo. Not bad!

[As I race to finish the book by Monday (!), here's a random but fascinating thing that popped in my alternative universe of aerial robotics. It's a post from a brilliant but eccentric engineer who goes by the name of Jack Crossfire. What follows are his words and images (I haven't verified them) - ca] 

Everyone's seen the Nevada nuclear test site on Google Earth:

And everyone's seen the videos & photos of operation Desert Rock IV in 1951, where soldiers happily watched a 21 kiloton nuclear explosion from trenches 4 miles away & sitting positions 7 miles away:

But on Google Earth you can see remains of trenches 2 miles from an explosion. Google doesn't have any any photos or videos of this experiment, Desert Rock V, except for these haunting remains on satellite photos:

Soldiers were placed 1 & 2 miles from a 43 kiloton explosion. Needless to say, most of them are no longer with us, but through the magic of Google Earth, you can see the eery remains of their trenches:

My least favorite question is "what's the next big thing in technology?" for a lot of reasons, from the excessive importance put on "big things" (whatever that means) to the small matter that I still can't predict the future, despite many years of fruitless practice.

What I usually say is that the industry I'm watching most closely these days is energy, not IT, and that not only do I not expect a new "big thing" in IT anytime soon, but we're still just figuring out how to absorb the genuinely revolutionary advances of the 1990s. It's hard to look around at the Web landscape today, from cloud computing to social networking, and see anything that wasn't at least sketched out a decade ago.

With that in mind, I happened to be reading a 15-year-old interview in Wired between Kevin Kelly and George Gilder, and it struck me as profound today as it was then. (Recall that this is before the Web!). Gilder has always been something of a hero to me, and it was delight to be reminded why. Choice quotes:

Gilder anticipating "Free" and, while he's at it, YouTube:

In every industrial revolution, some key factor of production is drastically reduced in cost. Relative to the previous cost to achieve that function, the new factor is virtually free. Physical force in the industrial revolution became virtually free compared to its expense when it derived from animal muscle power and human muscle power. Suddenly you could do things you could not afford to do before. You could make a factory work 24 hours a day churning out products in a way that was just incomprehensible before the industrial era. It really did mean that physical force became virtually free in a sense. The whole economy had to reorganize itself to exploit this physical force. You had to "waste" the power of the steam engine and its derivatives in order to prevail, whether in war or in peace.

Over the last 30 years, we've seen transistors (or switching power) move from being expensive, crafted vacuum tubes to being virtually free. So today, the prime rule of thrift in business is "waste transistors." We "waste" them to correct our spelling, to play solitaire, to do anything. As a matter of fact, you've got to waste transistors in order to succeed in business these days.

My thesis is that bandwidth is going to be virtually free in the next era in the same way that transistors are in this era. It doesn't mean there won't be expensive technologies associated with the exploitation of bandwidth - just as there are expensive computers employing transistors; but it does mean that people will have to use this bandwidth, they'll have to waste bandwidth rather than economize on bandwidth. The wasters of bandwidth will win rather than the people who are developing exquisite new compression tools and all these other devices designed to exploit some limited bandwidth.

Kelly asks "What is the fabric of the network?"

Photons. Electronics are not good for communications. Photons - optical computing - are. What makes photons so great for communication is they don't interfere with each other. They collide and pass on unaffected. You can send them two-way, and they are not subject to electromagnetic disruption. Many signals can flow through one fiber. But the fact that photons don't affect each other means they are cumbersome for computing, since you want interactions in computing. You need to have the charges affect one another - that's the heart of computing. The heart of the transistor function is that you can control a bigger force with a smaller force. But photons don't control each other. So for computing functions I still think that electronics will prevail; but for communications, photonics will prevail.

Just flew to Barcelona (next stop Lisbon, then PopTech in Maine) and on the plane I had a chance to read Larry Lessig's terrific new book, Remix: Making Art and Commerce Thrive in the Hybrid Economy. I think of it as something of the closer to his Creative Commons trilogy, which started with Free Culture and the The Future of Ideas.  As such, it's a capper to his decade of thinking and experimenting with ways to liberate human creativity from the dead hand of one-size-fits-all intellectual property law. It's an important book, just like his famously entertaining speeches, and a quick read that can be finished in an evening.

Here are some of my favorite quotes from Remix:

On the two economies, "commercial" and "sharing", and why money changes everything:

It would be very odd if a friend apologized for missing lunch and offered you $50 to make it up. And it would very, very odd if your girlfriend, at the end of a great date, offered you $500 to spend the night. Or if Wal-Mart asked all customers to 'pitch in and help Wal-Mart by sweeping at least one aisle each time you shop'. Or if McDonalds asked you to 'help out' by promising to buy hamburgers at least once a month. Money in the sharing economy is not just inappropriate; it is poisonous. And 'helping out' is not just rare in the commercial economy. It is downright weird."

On the natural coexistence of monetary and non-monetary economies:

No one is called a communist because he plays in a Thursday-evening softball league (competing with professional baseball) or helps clean up the local church (competing with the janitor of the church).

On how free things can sometimes be seen as superior:

For all of Wikipedia's flaws, the one thing people don't say is, 'Well, I don't trust Wikipedia because it's basically all advertising fluff.' Forgoing ads is a way to buy credibility, just as a judge forgoing bribes is a way to buy credibility.