In the first part of this post, we saw how Google engineer Paul Buchheit’s 20% side project which led to the creation of Gmail had its origins in his itch to fix the buggy Web emails available in the market.

He wanted to add a then unheard of 1 gigabyte of storage space so that users would never have to spend hours to sort and delete their mails. Ryan Tate’s book The 20% Doctrine says this was roughly five hundred times the storage space offered by competitors Hotmail.com and Yahoo! Mail.

The question then was how to finance the expenses for this free extra storage space. Tate says Buchheit’s manager Marissa Mayer wanted him to charge users for the extra storage. But instead, Buchheit started looking at contextual advertisements like in the case of AdWords. AdWorks shows web searchers in Google, advertisements based on their search terms on the right hand side as well as on top of the search results page. For instance, if someone searched for ‘hotel’, advertisements for hotels would turn up.

Buchheit wondered if the same logic could be extended to email. What if ads were shown on the side of emails based on the contents of the mail, he thought. It was brilliant on the face of it, but equally sounded creepy, says Tate. Mayer expressed her misgivings bluntly. “People are going to think there are people here reading their emails and picking out the ads, and it’s going to be terrible,” she recalled thinking in a Stanford University podcast done later.

The podcast also recounted how Buchheit actually broke his promise to Mayer on not to work on combining advertisements with email. “I remember leaving, and when I walked out the door I stopped for a minute, and I remember I leaned back and I said, ‘So Paul, we agreed we are not exploring the whole ad thing now, right?’ And he was like, ‘Yup, right’.”

Tate says Buchheit broke his word almost immediately. “Over the next few hours, he hacked together a prototype of the ‘ad thing’, a system that would read your email and automatically find a related ad to display next to it.”

HE USED A PORN FILTER TO CREATE ADSENSE

Tate also gives the details about how Buchheit went about creating the AdSense building blocks. Just as he adapted the Usenet search experience to create Gmail, he started working with another tool, a porn filter no less, to create AdSense. This was basically a code editor he had created to screen for adult content. Probably it was used to switch on and switch off Safe Search filters in Google Search Settings. “Normally, the filter examined a batch of known porn pages and listed words that occurred disproportionately within those pages. Other pages containing those words were then assumed to be porn. Buchheit instead turned the filter on Gmail messages, using the resulting keywords to select advertisements from Google’s AdWords database.”

Tate advises youngsters who are following side projects to copy Buchheit’s method of adapting old work. “As tempting as it is start from a clean slate, always look for opportunities to use something old to create something fresh,” Tate advises.

HOW BUCHHEIT WON OVER THE DECISION MAKERS

Although Buchheit directly rebelled against his boss in putting together the delivery mechanism of what turned out to be AdSense, Tate says it helped that Google had a culture where results prevail over preconceptions.

When the next day Marissa Mayer opened her Gmail account, only to see ads running on the side of mails, her immediate instinct was to summon Buchheit for an explanation. But she delayed action, thinking he deserved the mercy of sleeping for a few more hours after having worked the whole night. Tate writes, “While she waited, Mayer checked her Gmail. There was an email from a friend who invited her to go hiking — and next to it, an ad for hiking boots. Another email was about Al Gore coming to speak at Stanford University — and next to it was an ad for books about Al Gore. Just a couple of hours after the system had been invented, Mayer grudgingly admitted to herself, AdWords was already useful, entertaining, and relevant.”

Tate writes that like Mayer, Larry Page and Sergei Brin loved AdSense. “In short order, the Google high command decided AdSense would be a top priority. It was a no-brainer: Google’s main revenue source, AdWords, placed contextual ads alongside search results. But search results were just 95% of Web views; AdSense promised to open up the other 95 percent to ads, since it could go inside any Web page,” Tate writes.

According to Tate, it took just six months for AdSense to launch. In June 2003, it was made available to the public as a widget that any publisher could attach to any Web page. It generates more than $10 billion per year for Google. Gmail itself, for which AdSense was first developed by Buchheit, launched to the public on April 1, 2004, in what was initially thought of as an April Fool’s Day practical joke. Today it’s probably the world’s largest free Webmail service, as well as the pivot around which the Google Apps for Business suite functions.

So what are the lessons which we can take from Buchheit’s innovations in the development of Gmail and AdSense for people who run 20% projects:

• First of all your product should solve an existing itch.
• Then you should have the smartness to secure the backing of powerful decision-makers. So sponsorship is important, especially if you want the company to commit its best resources to what may well turn out to be a long project.
• Also, be ready to back your ideas with data. Buchheit was ready show through his prototypes that his email was actually an improvement over existing ones in the market and also that his advertisement system worked.
• Be ready to adapt old work. Buchheit adapted his learnings from fixing search in Google Groups to create Gmail. He used his code editor to filter porn sites to filter email text to select relevant ads.
• Leverage access to system administrators and computer systems. Buchheit had special access to the databases powering Google AdWords. Without dipping into them, he could never have got the ads to run alongside his Gmail texts and develop AdSense.
• Try snatching resources under the cover of a more established project. Buchheit was supposedly working on the high profile email project. But to make it stand on its feet and fund it, he developed AdSense on the side.
• Finally, you can get away with a little bit of defiance of authority so long as there are results to show for it. Buchheit gave his then boss Marissa Mayer an explicit promise that he would not work on developing AdSense. But he went ahead and did it anyway. But she herself became a backer of AdSense when she realized that it worked.

e.o.m.

We’ve been paying so much attention to Google Search that our treatment of other search engines may not look evenhanded. Competition is good for everyone. At the moment, Bing, Google’s worthy competitor from the Microsoft stable, is lagging behind in market share, but please don’t tell anyone at Microsoft that Bing cannot compete with Google head to head.

Recently, when Satya Nadella took over the CEO of Microsoft some experienced SEO analysts did not forget to point out that Search veterans were now heading Google (Larry Page), Microsoft (Bing), as well as Yahoo (Marissa Meyer). It has not been lost on many that Nadella once headed the unit which oversaw Bing as well.

My money is on Microsoft to invest enough resources on the Bing team that they always remain a close competitor to Google in terms of the quality of the Search results, if not market share. Bing offers plenty of value-added free tools for the searcher, which are not limited to Keyword Research and Webmaster Tools.

Bing also powers Search for Yahoo!, and together with Yahoo!, offers the combined Yahoo!Bing network for PPC advertising on the web. Even the combination doesn’t come anywhere near Google on both organic Search as well as PPC Ads, but still they remain in play as a serious act in town.

DOES BING OFFER SUPERIOR SEARCH?
This brings us to the question: Are there any areas in which Bing offers a Search experience superior to that of Google?

In my limited experience, there are a few.

For instance, Bing offers conversion to bitcoin, though Google doesn’t.

Not sure how relevant it is, but given the popularity graph of bitcoin, I’m guessing Google may soon offer the service.

Versus

Anything else? Sure, I noticed that when you search for books, Bing offers a superior experience. For one, it’s possible in Bing to return a results page without the presence of Amazon.com.

I’ve not seen this happen in Google.

Also, I noticed that Bing always picks up the average reader rating of a book from Amazon.com and all other booksellers in the results page.

But I noticed that Google doesn’t pick up the average reader ratings from Amazon.com, though it may do so from other booksellers:

Are we done? Not yet. I was merely trying to highlight that there are some areas in which Bing Search results could be superior. Let’s continue our research some more in the next post.

e.o.m.

The time has come for Indians to seriously think about having a search engine of our own. Google, Bing, and Yahoo (powered by Bing) are very powerful search engines. But that doesn’t mean there’s no room for others. Homegrown Search Engines play a meaningful role in many big world markets, including Russia, Ukraine, Kazakhstan, Belarus, China, South Korea, and the Czech Republic.

Yandex has around 60% of the search market in Russia, 43% in Belarus and a third (around 33%) of the market in Ukraine and Kazakhstan. Baidu has around 56% of the Chinese search market and Naver leads the South Korean market with around 70% market share. There’s also a search engine with a sizeable market share in the Czech Republic, known as Seznam. All of them compete with Google in their respective markets, and are no pushovers. Baidu and Yandex are Nasdaq listed companies. So those interested in researching more about these companies can look up their SEC filings, some of which are quite elaborate.

Yandex apparently has an edge in the Russian and neighboring markets because of the Search Engine’s ability to recognize the Russian inflection in search queries. Baidu has an impressive pedigree, with co-founder Robin Li boasting of sound search credentials, besides being one of the richest individuals in China with a net worth of more than $12 billion.
LI, A SEARCH PIONEER
Most Western narratives give one a misleading picture that Baidu's success is because it kowtows to Beijing and allows its search results to be censored. But it's a fact that internet giants like Google in the US were more than willing to hand over treasure troves of user data to the US government when their spy agency came calling, often voluntarily giving more than what was asked for. It's difficult to believe that working for the interests of the US government is benign, and heeding requests from Beijing is evil.

And here's a surprising nugget: Baidu's co-founder Robin Li is one of the pioneers of search engine technology, and we cannot outright dismiss the possibility that Baidu is successful because it returns better search in Chinese.

Even when they touch upon Robin Li's background, Western news media skirt around the fact that he's an American educated tech wizard, who patented search technology ahead of Brin and Page. He was one among a handful who arrived early at the conclusion that inbound links from other Websites are a crucial pointer towards the quality of a Website. Just as Larry Page had devised the PageRank algorithm named after himself, Li —an alumnus of SUNY, Buffalo — had worked out his own system, known as RankDex.

In his highly regarded book on Google called In the Plex, legendary tech writer Steven Levy had this to say about Robin Li:

“One day in April 1996 he was at an academic conference. Bored by the presentation, he began to ponder how search engines could be improved. He realized that the Science Citation Index phenomenon could be applied to the Internet. The hypertext link could be regarded as a citation! ‘When I returned home, I started to write this down and realized it was revolutionary,’ he says. He devised a search approach that calculated relevance from both the frequency of links, and the content of anchor text. He called his system RankDex.” 

He didn’t sit quiet with this insight though, notes Levy. Li first asked his company Dow Jones to file a patent, and when that didn’t happen, he bought a self-help book on patent applications and filed it on his own in June 1996. “But when he told his boss, Dow Jones re-asserted itself, and hired a lawyer to review the patent, which it re-filed in February 1997,” Levy writes. Remember, this was two years before Stanford filed for the PageRank algorithm in 1998. And Li’s insight on the centrality of anchor text way back in 1996 is simply amazing, considering the entire industry of search engine optimization (SEO) draws its competitive advantage from it.

When Dow Jones failed to monetize the patent, Li quit to join an internet company by name Infoseek. Eventually, he left the United States and found his fortune in the Chinese market. So the Baidu founder is someone with serious Search Engine engineering chops, and we should not blindly accept Western speculation that Baidu got where it's now simply due to government patronage.

Similarly, Yandex has a research lab in California’s Bay Area. So these homegrown Search Engines have good pedigree, are financially sound, and have market leadership which they won’t be ceding any time to multinational competitors like Google.

GURUJI, INDIA’S LOST SEARCH ENGINE
This begs the question: What about India? A random search found that there apparently existed a Search Engine in India called Guruji.com. Without research, and without speaking to its founders, I won’t be in a position to comment on what went wrong, but apparently it shut shop sometime in 2012. I could capture this screenshot of how Guruji.com looked like just before it shut down, from the Wayback Machine. Google, Bing, Yahoo, and their upstart competitors like Blekko and DuckDuckGo are formidable in what they do, but it would be foolish to simply abandon the market to these companies, considering none of them are still very competitive in searching in Indian languages. The field is open for a Search Engine powerhouse in Hindi, or major Indian languages like Bengali, Tamil, and Telugu.

Search Engine technology is not rocket science, and if India can develop the knowhow to commercially launch satellites for others, providing state funding for any fledgling start-up in the domestic Search Engine space until it finds its footing would be a wise investment for the future. There’s no guarantee how multinational Search Engines would behave in times of conflict which pits Western interests against India's. Therefore, it would be prudent that the country makes this a priority.