We’ve been paying so much attention to Google Search that our treatment of other search engines may not look evenhanded. Competition is good for everyone. At the moment, Bing, Google’s worthy competitor from the Microsoft stable, is lagging behind in market share, but please don’t tell anyone at Microsoft that Bing cannot compete with Google head to head.

Recently, when Satya Nadella took over the CEO of Microsoft some experienced SEO analysts did not forget to point out that Search veterans were now heading Google (Larry Page), Microsoft (Bing), as well as Yahoo (Marissa Meyer). It has not been lost on many that Nadella once headed the unit which oversaw Bing as well.

My money is on Microsoft to invest enough resources on the Bing team that they always remain a close competitor to Google in terms of the quality of the Search results, if not market share. Bing offers plenty of value-added free tools for the searcher, which are not limited to Keyword Research and Webmaster Tools.

Bing also powers Search for Yahoo!, and together with Yahoo!, offers the combined Yahoo!Bing network for PPC advertising on the web. Even the combination doesn’t come anywhere near Google on both organic Search as well as PPC Ads, but still they remain in play as a serious act in town.

DOES BING OFFER SUPERIOR SEARCH?
This brings us to the question: Are there any areas in which Bing offers a Search experience superior to that of Google?

In my limited experience, there are a few.

For instance, Bing offers conversion to bitcoin, though Google doesn’t.

Not sure how relevant it is, but given the popularity graph of bitcoin, I’m guessing Google may soon offer the service.

Versus

Anything else? Sure, I noticed that when you search for books, Bing offers a superior experience. For one, it’s possible in Bing to return a results page without the presence of Amazon.com.

I’ve not seen this happen in Google.

Also, I noticed that Bing always picks up the average reader rating of a book from Amazon.com and all other booksellers in the results page.

But I noticed that Google doesn’t pick up the average reader ratings from Amazon.com, though it may do so from other booksellers:

Are we done? Not yet. I was merely trying to highlight that there are some areas in which Bing Search results could be superior. Let’s continue our research some more in the next post.

e.o.m.

More nuggets from I’m Feeling Lucky to help us understand Sergey Brin better. Douglas’ experience with running the first set of banner ads for Google made him realize how much data driven the company’s founders were. Google had barter arrangements with Netscape and Go2Net. So the marketing department needed to come up with banner ads. “How many can you have by tomorrow?” Sergey asked. “Why don’t you start with a 100.” Douglas threw himself at the task, roping in a copy writer and an artist. All the banners were critiqued. The comments from the founders never seemed to end, until one fine day, Sergey again said: “We should try a bunch of these out. And see how they perform. We are wasting time by not running them and getting the ultimate feedback — clickthroughs.

Eventually the ads began to run, and Douglas says the pressure on him seemed to ease as he started the practise of plugging in data about which ads were attracting more CTRs into a spreadsheet, and hand-delivering them to Larry and Sergey. He would pull ads which were not working, and replace them with new banners. “I became a convert to the power of data persuasiveness,” Douglas writes.

Google is now justly famous for A/B testing its headlines and design. Its popular Google Analytics tool, offered free to website owners, even features a fairly extensive A/B testing function.

HOW A/B TESTING BEGAN AT GOOGLE
Wired magazine ran an article on the origins of A/B testing at Google some time ago. The rotation of banner ads narrated above by Douglas Edwards is of course not included in the history of the evolution of A/B testing at the search giant. This is what the Wired article said about it:
<<
At Google — whose rise as a Silicon Valley powerhouse has done more than anything else to spread the A/B gospel over the past decade — engineers ran their first A/B test on February 27, 2000. They had often wondered whether the number of results the search engine displayed per page, which then (as now) defaulted to 10, was optimal for users. So they ran an experiment. To 0.1 percent of the search engine’s traffic, they presented 20 results per page; another 0.1 percent saw 25 results, and another, 30.

Due to a technical glitch, the experiment was a disaster. The pages viewed by the experimental groups loaded significantly slower than the control groups, causing the relevant metrics to tank. But that in itself yielded a critical insight — tenths of a second could make or break user satisfaction in a precisely quantifiable way. Soon Google tweaked its response times and allowed real A/B testing to blossom. In 2011 the company ran more than 7,000 A/B tests on its search algorithm. Amazon.com, Netflix, and eBay are also A/B addicts, constantly testing potential site changes on live (and unsuspecting) users.
>>
‘DON’T INVOLVE EVERYONE; IT’S A WASTE OF TIME’
The UI team was debating names and designs for the browser button, a piece of code that allowed users to add Google search links to their browsers, writes Douglas. The mailing list had swelled to 10. Sergey found it ridiculous. Instead of exhaustive user testing and internal deliberations, Sergey said Google should just put up the service and test it when they get a chance. “Don’t forget. We can change it at the drop of a hat,” he said. ‘Launch first, iterate later’ was Sergey’s perspective. Then he expounded his philosophy. “I don’t think we should have any meetings about a project like this,” he said, “or any group emails except the one to users announcing the launch. Having everyone involved in every issue is not a good use of anyone’s time.”

 (To be continued. Vignettes taken from the book I’m Feeling Lucky by Douglas Edwards)

We continue from where Douglas Edwards, the former director of consumer marketing and brand management at Google, left us in his book I’m Feeling Lucky. Remember, we are using these vignettes to understand the thought processes which drive Sergey Brin.

Let’s welcome George Salah into the narrative. He had left Oracle to become Google’s Facilities Manager after bonding with the founders at a roller hockey game. Oracle being a very successful company, Salah wanted to import best practices from there to the fledgling Google. Douglas quotes him as saying, “I said to Larry and Sergey, ‘I don’t want to recreate the wheel every time. Are you okay with me creating a set of standards?’ They looked at me like I was crazy.”

“Absolutely not!" the founders declared. "We don’t want to have anything to do with standards. We don’t want anything ‘standard’.”

“I think that was about the time I began to go bald,” George told me, says Douglas. Sergey and Larry wanted their company to be completely different. George threw out everything he had learned in his career, and set about finding vendors, contractors, and architects who understood what the Google founders wanted, and that was always function over form, writes Douglas.

The author notes that he and his colleagues in the marketing department too were forbidden to do things the “normal and accepted way”. He quotes one of his former colleagues saying, “Larry and Sergey hated the idea of template approaches to marketing. They refused to stick to manufactured messages, did not use presentations, and talked about what they wanted to talk about. The media loved them for it.”

A CURE FOR AIDS? NOT IMPOSSIBLE
Douglas continues. It seems once the founders came to know of marketing’s intent to dictate product plans to engineering. They threw a monkey wrench at it, he writes. Sergey eventually issued a company-wide manifesto listing Google’s top three priorities as “product excellence, user acquisition, and revenues”. It left a lot of questions unanswered, but Douglas writes that he eventually came to realize that the founders intended to “pick a path to the future based on their gut instinct”. An engineer by name Chad Lester marveled, “They had so much self-confidence that Sergey was convinced he personally could find a cure for AIDS.” Douglas noted that the VCs on the board, including storied names like John Doerr and Mike Moritz, didn’t have control and could do nothing beyond try and guide the founders.

NO BODY ODOR PLEASE
Larry and Sergey had appointed a VP for corporate marketing and put her in charge of PR and promotions, but not the development of products. The board wanted a different leader to build that organization —someone with technical savvy, but not an engineer. Douglas notes that Larry and Sergey reluctantly agreed to take a look around. The qualifications required? The right candidate would have to communicate with coders, execute quickly, and be very, very smart. And yes, smell nice too! Douglas writes that Sergey once rejected an applicant in part because “I thought he had kind of a bad body odor.”

 (To be continued. Vignettes taken from the book I’m Feeling Lucky by Douglas Edwards)

When Douglas Edwards, the former director of consumer marketing and brand management, landed at Google for a job interview, among the questions that Sergey Brin peppered him with, one stood out: “What’s your GPA?”

In his book, I’m Feeling Lucky, Douglas says that even after he was hired, the HR department kept pestering him for his transcripts and SAT scores. Indeed, having high academic records was once a pre-requisite for a Google hiring, a factor perhaps influenced by the views of the founders themselves, who were after all brilliant students.

As we have seen in a recent blog post, Google’s reliance on academic accomplishments seems to be a thing of the past.

One of the curve balls which Brin threw at Douglas during the hiring interview was when he offered him five minutes to explain something complicated that Brin didn’t already know. Douglas chose general theory of marketing. By the time Sergey came back, he had prepared stuff to answer 10 minutes of questions, but they were not of much use as he was forced to find answers on the spot to the questions Brin fired at him one after another. Later he found out that this was routine for Brin. An hour wasted with a candidate wasn’t considered a total loss if Brin could take away some new insights on a topic he didn’t know.

NO DIRECTORS IN A FLAT ORG
Douglas had applied for the post Marketing Director. Another individual by name Shari Fujii had applied for the same post advertised on the Google website. When they joined both found that Google didn’t seem to have such a post, and instead, the company preferred to address them as managers. “I shrugged my soldiers and swallowed my pride,” writes Douglas. Sergey reminded everyone constantly that titles aren’t important because Google wanted a flat organization with fewer levels and less of bureaucracy

NO GREAT CODERS
Apparently both Brin and Page were no great shakes as coders. Craig Silverstein, the first employee of Google, would say that he didn’t trust the founders as coders because he found a lot of bugs in their early code. He categorized them as research coders who were more interested in writing code that works than writing code that was maintainable. Jeff Dean, another engineer, told Douglas that the early Stanford version of Google had this quirk: when something unusual happened, it would print out an error message. It read, “Whoa, horsey!”

(To be continued. Vignettes taken from the book I’m Feeling Lucky by Douglas Edwards)

The time has come for Indians to seriously think about having a search engine of our own. Google, Bing, and Yahoo (powered by Bing) are very powerful search engines. But that doesn’t mean there’s no room for others. Homegrown Search Engines play a meaningful role in many big world markets, including Russia, Ukraine, Kazakhstan, Belarus, China, South Korea, and the Czech Republic.

Yandex has around 60% of the search market in Russia, 43% in Belarus and a third (around 33%) of the market in Ukraine and Kazakhstan. Baidu has around 56% of the Chinese search market and Naver leads the South Korean market with around 70% market share. There’s also a search engine with a sizeable market share in the Czech Republic, known as Seznam. All of them compete with Google in their respective markets, and are no pushovers. Baidu and Yandex are Nasdaq listed companies. So those interested in researching more about these companies can look up their SEC filings, some of which are quite elaborate.

Yandex apparently has an edge in the Russian and neighboring markets because of the Search Engine’s ability to recognize the Russian inflection in search queries. Baidu has an impressive pedigree, with co-founder Robin Li boasting of sound search credentials, besides being one of the richest individuals in China with a net worth of more than $12 billion.
LI, A SEARCH PIONEER
Most Western narratives give one a misleading picture that Baidu's success is because it kowtows to Beijing and allows its search results to be censored. But it's a fact that internet giants like Google in the US were more than willing to hand over treasure troves of user data to the US government when their spy agency came calling, often voluntarily giving more than what was asked for. It's difficult to believe that working for the interests of the US government is benign, and heeding requests from Beijing is evil.

And here's a surprising nugget: Baidu's co-founder Robin Li is one of the pioneers of search engine technology, and we cannot outright dismiss the possibility that Baidu is successful because it returns better search in Chinese.

Even when they touch upon Robin Li's background, Western news media skirt around the fact that he's an American educated tech wizard, who patented search technology ahead of Brin and Page. He was one among a handful who arrived early at the conclusion that inbound links from other Websites are a crucial pointer towards the quality of a Website. Just as Larry Page had devised the PageRank algorithm named after himself, Li —an alumnus of SUNY, Buffalo — had worked out his own system, known as RankDex.

In his highly regarded book on Google called In the Plex, legendary tech writer Steven Levy had this to say about Robin Li:

“One day in April 1996 he was at an academic conference. Bored by the presentation, he began to ponder how search engines could be improved. He realized that the Science Citation Index phenomenon could be applied to the Internet. The hypertext link could be regarded as a citation! ‘When I returned home, I started to write this down and realized it was revolutionary,’ he says. He devised a search approach that calculated relevance from both the frequency of links, and the content of anchor text. He called his system RankDex.” 

He didn’t sit quiet with this insight though, notes Levy. Li first asked his company Dow Jones to file a patent, and when that didn’t happen, he bought a self-help book on patent applications and filed it on his own in June 1996. “But when he told his boss, Dow Jones re-asserted itself, and hired a lawyer to review the patent, which it re-filed in February 1997,” Levy writes. Remember, this was two years before Stanford filed for the PageRank algorithm in 1998. And Li’s insight on the centrality of anchor text way back in 1996 is simply amazing, considering the entire industry of search engine optimization (SEO) draws its competitive advantage from it.

When Dow Jones failed to monetize the patent, Li quit to join an internet company by name Infoseek. Eventually, he left the United States and found his fortune in the Chinese market. So the Baidu founder is someone with serious Search Engine engineering chops, and we should not blindly accept Western speculation that Baidu got where it's now simply due to government patronage.

Similarly, Yandex has a research lab in California’s Bay Area. So these homegrown Search Engines have good pedigree, are financially sound, and have market leadership which they won’t be ceding any time to multinational competitors like Google.

GURUJI, INDIA’S LOST SEARCH ENGINE
This begs the question: What about India? A random search found that there apparently existed a Search Engine in India called Guruji.com. Without research, and without speaking to its founders, I won’t be in a position to comment on what went wrong, but apparently it shut shop sometime in 2012. I could capture this screenshot of how Guruji.com looked like just before it shut down, from the Wayback Machine. Google, Bing, Yahoo, and their upstart competitors like Blekko and DuckDuckGo are formidable in what they do, but it would be foolish to simply abandon the market to these companies, considering none of them are still very competitive in searching in Indian languages. The field is open for a Search Engine powerhouse in Hindi, or major Indian languages like Bengali, Tamil, and Telugu.

Search Engine technology is not rocket science, and if India can develop the knowhow to commercially launch satellites for others, providing state funding for any fledgling start-up in the domestic Search Engine space until it finds its footing would be a wise investment for the future. There’s no guarantee how multinational Search Engines would behave in times of conflict which pits Western interests against India's. Therefore, it would be prudent that the country makes this a priority.

The indictment this week in New York of Silk Road founder Ross Ulbrict for the second time is once again a reminder to the existence of the Dark Net or Invisible Web —that portion of the World Wide Web which has still not been indexed by the Search Engines of the World. Silk Road, for instance, could be accessed only through the Tor anonymizing software, and transactions were conducted using Bitcoin, the peer-to-peer digital currency.

Silk Road was a platform which allowed individual sellers of illegal drugs like heroine and LSD to find buyers in exchange for payment in Bitcoins. Its existence was not an aberration. Apparently up to 90% of the World Wide Web, and perhaps maybe more, remain invisible to search engines like Google and Bing. Not all of it is sinister though. Archives of academic journals which exist in walled gardens are a case in point. They are accessible to only those who have a gate pass.

There are communities all over the Web which are accessible only to members, and which do not allow indexation by search engines. Some are worried that the Dark Net will be used by terrorists and criminals to share secret messages and plot attacks. Hence, law enforcement authorities themselves are actively trawling the Dark Net with their own apps. In fact, the Tor software widely downloaded and used for hosting and accessing anonymous sites, was developed with funding by the US State Department!

HOW TO CODE MESSAGES WITHIN FLICKR
Others say that there’s no need for criminals and terrorists to go to the Dark Net to share secret messages when they can as well use popular platforms. For instance, it’s said that the digital codes of photographs uploaded in Flickr.com are used by some to pass on coded messages. Alternately, sequences of specific pictures with meanings can be posted on Flickr to convey a message. Apparently, coded messaging happens in full public view even through Twitter.

I guess the Dark Net would remain a reality so long as the regulators of the World Wide Web do not make search engine indexation a must for all online platforms. Such a regulatory stance may also have to confront the tendency of government agencies everywhere to be active on the Invisible Web. For instance, in his book Viral Loop, Adam L Penenberg speaks about an e-bay like auction market in Switzerland called WabiSabiLabi which sells blackmarket hacker code. And the biggest buyer in the blackmarket hacker code auction mart? Surprise, surprise, it is the US government! The Americans are apparently active there so as to stockpile software ammunition in anticipation of cyber warfare!!

So the combination of lax regulation, vested interests of government agencies, and the technological prowess of activists underground looks set to ensure the longevity of the Invisible Web.

So the answer to those who worry about the Dark Net could be that it would produce platforms which are capable of confronting and taming the toxic elements within.

e.o.m.

In theory, online search is expected to expose us to new experiences and vistas. There’s no limit to what we can explore and discover. But in practice, people universally restrict themselves to searching for things familiar to them.  People with peculiar preferences and tastes, who were previously shy about openly expressing them, tend to assemble around online water coolers which cater to their interests. Once in the company of the likeminded, the middle ground soon vanishes in the echo chamber where everyone reinforces the increasingly strident voices being raised.

PERSONALISATION MUDDIES THE GAME
The problem of echo chambers is worsened by the personalization route adopted by Search Engines.

Google and other Search Engines now return different search results for the same terms searched by different users. This is because these searches are customized and personalized for the user.  Many critics have raised the valid concern that this personalization is at the cost of creativity.

For instance, if you were to search in Amazon for a book, you will find it recommending other books which you might like. Very rarely would you be surprised by some original recommendation, since the suggested books are often based on data from your previous searches.

Providing a narrow and relevant information environment makes search useful, no doubt. But the absence of randomness can hurt in other ways. Creativity is often given a boost by the introduction of totally random ideas or stimuli. Think about the apple and Isaac Newton!

Eli Pariser argues in the book Filter Bubble that the search for perfect relevance and the kind of serendipity that promotes creativity, push in opposite directions. He says that ‘if you like this, you’ll like that’ can be a useful tool, but it’s not a source for creative ingenuity. Because, “by definition, ingenuity comes from the juxtaposition (placing side by side) of ideas that are far apart, and relevance comes from finding ideas that are similar”.

What is relevant may not always lead you to serendipity or happy chance finding which is foundational for creativity. So let your search lead you to explore new experiences occasionally. There’s the cost of time, no doubt. But you might strike gold once in a while.

THERE’S enough room for a vertical search engine for eBooks. The few out there are mostly amateur efforts by individuals. As of now, no big corporation has poured good money into developing one. And for a variety of reasons, including the litigation and uncertainties over Google’s book scanning project, the search engine giant has not gone all out into eBook search and incorporated it as a vertical, just like it has done for flights and hotels, after acquiring or cutting deals with sites offering vertical searches. As of now, Google Books is primarily a source for copyright-free eBooks, and Google Play books are restricted to the Android and iOS platforms.

But that could change in the foreseeable future. In the meanwhile, Amazon’s Kindle Store remains the most powerful eBook store for the simple reason that it stocks almost every eBook available. Not to forget its global reach. Though there are more eBook stores on the industry standard ePub platform, by its sheer heft in the market, Amazon with its Kindle eBooks packs a punch.
All ePub bookstores are search constrained
Barnes and Noble’s eBook store which stocks ePub books with DRM comes next. Kobo, eBookmall, diesel-eBooks, and ebooks.com are the other biggie eBook stores out there. From my experience, all of them have limitations in search. Discovering even eBooks which are available are often a problem. The stores reveal them only after repeated searches, enough to deter all but the most determined. Diesel and eBooks.com have particularly bad in-house search engines.

But by vertical search engine for eBooks, we are primarily referring to search engines specialized in retrieving information about eBooks from all major eBook stores. Basically, here customers are looking out for price comparison, details about formats and devices on which they’re available, etc. There are a handful, and as mentioned before, most of them are amateur efforts.

This post will restrict itself to listing them out. A fuller review of their strengths and weaknesses has to wait for a later post.

These are the eBook search engines which I have come across:

• ·         Inkmesh  http://inkmesh.com
  
• ·         AddAll Ebooks  http://ebooks.addall.com
  
• ·         Luzme: http://luzme.com
  
• ·         Leatherbound: http://leatherbound.me/
  

In the next post, I will analyze their pluses and minuses.

THE past year was a tough one for SEO practitioners with Google drastically curtailing information on the keywords used to access websites made available through Google Analytics.
Keywords are still available for AdWords users through the Keyword Planner as well as for Webmasters through the Google Webmaster Tools, but the quality of the keywords returned are not comparable with those returned by Google Analytics. SEO practitioners will surely perfect workarounds to tide over the situation. But what's not discussed much is that the move could end up benefiting internet analytics companies like Hitwise, Comscore, Compete, Quantcast, etc.
The competitive analysis packages by these companies already cost an arm and a length. Now it looks like Google has indirectly made it more lucrative for them to charge for keywords too.
ISPs STILL PACK A PUNCH
Many people think that only search engines have access to the keywords used by searchers on the Web. But consumer connectivity to the Web is provided by the internet service providers (ISPs), and although they're now well past their heydays, the ISPs still pack a punch when it comes to accessing user search data.
Hitwise Global research head Bill Tancer has revealed how the process works through his book Click. He says the company collects search data on 10 million-plus users in the US alone. The company collects it from both ISPs as well as opt-in panels. The break-up is 7.5 million-plus from multiple ISPs across the country, and the remaining from opt-in panels. Opt-in panels are groups of internet users who have agreed to be monitored, and whose demographic details are made available to analytics companies, all for a price.
ISP and opt-in data about usage are updated every day. Search-term data is updated on a weekly basis. Tancer claims data privacy is protected by anonymizing and aggregating data, and by scrubbing search data terms off personally identifiable information. After Edward Snowden’s revelations, all such assurances have to be viewed skeptically.
So with internet analytics companies having access to search data in such a comprehensive manner, surely they have the wherewithal to step in and provide commercial solutions to any dearth of keyword data on account of Google's shift in policy?
Let's wait and see as to who will be the ultimate beneficiaries of Google's move. Will it be rival search engines like Bing? Or will it be these internet analytics companies?

(392 words)

More from Bill Tancer's Click. He found out that 'How To' queries represented nearly 3% of all searches in the United States. This was around 2007. The number might have increased by now. At that time it was the most commonly searched phrase in search engines.
Tancer says though 3% may not look like a large number at first blush, it is indeed shockingly high, considering the hundreds of millions of queries that search engines try to answer every day. Just as in searches incorporating New Year resolutions, here too he was able to identify some patterns. For instance, the 'How To' phrasal search coincided with the US school calendar: it peaked in summer, and ebbed during winter vacation and spring break!
In his research into popular 'How To' queries, Tancer found 'How to tie a tie' regularly topping the charts. But when he checked the data for other countries, he found a noticeable difference. In the UK, for instance, it was only the 51st popular 'How To' query. Perhaps that's because ties are part of the attire of schools in the UK, and tying a proper necktie becomes second nature? Tancer wonders.
Tancer continued with his substantial research into the question, and made an effort to group the 'How To' queries. He found that the most prevalent type were the general knowledge 'How To' questions (@57%), followed by queries of a sexual nature (eg. 'How to Kiss') @ 17%. Self-improvement 'How To' queries (eg. 'How to Lose Weight') @12% and searches for illegal activities (eg. 'How to Grow Weed') @ 9.5% brought up the rear.
Here's a list of the top 10 US 'How To' queries unearthed by Tancer:

1. How to tie a tie
   
2. How to have sex
   
3. How to kiss
   
4. How to lose weight
   
5. How to write a resume
   
6. How to levitate
   
7. How to draw
   
8. How to get pregnant
   
9. How to make out
   
10. How to make a video
    

Comments are welcome.