Google is the dominant search engine in the United States. The distant second most-used search engine is Microsoft’s Bing, which continues to lag behind Google despite Microsoft’s efforts to promote it in Windows 11 and 10. I tend to use alternative search engines, and I discussed many of the options in a 2021 alternative search engine review. However, as I explained in that review and elsewhere, most alternative search engines use Bing’s index to return results, with a smaller number that rely on Google’s index. While the existence of (relatively) privacy-friendly alternatives to the Google/Bing duopoly is a good thing, that most of the alternatives rely upon Google and Bing for their results highlights that there is a distinct lack of genuine diversity in generalist search engine options. In this article, I will explain how the Google/Bing stranglehold on English-language search creates two points of failure for people running websites and other projects that rely upon search for discovery. Specifically, we will examine the story of one website that was briefly de-indexed by Bing, and as a result became invisible to users searching with DuckDuckGo.
As strange as it may be to say, ensuring that a website is indexed by Microsoft’s Bing is important to many potential visitors who are interested in online privacy…
Most Alternative Search Engines Use Bing or Google
In my 2021 alternative search engine survey, I examined seven alternative search engines in some detail. Most of these search engines do not rely primarily (or at all) on their own index. Of the seven, four (DuckDuckGo, Qwant, Swisscows, and Peekier, the latter of which I later reviewed in more detail) rely primarily on Bing for their results while one, Startpage, relies primarily on Google. Two of the five, Mojeek and Gigablast, use their own independent search indexes. Subsequent to the publication of my article, Brave Search entered the search market and states that it draws most of its results from its own independent index.
In 2022, I have profiled a number of small search engines (or wrappers) that appear in our analytics logs. Most of these search tools use Bing’s index (see e.g., Publisher’s Clearing House, Ghostery Glow, Becovi, AlohaFind, Oscobo, and Lilo. A couple, Gibiru (probably) and Lukol (see HN discussion) use Google.
There are alternative search engines with their own indexes see Mr. Rohan Kumar’s excellent guide, although most of the options beyond Mojeek, Gigablast, and Brave Search are either primarily non-English search engines (e.g., Baidu, Yandex, or Naver) or niche search engines, a few of which I list in our Blogroll. Google and Bing not only dominate the search engine market, they also dominate the alternative search engine market.
Google’s and Bing’s Dominance Creates Single Points of Failure
In a 2021 article, I wrote about a very interesting 2021 incident wherein Bing censored image results for certain search queries having to do with the Tiananmen Square protests and massacre. However, instead of focusing primarily on Bing itself, I looked at the downstream effects of Bing’s censorship. For example, I confirmed that Bing’s censorship filtered down to search engines that rely primarily on Bing’s index. While DuckDuckGo was far from the only affected search engine, that is the one I will focus on in this post, both because it is the most popular alternative privacy-focused search engine in the United States and also because I will be discussing another story related to the relationship between Bing and DuckDuckGo.
I assume that most people use DuckDuckGo because of its promises of privacy and because its search interface is less obtrusive than Google’s and Bing’s (note that I am not focusing on the merits or demerits of DuckDuckGo in this post beyond the effects of its being tethered to Bing). To the extent that some may use DuckDuckGo to escape the big tech search bubble, the Tienanmen Square censorship incident highlighted that DuckDuckGo’s value in that era is limited by the fact that its wagon is tightly hitched to Bing’s (this is true no matter how much DuckDuckGo sometimes tries to obfuscate the fact). While DuckDuckGo’s results are somewhat different because of its lack of profiling and DuckDuckGo does offer its own slate of features and tools, it is still very much affected by Microsoft’s upstream indexing and algorithm decisions.
The Effect of DuckDuckGo’s Bing Dependence For Webmasters
I came across a very interesting post on the Cheapskate’s Guide blog titled Microsoft De-Indexed My Website from Bing but says It Will Re-index It Soon (see archived version). The article was published on November 18, 2021.
As the title suggests, Cheapskate’s Guide was briefly de-indexed by Bing, meaning that it stopped appearing in Bing search results after having previously appeared. The author speculates about the reason for Bing’s de-indexing, but I am only concerned for purposes of the instant article with the fact of the de-indexing. The author (“Cheapskate”) noticed the de-indexing indirectly:
I came in a indirect way to suspect that Bing had de-indexed cheapskatesguide. Bing has never brought much search traffic to this website, so I would likely not have noticed a decline in search engine traffic from there. Duckduckgo, however, was just beginning to send a non-negligible amount of traffic, and that suddenly dropped to almost zero last week.
Cheapskate
The phenomenon here is almost identical to the Tienanmen Square censorship story that I covered last year. Bing temporarily removed Cheapskate’s Guide from its index, meaning that no searches in Bing would return Cheapskate’s Guide articles. Because DuckDuckGo uses Bing’s index for its general search results, searches in DuckDuckGo stopped returning Cheapskate’s Guide articles because those articles were no longer in the index that DuckDuckGo uses. Thus, even though DuckDuckGo itself almost certainly played no role in the adverse action taken against Cheapskate’s Guide, it stopped returning results for the site because of an upstream decision made by Bing.
While The New Leaf Journal has never been de-indexed by either Google or Bing, we can relate to the Cheapskate’s Guide’s situation in one aspect. Befitting its dominant global market position, Google is our largest search engine referrer by a significant margin and our largest referrer period. However, after Google, the search engine referrer order no longer perfectly follows search engine popularity. DuckDuckGo, not Bing, is our second largest search engine referrer. Moreover, while DuckDuckGo is significantly behind Google in raw referrals, it significantly outperforms Google for New Leaf Journal refers based on its market share compared to Google, whereas Bing, which comes in third for us, performs about as one would expect (give or take a few points depending on the week) relative to Google based on market share. While we receive enough traffic from Bing that I would notice if the traffic suddenly disappeared, the loss of DuckDuckGo as a referrer source would be more significant.
While DuckDuckGo is our biggest Bing dependency, it is not our only Bing dependency. Ecosia, another alternative search engine that relies on Bing, has been a decent source of New Leaf Journal referrals, coming in as high as fourth place in some months. We have also received semi-regular traffic from Yahoo, Qwant, Peekier, MetaGer, all of which rely on Bing. Thus, were Bing to ever de-index our site, we would not only lose our Bing traffic, but also traffic from DuckDuckGo and a smattering of smaller alternative search engines. For Google’s part, we also receive decent traffic from Startpage (on par with Ecosia and Yahoo), although Startpage would be low on my list of concerns in that scenario.
(Note: We do receive decent traffic from Yandex and, as of late, Brave, and occasional hits from Mojeek, but the number of visitors we receive from these non-Google/Bing indexes is still small compared to the more popular search engines.)
The Importance of Bing to Webmasters
Unless a website is based in China, being indexed by and in good standing with Google is a categorical necessity for maximizing search engine traffic (Google is the dominant search engine in nearly every country in the world). However, it is possible to perform decently with Google while only receiving negligible traffic from Bing. But as Cheapskate’s Guide and The New Leaf Journal show, certain websites perform better with DuckDuckGo than Bing, and I would not be surprised if there are European websites that could report similar outcomes with Qwant, Ecosia, or MetaGer, or Asian websites with similar accounts regarding Yahoo. Thus, regardless of whether Bing itself is a priority, being in good standing with Bing is a necessity in order to ensure that people who use the most popular privacy-focused alternative search engines will find the project.
Before continuing, I will speculate (without decisive evidence) that the Bing issue is especially pertinent to websites that write regularly about issues related to tech and tech privacy, as well as websites that publish content of interests to audiences that are otherwise skeptical of Google (note, for example, that many DuckDuckGo users are likely unaware of the primary source of DuckDuckGo’s search results). The New Leaf Journal publishes articles of interest to these audiences, but we also publish a large amount general writing and blogging content that is unlikely to be of disproportionate interest to the kinds of people who are interested in web privacy. Because DuckDuckGo does not offer webmaster statistics, I am not entirely sure what articles are most popular with DuckDuckGo visitors, although I can deduce some based on our Google Search Console statistics (note: Google Search Console offers Google Search statistics, we do not use Google Analytics at The New Leaf Journal).
Conclusions: Fixing the Issues
In my Tienanmen Square censorship article, I argued that the incident highlighted the importance of there being more diversity in the search engine market. Cheapeskate Guide’s Bing horror story highlights another reason why this is the case. At the present, too many alternative search engines are wholly dependent on Bing. The effect of this is that even if a website receives non-insignificant non-Google traffic, the vast majority of that traffic still depends on a single point: being indexed by Bing. Being de-indexed by Bing means being effectively de-indexed by most of the universe of private search engines, and losing access to many potential visitors.
I preface my foregoing analysis by noting that I am not at all opposed to alternative search engines that use Google’s and Bing’s indexes. I personally have both Startpage (Google wrapper) and DuckDuckGo (Bing wrapper), and Peekier (Bing wrapper) in my search engine rotation, although I have been shifting to relying more on independent search engines, site-specific searches, and following links in articles in my RSS feed. These alternative search engines are valuable in that they offer searchers comparatively privacy-friendly ways to find mainstream search results from the biggest search indexes, and in certain cases some of these front-ends offer unique functionality or features. There is an important place for these kinds of meta search engines.
With that being said, the total dominance of Google’s and Bing’s indexes in the English-language search market is a problem, not only for webmasters or for the free-flow of information, but also for users and searchers who are not well-served in all respects by the current trends in search rankings by the big-two. Too much of the common internet experience depends on decisions made by Alphabet (Google’s parent) and Microsoft. I will conclude by discussing the current state of affairs and potential solutions from different perspectives.
Webmasters
As strange as it may be to say, ensuring that a site is in good standing with Microsoft’s Bing is very important for ensuring that a website is discoverable by privacy-minded users and people who are generally skeptical of Google or companies like Microsoft. Without delving into the merits of DuckDuckGo or any of the other alternative search engines that rely on Bing, these search engines are of interest to people who have an interest in privacy or who think that they provide important alternatives to Google. While it may be tempting to neglect Bing, doing so does a disservice to many potential visitors to a website who do not use Bing. (The principle here is similar to my argument that WINE and other tools that allow Linux users to run Windows games are good for gaming on Linux even if they create disincentives for building native-Linux games in the short-term.) For this reason, I also strongly advise websites that preclude indexing by everyone except Google to allow Bing as well as reputable alternative search engines that run their own web crawlers (e.g., Mojeek, Brave, Marginalia, InfoTiger, etc.).
It is also worth exploring other ways to share websites. Some small and niche search engines accept site submissions. For example, see the very interesting Infotiger project. Our Blogroll includes some other small web portals that accept submissions. While there is no substitute for the big indexes, adding different channels of content distribution (with a focus on those that are not subject to the arbitrary whims of big tech) can increase a site’s durability for dealing with unforeseen events such as being de-indexed by Bing.
Users
In previous articles, I specifically cautioned against using social media such as Facebook and Twitter as portals to the internet. I articulated the same concerns with respect to search, although I did not focus squarely on the Google/Bing index duopoly. A full exploration of increasing search horizons is beyond the scope of the instant article, but I recommend trying search engines with independent search indexes (Brave and Mojeek are emerging as interesting generalist options in this area, albeit Brave is much more usable in a practical sense). I also recommend trying smaller search engines with their own indexes. Marginalia is a particularly important independent search engine project that works to index the small web. However, I encourage people to look beyond search. Subscribe directly to interesting websites via RSS/ATOM feeds and develop a system for saving links from articles in those articles. Taking advantage of site-specific search, whether it is a site’s own search or a domain-specific search run through a search engine is also a great way to find content on sites that have useful information.