Search Engine NLJ Domain Search Test

I wrote an article three years ago about using DuckDuckGo to conduct domain-specific searches of Project Gutenberg. Project Gutenberg is a great case for using an external search engine to conduct domain-specific searches since an external search engine such as DuckDuckGo (granting DuckDuckGo is more of a Bing front-end than search engine) searches all text in Project Gutenberg’s books while Project Gutenberg’s own built-in search is not full text. However, I explained in the article that using an external generalist search tool to conduct site-specific searches works for any site that is indexed by the search engine in question (not a given as we learned here in 2023). I decided to try a fun experiment using external search engines to search The New Leaf Journal.

Survey Methodology

I needed a suitable search query for my experiment. Specifically, I wanted a query that someone with some knowledge of The New Leaf Journal might try and that has a significant number of articles that would serve as acceptable results. I decided to go with the query visual novel review. I have reviewed many visual novels at The New Leaf Journal, including 29 for a single project, so that a general query can produce a many appropriate responses. Moreover, I could see someone who is interested in visual novels stumbling upon one of my reviews and then trying a general query to see what else I have covered.

For this experiment, I will search The New Leaf Journal with the following search engines:

• Local: The New Leaf Journal Search
• Google & Derivatives: Google, Startpage, Ecosia (Google index)
• Bing & Derivatives: Bing, DuckDuckGo, Yahoo!, Ecosia (Bing index), Qwant
• Independent: Brave Search, Mojeek, Yep, Marginalia Search
• International: Yandex, Seznam, Cốc Cốc, Yahoo! Japan, Goo

I am grouping major search engines together with search engines that use their index. The survey is not all-inclusive. For example, there are many Bing derivatives floating around, but I limited our selection to what I consider to be the four most notable of the bunch. The “international” search engines are what I understand to be independent search engines primarily aimed at non-English speaking users.

Regarding Ecosia, I learned while doing this project that it now allows users to choose whether it uses Google’s index or Bing’s. Alternatively, users can let Ecosia decide which one to use for a particular query. I tested Ecosia separately using the Gpoogle and Bing options for this survey.

There were a couple of others that I looked at but could not include:

• Norton Safe Search: I wanted to include Norton Safe Search, which is based on Ask Search (which in turn is based on Google), solely because my review of Norton Safe Search has been popular for some unknown reason. However, Norton does not appear to support domain-specific search, so I had to leave it out of the survey.
• Stract: Stract is a very neat non-commercial free and customizable and open source engine that has been on my review to-do list for some time. I would have included it in the Independent category but for the fact that thenewleafjournal.com appears to have been blocked from appearing (perhaps the proper term is Bing’d)despite having been previously available (my smaller side-project, The Emu Café Social, as well as my weekly newsletter, are available). Thus, to my great disappointment, I cannot include Stract in our survey at this time.
• Alexandria Search: This is another open source search engine. It gets its results from the Common Crawl index. To the best of my knowledge thenewleafjournal.com has never been in its index and this remains the case, so it is not included. (Fun fact: I once shared Alexandria on Hacker News and it made page 1).
• Right Dao: A niche search engine with its own index – but our site is not well indexed by it.
• InfoTiger: InfoTiger is another independent search engine that is now used by MetaGer. Its availability is spotty and it was not working when I tried to test it (no link since the link often 404s, but InfoTiger is easy to find through regular search).
• greppr: This is a neat-looking small-web focused independent search engine which I discovered through a post on search engines with independent indexes. The New Leaf Journal seems well-indexed but the search engine does not yet have domain-specific search functionality, so I cannot test it with my query at the moment.
• Baidu: Baidu is the biggest search engine in China. I believe we are somewhat indexed because we occasionally get Baidu referrals, but my site-specific query attempts yielded no results.
• Naver: A South Korean search engine that I wrote an article about. No results were returned.
• Kagi: Kagi is a commercial search engine (very popular on Hacker News) which allows users to curate their results. I am not a Kagi customer, so I could not include it in the survey.

I collected all of the search results using fresh guest profiles on the Brave web browser on the morning of June 8, 2024. Your results may differ. Many factors, including whether you are logged into an account (or otherwise have stored cookies for the search engine in question), your location, or the time you try to search may lead to your receiving different results than what I have.

For the purpose of my survey, I decided to focus on the top three results from each search tool. Most people click on one of the first results (which is unfortunate for our search referral numbers). Moreover, the first three results give us a good idea of how the search engine understands the query and applies it to prioritize just three from the 50-or-so plausible articles it could return.

At the conclusion of the survey, I will give my subjective opinion – as the person who wrote all of the articles in question and understands their place in the larger scheme of The New Leaf Journal – on which search engines did the best job with this particular query.

Now let us look at the results.

(Note: You can click to expand all of the images in this article.)

Local Search

Before we use external search tools to look for visual novels, let us try our on-site search. The New Leaf Journal’s search is far from perfect, but it is full text (thanks to Relevanssi Light) and I put a bit of work into making it somewhat useful.

All three of its results are unique. First is my introduction article for the al|together visual novel review project and second are two of my three al|together visual novel ranking articles. I am sad to say that our on site search did not do a great job. While my al|together ranking articles are reviews of sorts, I would prefer that the query return specific visual novel reviews. The first place result – my introduction to the al|together project is not a review and is only somewhat responsive to the extent it links to 30 reviews within its text. The fourth-place result is awful since that is not a review at all. It takes until fifth and sixth place for actual reviews of specific visual novels to appear. Our on-site search is good for matching the words in a query to words in articles but not good at making sense of them.

Google & Friends

While I do not personally use Google (directly), it is by far the most-used search engine around the world (at least outside of China and Russia). Some may even say it is a monopoly. So it makes sense to start with the search engine that someone would most likely use to search The New Leaf Journal (if this ever happened at all, that is). Google returned the following three results:

Google is the only external search engine to have not returned a single one of my visual novel reviews (the same problem I noted with our on-site search). Instead, it led with my category archive for visual novel reviews followed by my sub-category archive for reviews from my Insani/al|together project and my full visual novel category. My category archive for visual novel reviews is a fair first choice since that would bring the reader to my full collection. I am not as happy with the second and third ranked options: The Insani/al|together category is redundant when the top result is the broader visual novel review category. In the context of a query for visual novel reviews, the full visual novel category is not a great result because while it does include all of the reviews, there are a number of visual novel articles that are not reviews (as we will see in the next two search tests). The highest ranked specific review is my review of The Boy I Like?, which I suspect is fourth because it is my most recent review.

Our next test is Startpage. This is a privacy-focused search that uses Google’s index. Despite relying on the same index as Google proper, it returned different results than Google did:

Google’s top result – the category archive for visual novel reviews, comes in second place on Startpage. First place is my article on comparing my al|together ranking to Visual Novel Database user scores for the same novels – which I think is an odd choice since it is not, itself, a visual novel review, but instead an article about visual novel reviews. In third we have an actual visual novel review, my May 2021 piece on The Poor Little Bird. I am surprised to see The Poor Little Bird review in a top-three. It is one of my earlier reviews, the novel itself is little-known, and I do not think it is among my most interesting review articles, despite coming with a full story analysis section (unusual for my reviews) – but Startpage likes what it likes.

Finally, I will look at Ecosia, something I did not know could fall into the “Google” category prior to undertaking this project. Ecosia used to be solely Bing-based, but it now allows searchers to choose whether to rely on Google’s or Bing’s index. (Ecosia’s unique feature is that it uses its revenue to plant trees.) I first tested it with Google’s index:

Ecosia charts its own course as a Google front-end. Like Startpage, we have a non-review in first, here being my piece on survive and advance visual novel choices. This, is a bad result, for this is an article about a certain type of visual novel choice structure, not a review. The second and third ranked results are reviews. First is my first visual novel review, which covered Bad End. Bad End is a niche horror visual novel and not among my best reviews, although it is unique in that I reviewed a now obsolete Amazon Store version on a 2013 Kindle Fire HDX instead of the better-known (that is relative here) version Steam. Third is my review of ACE Academy, an original English novel that is available on Steam and is a necessary prerequisite to our current best-performing review of Kaori After Story. ACE Academy is an unambiguously good result. I could not help but notice however that both Bad End and Ace Academy were linked to and briefly discussed in the survive and advance article.

Bing & Friends

At this time last year, we were still suffering from having been in Bing’s de-indexation doghouse (which, in turn, also left us on the outside looking in from every search engine that uses Bing). But we escaped and are now fully indexed by Bing, with no search troubles other than the ordinary generally downward referral fluctuations any small website admin can expect. Let us see how Bing handled my query:

Bing, like Google and Startpage, has my Visual Novel Reviews category in the top-three, here in second place. First and third are actual visual novel reviews. Bing put my February EDDA Café review in first – which is a fair choice. EDDA is an accessible original English language visual novel, it has done relatively well (by my visual novel review standards) in terms of visits, and I think it is a good article by my review article standards. In third place Bing has my 2021 pre-al|together project review of LoveChoice, which is available on Steam (as I note in the title of the review). While LoveChoice is not high on my personal list of favorite review articles, it is one of the better-performing visual novel reviews historically and its tendency to be on sale on Steam gives it more of a hook than my many reviews of ancient localizations of freeware Japanese visual novels.

Bing has a feature that Google does not. Under the first result for EDDA, it lists five visual novel reviews as short resorts – showing only the name of the novel being reviewed instead of the full title. While I am not evaluating those, I will say that Bing turned up some more interesting reviews in the extra links than it did for two of the main selections.

DuckDuckGo is one of the two best-known Bing derivatives in the United States, along with Yahoo! It has a focus on user privacy and while its main search is effectively Bing without personalization and trackers, it offers some unique additions to the basic search project (not to mention its own web browser, email aliasing service, and a few other goodies).

I have found that many people do not understand that DuckDuckGo draws its actual search results from Bing. If you had any doubts, you can see the identical top three results here when compared to a fresh Bing search with no profile.

Speaking of Yahoo! from before, I decided to try that too. Yahoo was once, long ago, a proud search engine and web directory. Now it is a Bing front-end (a pointless one if you ask me, but I digress). I decided to include it since there are still a non-insignificant number of Yahoo! users out there.

Yahoo has the same top-two as Bing and DuckDuckGo, but in third we see The Poor Little Bird, which was also in Google-based Startpage’s top three. Go figure. Again, while The Poor Little Bird is certainly responsive, it would not be one of my preferred results. I will note that on one occasion I used unwittingly used Yahoo! for a few searches because Bodhi Linux had it set as the default search engine in its default Chromium browser at the time. The results were less than satisfactory and felt significantly different than DuckDuckGo, which I am much more familiar with, despite also using Bing’s index.

Now let us return to Ecosia, this time using Bing instead of Google as the back-end.

The ads are unfortunate – but the results are very good. EDDA is a sound top choice and my review May Sky comes in second. May Sky is not only a good review article (if I do so say myself) but also my review of the best al|together novel (again, if I do say so myself). In third we have the full visual novel review category archive – which is the correct archive to use in this search. Ecosia-B also includes solid extra-results under the first and second main results, albeit I question including May Sky as an extra result for Edda when it is also the second main result.

Finally, I looked at Qwant, a French-based search engine that I think is still largely a Bing derivative (but some things may have changed since I last checked):

Qwant’s results are the same as Bing and DuckDuckGo, but it swaps first and second, putting the visual novel review category archive in first (which Google does as well) and EDDA in second. The only other notable difference between Qwant and Bing/DuckDuckGo is the different extra link selections.

Independent Search Engines

Now let us look at three English-first independent search engines. By “independent,” I mean these search engines maintain and use their own indexes. I know there are some questions about Brave Search, but I am comfortable with its inclusion here. Speaking of Brave, let us start with that one:

Seeing that Brave’s top three results are 100% identical to the Google derivative Startpage’s, maybe I should be less confident about putting Brave Search in the fully independent category (not the first time I have had questions). It shifts slightly after the top three – but Brave seems uncomfortably Google-index like on this particular New Leaf Journal site search. In any event, you can refer to my assessment of Startpage’s choices and apply it to Brave.

Next, let us look at Mojeek. Mojeek is a genuinely independent UK-based search engine with a large index, although the result quality is, in my view, hit or miss depending on what kind of search you are doing.

For this search, I dare say Mojeek scores a hit. It put my ACE Academy review in first, which is perfectly fair since I think it is a good review and it is readily available on Steam. In second and third we see my reviews of Red Shift and May Sky. While they are obscure in the grand scheme of visual novels, they are two of the better-known al|together pieces and they both rank among my top-five al|together novels, coming in fifth and first respectively out of 31.

Next we will try an upstart (still beta) independent search engine: Yep. Yep is run by Ahrefs, which is an SEO service with some free features and expensive paid plans (note: I have used Ahrefs’ free backlinks checker). Ahrefs decided to create a search engine based on its own index – perhaps to incentive webmasters to not block its crawlers. My domain-specific search seemed to work:

It leads with my al|together review archive, which is fine but an inferior choice to the overall visual novel review archive for this specific query. In second we see my recent (March 2024) review of Boku no Shokora. That strikes me as an odd choice – it is not an easy novel to find and run in English, and it certainly was not among the best pieces I reviewed, but I think the review itself was good, so I have no major complaints. In third we see my ACE Academy review, which is, for reasons I articulated above, a good pick.

Finally, let us examine Marginalia Search. Marginalia is the only non-generalist search engine in the selection. It is a free and open source search engine which focuses on independent and non-commercial websites (it has come in handy for some of my research projects). While it is not a search engine for all purposes, it has done a good job of indexing my independent non-commercial site, so I included it.

Marginalia goes with LoveChoice in first place and Bad End in second. While I do not love (pun intended) either choice (pun also intended) for reasons I explained previously, both are fair in that these novels are more readily accessible than many of the novels I reviewed due to their being available for purchase on Steam. I am curious whether Marginalia’s preference for them is influenced by the fact that they were my first two – and thus oldest – visual novel reviews. In third we see Crimsoness for the first time. I reviewed Crimsoness in May 2023. It is one of the better-known al|together pieces, evinced by the fact it has its own page on TV Tropes. I think it is a very good top three pick.

International Search Engines

Lastly, I decided to look at three international search engines that (A) have appeared in The New Leaf Journal before and (B) use (insofar as I understand) their own search indexes.

First up is the Russian focused Yandex search engine. While Yandex is primarily for Russians, it does have a fully English-facing search site (as well as Webmaster resources).

Yandex’s top three is responsive, but unusual. Yandex leads with my visual novel review category. In second we see my 2022 review of Save the Date, a fun English-language novel from 2013. It has been on the periphery of the top-three in other searches, but this is its first top-three appearance. I perceived Save the Date as having some niche popularity around when it was released – but I do not think that remains the case today. In third we have my review of the al|together piece Adagio. This one I am less satisfied with. Adagio is obscure, takes five minutes to read, and is not particularly interesting. I also do not think my review is notable as an article because I did not have much to say about the novel.

Next let us look at a Czech search engine called Seznam. I discussed it once in the context of whitelisting independent crawlers:

Seznam, like Marginalia and Mojeek, does a great job in choosing actual review articles. Collage, in first place, is one of the best and most interesting novels I reviewed. It is obscure, but one that I hope my review will inspire some people to try. In second we have Moonshine, which I perceive as relatively well-known by al|together standards and a fair choice. Like Marginalia, Seznam has Crimsoness at third – and this is another good choice. Seznam also has nice depth. While my survey only considers top-three rankings, spots 4-6 would have also been fair top-three choices.

Cốc Cốc is a Vietnamese search engine that appears to be fully independent. Let us see how it handled my query:

Cốc Cốc is only the second search engine after Mojeek to include Red Shift, but here we see Red Shift in first place. That is a solid result. Bad End comes in second, which we also saw in Bing-based Ecosia and Marginalia. Finally, My Black Cat, which appeared regularly as an extra result in the Bing group, makes its second top-three appearance after appearing in Qwant’s list. Cốc Cốc did well in choosing Red Shift but its fourth and fifth choices – May Sky and ACE Academy, would have been better than its second and third choices.

Japan-focused Search

Because most of my visual novel reviews are of localizations of Japanese visual novels, I decided to try two Japan-focused search tools to see what they came up with for my English-language searches.

First is Yahoo! Japan, which is distinct from Yahoo! outside of Japan. I was initially confused about what powered it search and my confusion was not ameliorated by its Wikipedia article. I learned after some digging that Yahoo! Japan’s results are powered by Google, at least through 2025 (it is reportedly considering switching to Naver, which may be a bad thing for our limited prospects in the Japanese marketplace). However, Yahoo! Japan interprets queries against Google’s index in a way that is supposed to be beneficial for Japanese-language searchers, meaning that it may be more distinct from Google than English-language derivatives such as Startpage. With that introduction out of the way, let us see how it did with my decidedly non-Japanese query:

Yahoo Japan leads with EDDA like the U.S. edition, but in second we see my archive for al|together and Insani reviews instead of the full visual novel archive (that is a minor downgrade). There is a unique novel in third – my review of an al|together 2006 piece called Night of the Forget-Me-Nots. That would be far from my third choice to appear in the top three, but I will take it over the more commonly ranked Bad End.

Because most of my visual novel reviews cover localizations of Japanese visual novels, I decided to do a bonus test on Goo, a Japanese-focused Google front-end which previously collaborated with Yahoo! Japan (see Yahoo! Japan Wikipedia entry linked in previous section).

Goo does well. Unlike Google and Startpage, it excludes category archives. In first we have EDDA Café, which is a decidedly English-first visual novel (with a Japanese localization, however) and a Bing favorite. In second we have Moonshine, which I again note is one of the more prominent (again, relative) al|together pieces. Finally, in third we have I, Too, Saw Dreams Through Air, an al|together 2005 translation that I will venture is little known even by the standards of that niche project. However, for whatever it is worth, I think my I, Too, Saw Dreams Through Air review is well on the right side of average in the visual novel review collection, so I am happy with seeing it in the top three. One interesting note is that we have Narcissu, which is by far the most well-known al|together novel (thanks in part to being free on Steam), in fifth. This was the first time I saw Narcissu close to a top-three, which I will mark as a surprise.

Grading the Results

I will provide my subjective grades for the results. I have employed a school grading system, A-F. While I am not using a scoring system, I will lay out my criteria:

• I am only considering the actual three main search result. Extra results under the result, such as what we saw in Bing, DuckDuckGo, and Qwant, will not be considered.
• I am taking into account search intent. Searching a single site specifically for visual novel reviews is a different thing than searching the web for visual novel reviews. I assume that someone searching my site for visual novel reviews already found at least one of my reviews and is curious to see what else I covered (but apparently not inclined to follow links from my reviews or check the category archive).
• I will subjectively weigh the results in descending order of significance, giving most weight to the top result and least weight to the third.
• I allow for one archive-type result (either a category archive or an article which largely links to reviews) but ding the search of it includes more than one. That is, each search should have at least two actual visual novel reviews. Some archive results are better than others (e.g., the full visual novel review category is better than the broader visual novel category or the narrower al|together/insani review category).
• I will give preference to search engines that give results that provide readers of a broad idea of which kinds of novels I review. For example, I prefer selections that include at least one of my non-Japanese novel review and at least one of my al|together reviews.
• I have my own opinions of which reviews are the best to rank high. These opinions are informed by the relative merit of what I am reviewing (the better the piece and the more readily available it is, the more likely it is that the review will be useful to a potential reader), and my own assessment of how interesting my review is. While I think my later reviews are generally better than my early reviews, I will note that a review is something of a conversation between me and the piece. Some novels give me much more to work with than others.
• I want to see that the search algorithm (or non-algorithm in the case of our site search) put some “thought” into the choices. For example, if I see my first three or most recent three reviews, I will downgrade the search since the quality of the results is tied to when the articles were published.

Now for our grades.

• A: Ecosia (Bing); Goo; Mojeek
• B: Cốc Cốc; Marginalia Search; Seznam; Yahoo! Japan; Yep
• C: Bing; DuckDuckGo; Qwant; Yahoo!; Yandex
• D: Google; Ecosia (Google); Startpage; The New Leaf Journal
• F: Brave Search

I only gave out three A’s. While the Japan-based Google front-end Goo did fine work, I do not recommend it as a general-use practical tool for searching The New Leaf Journal. Mojeek delivered a perfect return for my query, including my best original English-language visual novel review and my reviews for what I think are the two best al|together pieces (my Collage review is one of my better al|together articles). I gave a surprising A-grade to Ecosia-Bing. It led with EDDA (a good original English language VN review) and followed with May Sky (my review of my favorite al|together novel) and my full visual novel review category archive. I will do more testing with Mojeek to see how well it works as a New Leaf Journal-search engine since its strong privacy policy and own search index would make it a great choice if it works well as well for other queries as it did for this one.

The B-group all satisfied my criteria, but my subjective assessment of their article selections was less than that of my three A-tier picks. I could have easily given Seznam an A since it picked three good al|together reviews, but I noted in my criteria that I wanted to see at least one novel review from outside my al|together project. Marginalia Search is worth exploring as a possible external search tool for The New Leaf Journal (especially recommendable because it is fully open source and privacy-friendly). I was impressed with Yep for this single query, but I have not tested it much beyond this small project.

The C group features Bing, DuckDuckGo, and Qwant, which were all nearly (or in the case of DuckDuckGo and Bing) identical. Yandex did a better job than the Bing group of satisfying my criteria, but my subjective distaste for placing the Adagio review in third bumped it down from B-group. Of these, DuckDuckGo is probably a solid choice as a New Leaf Journal search tool, notwithstanding its Bing-based mediocrity on this specific query.

The D group all messed up in one way or another, led by Google and our site-specific search failing to include a single, specific, visual novel review. Google-flavored Ecosia did better than Google and Startpage on its second and third choices, but it gets a D for its very odd pick for the top article.

I am giving Brave Search an “F” for this particular query. Brave Search touts itself as having its own independent search index. In its options, the user can opt to enable Google fallback mixing, which allows Brave to use Google’s index for queries where its own index is not yet up to par. I had that toggle disabled for this test. Nevertheless, I am reasonably confident from seeing that Brave’s results were nearly identical to Startpage’s that Brave silently fell back to Google even though I had the toggle disabled. I tried another query limited to my domain for insani visual novel and in that case, Brave provided original results that were very distinct from Google and Startpage, leading me to be confused why it failed to do so for visual novel review. In any event, Brave should make it more obvious when it falls back on Google and it should avoid doing so when the user has disabled the Google fallback option. I offer these critiques as someone who uses Brave as a secondary web browser and mixes Brave Search into my search diet.

Final Thoughts

This was obviously a very limited, single-query test, to see how a variety of search engines and front-ends handled a general search limited only to our domain. Most handled it acceptably well, although some, for example Mojeek, did better than others. I may try some other queries in different subject areas in the future. If you are interested in searching our site with a general-use search engine, see my new resource page on the subject.

I conclude by noting that I was surprised to not see my reviews of Insani’s two most well-known visual novel localizations – Narcissu and True Remembrance in any of the top threes (or in the case of True Remembrance, anywhere close to a top three). Go figure.