Have you ever heard of IndexNow?
IndexNow is an initiative from the Microsoft Bing and Yandex search engines that is also now supported by Google. It’s designed to overcome the acute difficulties many sites have in getting any of the pages/posts indexed. I often encounter relatively new websites which have not been able to get any/all pages indexed.
Bing and Yandex created IndexNow, an open-source indexing technology and launched it in October 2021. IndexNow makes it simple for webmasters to alert search engines when new pages are published or existing pages are amended or deleted. After getting a notification, search engine crawlers will promptly visit the relevant pages for indexing purposes. This is greatly appreciated if you are an SEO specialist.
IndexNow is not part of the standard indexing protocol used by Bing or Yandex. It’s actually a completely separate indexing protocol, only used by search engines on websites that have manually activated this new indexing protocol. They will continue to use their usual index methods if you have not enabled IndexNow on your website.
Push-based Indexing
IndexNow uses a pinging technique to tell search engines when pages are published, changed, or removed. It is an example of push-based indexing. By default, most search engines use pull-based indexing. Crawlers from search engines will crawl the internet, visiting websites and individual pages to index them.
Search engine crawlers will only visit a website or page upon request with push-based indexing. Because it includes pinging, IndexNow is a type of push-based indexing. A ping is a signal sent to search engines informing them that a page was recently published, changed, or deleted. You can use IndexNow to ping Google, Bing or Yandex to draw their respective crawlers to these pages.
What benefits does IndexNow Offer You?
Primarily, it will result in faster indexing because it is a type of push-based indexing. If your SEO agency utilises IndexNow on your website, Google, Bing and Yandex will index your website’s new pages faster. Pings are made up of the name of the search engine, the URL of a page, and a unique key. If you ping Google, Bing or Yandex, they will immediately visit the page. As a result, newly published pages are indexed more quickly.
The IndexNow protocol improves internet efficiency by instantaneously notifying participating search engines of updates to your website. For example, whenever you alter your content or create a new post or page, the IndexNow protocol will notify search engines. As a result, search engines will crawl and index your modifications more quickly.
One of the most significant efficiency improvements for SEOs is that the IndexNow protocol reduces exploratory crawls and increases crawl efficiency. This is due to the protocol informing search engines about changes to your website.
You are not required to ping each search engine. IndexNow’s pinging method is intended to alert all search engines that support the new open-source indexing protocol. When you ping Bing, for example, the notification is also sent to Google, Yandex and any other search engines that have adopted IndexNow.
The IndexNow protocol improves internet efficiency by instantaneously notifying participating search engines of updates to your website. For example, whenever you alter your content or create a new post or page, IndexNow will notify search engines. As a result, search engines will crawl and index your modifications more quickly.
How to Implement IndexNow
For a long time, providers of SEO services for WordPress around the world have wished for instant indexing. Finally, the IndexNow protocol has brought that idea to fruition. “How to implement IndexNow on your website?” is the 64 million dollar question.
This is the painful part… it is fair to say that it is mind-numbingly tedious and complicated to get set up on both Google and Bing. In order to use IndexNow, you need:
An Application Programming Interface (API) key from Bing
A Google Cloud Platform Service Account and API key, and:
A WordPress plugin, either the:
Instructions:
Fortunately, the guys at RankMath have an excellent tutorial on getting the Google API and Search Console settings done. Even if you are already using Yoast, AIO SEO or other SEO plugins, you can still use the RankMath Instant Indexing module. See: https://rankmath.com/blog/google-indexing-api/
Similarly, AIOSEO have a tutorial here: https://aioseo.com/what-is-indexnow-for-seo/
Real World Results with IndexNow
In the first few months of 2022, I’ve worked on several websites with indexing issues:
Example 1: Electrician
An electrical services business in Canterbury with a site that was a year old – and not even the Home page was indexed. Okay, so it had serious content quality issues: spelling, punctuation, grammar etc., broken links, poor Titles and descriptions, keyword-deficient URLs, no image alt texts and no internal keyword links within content… The Yoast SEO plugin was badly configured. But even with all that fixed, Google ignored it for weeks…
Within 24 hours, EVERY page on the site was indexed!
- The home pages were ranking for several keywords
- 2 internal pages had attained rankings
Example 2: Real Estate Agent
A real estate agency in Washington DC had bizarre indexing issues on their website which was hosted on Bluehost. I ran the same IndexNow process on this website. Zero response… for some weird reason, Google could not access pages on this website and for the life of me I could not see HOW or WHY it was getting blocked – it was simply inexplicable…
After spending quite a lot of time researching this issue, I found an article someone else had written after experiencing pretty much identical circumstances with a website hosted on Bluehost:
– https://support.google.com/webmasters/thread/127080064/wordpress-site-blocked-from-indexing-unable-to-fetch-sitemap-or-robots-txt?hl=en
Where the:
1 – The domain is not housed in the top level of /public_html/ and is in a sub-directory
2 – the name servers allowed recursive queries
Bluehost had apparently been unable to solve the problem – we short-circuited that by relocating the site to A2Hosting…
Within 36 hours of relocation and resubmission of all pages, the entire site was fully indexed!
Having Trouble Getting Your Site Indexed?
I’m able to provide this as part of a WordPress SEO package…
Page last updated on Wednesday, October 11, 2023 by the author Ben Kemp