11th December, 2015 | Tutorials |

How to get your WordPress Site Indexed by the Major Search Engines

Using Wordpress? Get our lightening fast Wordpress Optimised Hosting.

Get Wordpress Hosting

One of the first things to think about when one creates a blog or site is to have it visited and read by as many people as possible. Readership and traffic make or break a site. So, how does one bring visibility to a site? Friends, family and social media can give some traffic, but it soon reaches a plateau.

The one sure-shot method to bring in traffic is to try to get organic traffic i.e., make your site visible to search engines. This potentially opens up your site to the entire world. If your content is good, and you have submitted your content to the search engines, anyone in the world can visit your site.

So, how do you get your site visible to search engines? For your site to be visible get your site's pages indexed by search engines. That brings up the next question. What is Search Engine Indexing?

Search Engine Indexing

Search Engine Indexing is the process of collecting, parsing and storing data by a search engine. The index is the place where all the collected data is stored. Using this index, a search engine generates results for search queries. Without an index, the search engine may take several times longer to return results and it would be inefficient in terms of time and effort involved.

Search engines use crawlers or bots or robots to collection information from the various sites on the internet. How do these crawlers or bots work? They visit a web site, read its content, and then follow the hyperlinks on that page and thus move from page to page. This is how crawlers gather information site by site, page by page. If other sites have linked to your site, the crawler visits your site too. Some of the popular crawlers are Googlebot, Bingbot, Swiftbot, Fast Crawler, Polybot and GRUB.

Get your site visible

Now that you know that for your site to be visible, search engines have to index your site and its pages, what should you do about it?

First, create an XML sitemap. A sitemap is a list of all the pages on a website that is available for anyone to see. A website uses an XML sitemap to indicate the pages available to the search engines. This allows search engines to better crawl your website. Sitemaps are very important from the SEO (Search Engine Optimization) perspective.

###XML Sitemap###

Though there are many ways to create an XML sitemap in WordPress, the easiest way is to use a simple plugin. Sometimes XML sitemaps come bundled with a host of other features in SEO plugins like Yoast SEO. In other cases, you can use a simple plugin for the purpose of XML sitemaps only like Google XML sitemaps.

###Installation of Yoast SEO plugin###

Installation of Yoast SEO plugin is similar to any plugin installation. From your WordPress admin dashboard, click on Plugins > Add New. Enter Yoast SEO in the search box and from the results, click against Install Now next to the Yoast SEO plugin.

Activate the plugin. From the admin dashboard, click on SEO > XML sitemaps to see a screen similar to the one below.

Ensure that you check against Check this box to enable XML sitemap functionality. Click on Save Changes.

Now, if you click on the XML Sitemap button in the same screen, your site's XML sitemap will be opened in a new window with the URL your-site-domain/index.php/sitemap_index.xml.

Here you will find a list of 4 (or more if you have custom taxonomies) xml files and their links

Each of these is a sitemap in itself containing a list of posts/pages/categories/tags. If you click on one of these, you will get the list of posts and their links.

Note: In case, you get a Page not found error when you click on the XML Sitemap button, the most probable reason is that your permalink structure is not saved. Go to Settings > Permalinks and click on Save Changes. Now go back and click on the XML Sitemap button to see your XML sitemap.

After creating a sitemap, the next step is submission to popular search engines.

###Submit your sitemap to Google###

Google is one of the most popular search engines and so, you should definitely submit your sitemap to Google.

Register for a Google Account, if you already do not own one. Go to the Google Webmaster Tools site, login with your credentials.

You will see Add a property button.

Click on it and in the popup box that appears, enter the URL of your website. Click on Continue.

On the next screen title Search Console, you have to verify the ownership of your website. You can verify the ownership by the Recommended method of HTML file upload. Download the HTML verification file and place it in the WordPress root directory. Ensure that the file's ownership and permissions are such that it is readable by all.

But, since you are using the Yoast plugin, you can use an easier method. Click on the Alternate methods tab. There are 4 alternate methods. Click on the radio box against the HTML tag method. You will find a meta tag as shown in the image.

Copy the value given within quotes for content (content="XXX"), as shown in the figure.

From your WordPress admin dashboard, click on SEO > General. There are four tabs on this screen, click on the Webmaster Tools ** tab. Paste the copied value in the previous step in the input box for **Google Search Console. Click on Save Changes.

Back at Google Webmaster tools, click on the Verify button. You will get a message like "Congratulations, you have successfully verified your ownership of" if the verification is successful. Continue to the dashboard.

Here the Current Status is shown with Crawl Errors, Search Analytics and Sitemaps. Click on Sitemaps.

In the next screen, click on ADD/TEST SITEMAP.

In the pop-up box that appears, enter your sitemap file name (the url after the base url) sitemap_index.xml.

After adding your sitemap, the status will appear as Pending. It typically takes a few hours to a day for Google to crawl your website. Once Google has crawled your website, you can see the Statistics.

Overtime, the statistics improves and you can see enough data to make meaningful inferences.

###Submit your sitemap to Bing###

Another popular search engine is Bing. Submit your sitemap to Bing as well. Go to Bing Webmaster Tools and login with your Microsoft account or Bing account.

On the Dashboard, click on My Sites. Enter your site's URL in the input box labelled Add a Site. Click on ADD.

In the screen title Add a Site, give your site's URL, location of the XML sitemap and an estimated time duration when you get maximum traffic. There are five options provided in the drop-down list. This helps Bing crawl your site when you do not have much traffic. Click on ADD.

The next step is verification. Bing offers 3 methods. As in the case of Google, you can use Option 1, download the verification XML file and upload it to the WordPress root directory, taking care of providing required permissions.

Alternately, you use Option 2 in combination with Yoast SEO plugin. Copy the value for the content (content="YYY") given within quotes.

In the WordPress admin dashboard, click on SEO > General. As noted earlier, there are four tabs on this screen, click on the Webmaster Tools ** tab. Paste the copied value in the previous step in the input box for **Bing Webmaster Tools. Click on Save Changes.

Head back to Bing Webmaster Tools and click on VERIFY. If verification is successful, you will be redirected to the Dashboard. The Sitemaps status is Pending. It may take time for bingbots to crawl and index your site completely after which the status changes to Success.

Over time, your site's Statistics improves and you can infer useful insights from the data that is collected. For instance, look at this Site Activity of a website for Last 30 days. The changes in the number of clicks from search, number of pages crawled, the number pages indexed and other details are provided - both the absolute value and the percentage increase or decrease. If any regular pattern changes, it is a trigger for you to think of what recent changes you made to the site and how it is affecting your site.

With that, you have made your site visible to the two most popular search engines. You can also submit your site to Yandex, Alexa, Pinterest etc. The more the number of search engines or crawlers that visit your site, the more will be the number of pages of your site that get indexed. This is the way to improve your organic traffic.

(For the purpose of this tutorial, WordPress version 4.3.1 and Yoast SEO version 3.0.3 were used)

Using Wordpress? Get our lightening fast Wordpress Optimised Hosting.

Get Wordpress Hosting