A sitemap, as you may already know, is the most important part of a website. It contains a list of links of a website and helps search engines in crawling/indexing pages which it may not find it. Today, I have good news for the users of Website Crawler. WC can now crawl the links it finds in the sitemap file. Yes, that’s right. This feature was on our checklist and we have rolled it out today.
In case you’re wondering how to use the new XML crawl feature of Website Crawler, here’s a tutorial that you can follow.
How to make Websitecrawler crawl sitemap links?
Enter the direct link to the XML format sitemap file of your website in the large text box you see on the Website Crawler’s homepage. Don’t worry about the file’s size. WC can analyze sitemaps containing 25000+ URLs.
Once you enter the URL, enter the number of URLs you want the website crawler to crawl in the “Depth” textbox. Now, click the submit button.
Note: Free account of WC supports 550 URLs i.e. no matter how big the sitemap file is, only up to 550 URLs will be processed. Our Silver (paid) plan supports 2500 URLs.
When you hit this button, WC will start crawling your website URLs. To see the status i.e. current URL being crawled and the list of processed URLs, click the “Status” button.
Once WC finishes processing the sitemap links, you’ll see a new form that asks you to enter your email address. Enter your email ID and click the submit button. WC will now send a verification email to your inbox. Enter the 3 digit code in the textbox and hit the button “Submit”. WC will now create your websitecrawler.org account through which you can see the On-Page SEO reports of your website.
Conclusion: If you want Website Crawler to analyze a fixed set of URLs or the links of the sitemap files only, enter the link to the sitemap file instead of a website URL in the large text box that you’ll find on WC’s home page.
