Where To Debug Errors In Your Sitecore Search Index

Step-by-step process to locate the real issues

February 29, 2024

By David Austin

Have you been setting up a source in Sitecore Search and had trouble determining what exactly went wrong? There’s a good chance, if you were like me, you were looking in the wrong location.

Web crawler error log with a list of documents showing 'Error crawling document' messages, indicating issues with the crawling process.

Seeing a bunch of errors that simply state Error crawling document endlessly listed can be a bit daunting and confusing.

While within the Sitecore Search console, navigate to Analytics > Sources > Overview.

Screenshot of an analytics dashboard with a navigation menu on the left, highlighting 'Sources' and 'Overview'. A graph displaying 'Items Indexed' over the last 7 days for 'ABC Company' is visible with a peak in activity

Once on the page, click on the source listed in the Source Details section. Once it loads you’ll be presented with a report on the last run of the source.

Detailed view of a source crawler summary showing 'Last Run Processing Summary' with status finished, start and end times, duration, and number of items indexed.

Scroll down until you see the Job Run List section and click on the last run indexing.

List of web crawler job runs showing one successful run with 276 items indexed and a subsequent failed run with zero items indexed.

Once the report opens, you’ll have a few ways to view the failed items in your index.

Using View Details to View Issues

The first way is to use the View Details bar at the top of the report.

Overview of web crawler run details including 'Specs Analyzed', 'Crawled', 'Indexed', 'Validated', with a notice of 21 URL scanning analytics errors and a button to 'View Details'.

You can see upon clicking View Details a modal popup will show displaying a list of errors complete with the urls with the errors and the issue on them. You can see below the wide variety of errors that are possible.

Popup window detailing web crawler error messages including 'Something went wrong for url', 'invalid payload size', and 'context deadline exceeded'.

Using the URL Details Table at the Bottom of the Page

If you scroll down on the report, you’ll get a tabulated list of URLs, the result and reason for their failure. If you use the Result filter, you can filter specifically by Failed.

Table of URL crawling errors with timestamps, showing multiple 'Failed' statuses and various error messages such as 'Something went wrong' and 'Client.Timeout'

Review

The types of errors you can get are varied and ultimately could depend on a number of factors which could include issues in the source configuration, but also on the site itself. These could be broken links, invalid SSL certs, bad redirects, etc.

Hopefully this makes it easy to debug something that initially seemed rather ominous.



Image of Fishtank employee David Austin

David Austin

Development Team Lead | Sitecore Technology MVP x 3

David is a decorated Development Team Lead with Sitecore Technology MVP and Coveo MVP awards, as well as Sitecore CDP & Personalize Certified. He's worked in IT for 25 years; everything ranging from Developer to Business Analyst to Group Lead helping manage everything from Intranet and Internet sites to facility management and application support. David is a dedicated family man who loves to spend time with his girls. He's also an avid photographer and loves to explore new places.

Second CTA Ogilvy's Legacy

Today, David Ogilvy's influence can still be felt in the world of advertising.

Ogilvy's Influence Example
Emphasis on research Market research is a crucial part of any successful advertising campaign
Focus on headlines A strong headline can make the difference between an ad that is noticed and one that is ignored
Use of visuals Compelling images and graphics are essential for capturing audience attention