Regenerate the Sitemap on a Local XM Cloud Docker Development Environment

Introducing the Sitecore XM Cloud sitemap developer utilities

August 19, 2024

By Jeff L'Heureux

Sitemap is Empty in a Local XM Cloud Development Environment

In a local XM Cloud Docker development environment, the /sitemap.xml front-end route always returns an empty XML file.

Browser showing an empty sitemap XML with no data below the root.

When a headless site is created, the sitemap media library item that is scaffolded is empty and has a size of 0 bytes.

Sitecore Content Editor with a protected sitemap item highlighted.

This makes it impossible to debug the \src\pages\api\sitemap.ts API route handler and modify it to our needs.

This article dives into the sitemap regeneration process and solutions for local XM Cloud Docker development environments.

Impossible to Regenerate the Sitemap

In a hosted XM Cloud environment, this sitemap item is regenerated at the end of a publish operation of any item. An OnPublishEnd processor checks the SXA sites sitemap settings and regenerates the sitemap if enough time has elapsed since the last regeneration.

In a local Docker development environment, we cannot publish as we do not have a web database or a valid Experience Edge target. Attempting to publish results in an exception:

#Exception: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
---> Sitecore.ExperienceEdge.Connector.ContentHubDelivery.Exceptions.EdgeApiException: [ExperienceEdge Publishing]: An error has occurred while publishing JobStart event 9614b999-f4dc-4fb7-ab08-75bf8a2bf429 for publishing job '638580401015070000'. Please see inner exception for more details.
---> Sitecore.Exceptions.ConfigurationException: Invalid Authority connection string. Required parameter 'url' is missing.

Thus, the sitemap is never regenerated and the media library item is never updated.

Introducing the Sitecore XM Cloud Sitemap Developer Utilities

Faced with that limitation and the need to replicate a production sitemap issue in a development environment, I found creative ways to regenerate the sitemap media library item and get valuable information.

The end result is a configurable utility page that can be deployed to a local XM Cloud CM. There are a few steps to use it:

  1. Ensure your local XM Cloud Docker Compose project is running and the CM is healthy and accessible.

  2. Create a generateSitemap.aspx file in your project \docker\deploy\platform folder with the code below:

     <%@ Page language="c#" %>
 <%@ Import Namespace="Microsoft.Extensions.DependencyInjection" %>
 <%@ Import Namespace="Sitecore.Abstractions" %>
 <%@ Import Namespace="Sitecore.Configuration" %>
 <%@ Import Namespace="Sitecore.Data" %>
 <%@ Import Namespace="Sitecore.Data.Items" %>
 <%@ Import Namespace="Sitecore.DependencyInjection" %>
 <%@ Import Namespace="Sitecore.Events" %>
 <%@ Import Namespace="Sitecore.Globalization" %>
 <%@ Import Namespace="Sitecore.Pipelines" %>
 <%@ Import Namespace="Sitecore.Publishing" %>
 <%@ Import Namespace="Sitecore.Sites" %>
 <%@ Import Namespace="Sitecore.Web" %>
 <%@ Import Namespace="Sitecore.XA.Foundation.SiteMetadata.EventHandlers" %>
 <%@ Import Namespace="Sitecore.XA.Foundation.SiteMetadata.Sitemap" %>
 <%@ Import Namespace="Sitecore.XA.Foundation.SiteMetadata.Pipelines.Sitemap.GenerateSitemapJob" %>

 <script runat="server">
     // IMPORTANT
     // Change the values of those variables to match the Headless SXA site name and home item you want to generate the sitemap for.
     private const string SITE_NAME = "sxastarter";
     private const string HOME_ITEM_ID = "{BEE17B56-2406-4935-A730-CFE90356BE2C}";
 </script>

 <!DOCTYPE html>
 <html>
     <head>
         <title>Sitecore XM Cloud Sitemap Developer Utilities</title>
         <meta content="C#" name="CODE_LANGUAGE">
     </head>
     <body style="font-family: sans-serif">
         <h1>Sitecore XM Cloud Sitemap Developer Utilities</h1>
         <form runat="server">
             <div>
                 Start with this button. It calls the SitemapCacheClearer.OnPublishEnd method with a mock OnPublishEnd event. The Sitecore CacheItemClearer should create one Sitemap refresh job per SXA site. It should save the sitemaps to the media library. You can track the jobs in the <a href="/sitecore/admin/Jobs.aspx" target="_blank">jobs admin page</a>.
             </div>
             <br />

             <asp:Button id="SitemapCacheClearer"
                Text="1. SitemapCacheClearer.OnPublishEnd()"
                OnClick="SitemapCacheClearer_Click"
                runat="server"/>
             <br /><br />

             <div>
                 If the above button does not generate the sitemap media library items, try these buttons in order.
             </div>
             <br />

             <div>
                 This one runs the sitemap.generateSitemapJob pipeline for the site you defined in the SITE_NAME variable. It should save the sitemap to the media library.
             </div>
             <asp:Button id="GenerateSitemapJob"
                Text="2. sitemap.generateSitemapJob pipeline"
                OnClick="GenerateSitemapJob_Click"
                runat="server"/>
             <br /><br />

             <div>
                 This one runs only the GenerateSitemap processor of the sitemap.generateSitemapJob pipeline for the site you defined in the SITE_NAME variable. It does not save the sitemap to the media library.
             </div>
             <asp:Button id="GenerateSitemap"
                Text="3. GenerateSitemap.Process()"
                OnClick="GenerateSitemap_Click"
                runat="server"/>
             <br /><br />

             <div>
                 This one runs the ItemCrawler and lists the page items it found for the HOME_ITEM_ID variable you defined. It does not save the sitemap to the media library.
             </div>
             <asp:Button id="ItemCrawler"
                Text="4. ItemCrawler.GetItems()"
                OnClick="ItemCrawler_Click"
                runat="server"/>
         </form>
         <br />

         <h2>Output</h2>
         <code>
             <%=Output %>
         </code>
     </body>
 </html>

 <script runat="server">
     private string output = "None yet. Please click one of the buttons.";

     void SitemapCacheClearer_Click(Object sender, EventArgs e)
     {
         output = GetTime() + " Starting SitemapCacheClearer.OnPublishEnd()\n\n";

         SitemapCacheClearer clearer = new SitemapCacheClearer();
         Database master = Factory.GetDatabase("master");
         List<string> targets = new List<string>(new string[] { "Edge" });
         PublishOptions options = new PublishOptions(master, master, PublishMode.SingleItem, Language.EnglishLanguage, DateTime.Now, targets);
         Publisher publisher = new Publisher(options);
         SitecoreEventArgs args = new SitecoreEventArgs("OnPublishEnd", new object[] { publisher }, new EventResult());
         clearer.OnPublishEnd(null, args);

         output += "Cancelled?: " + args.Result.Cancel.ToString() + "\n";
         if (args.Result.HasMessages)
         {
             output += "Messages:\n";
             foreach (string message in args.Result.Messages)
             {
                 output += "- " + message + "\n";
             }
         }
         if (args.Result.HasReturnValues)
         {
             output += "Return Values:\n";
             foreach (var returnValue in args.Result.ReturnValues)
             {
                 output += "- " + returnValue.ToString() + "\n";
             }
         }

         output += "\n" + GetTime() + " SitemapCacheClearer.OnPublishEnd() done. The Sitemap should have been saved to the media library.";
     }

     void GenerateSitemapJob_Click(Object sender, EventArgs e)
     {
         output = GetTime() + " Starting the sitemap.generateSitemapJob pipeline.\n\n";

         SiteContext siteContext = SiteContextFactory.GetSiteContext(SITE_NAME);
         GenerateSitemapJobArgs args = new GenerateSitemapJobArgs();
         args.SiteContext = siteContext;
         BaseCorePipelineManager pipelineManager = ServiceLocator.ServiceProvider.GetService<BaseCorePipelineManager>();
         pipelineManager.Run("sitemap.generateSitemapJob", (PipelineArgs)args);
         var messages = args.GetMessages();

         output += "Aborted?: " + args.Aborted.ToString() + "\n";
         output += "Suspended?: " + args.Suspended.ToString() + "\n";
         if (messages.Length > 0)
         {
             output += "Messages:\n";
             foreach (PipelineMessage message in messages)
             {
                 output += "- " + message.Text + "\n";
             }
         }
         output += "\nGenerated Sitemap:\n";
         foreach (string item in args.SitemapContent.Values)
         {
             output += item + "\n";
         }

         if (args.Aborted || args.Suspended)
         {
             output += "\nThe pipeline was aborted or suspended. The Sitemap might not have been saved to the media library.\n";
         } else {
             output += "\nThe Sitemap should have been saved to the media library.\n";
         }

         output += "\n" + GetTime() + " sitemap.generateSitemapJob pipeline done.";
     }

     void GenerateSitemap_Click(Object sender, EventArgs e)
     {
         output = GetTime() + " Starting GenerateSitemap.Process()\n\n";

         SiteContext siteContext = Sitecore.Sites.SiteContextFactory.GetSiteContext(SITE_NAME);
         GenerateSitemapJobArgs args = new GenerateSitemapJobArgs();
         args.SiteContext = siteContext;
         GenerateSitemap processor = new GenerateSitemap();
         processor.Process(args);
         output += "Generated Sitemap (Not saved to the media library):\n";
         foreach (string item in args.SitemapContent.Values)
         {
             output += item + "\n";
         }

         output += "\n" + GetTime() + " GenerateSitemap.Process() done.";
     }

     void ItemCrawler_Click(Object sender, EventArgs e)
     {
         output = GetTime() + " Starting ItemCrawler.GetItems()\n\n";

         Database master = Factory.GetDatabase("master");
         Item homeItem = master.GetItem(HOME_ITEM_ID);
         ItemCrawler crawler = new ItemCrawler();
         IList<Item> items = crawler.GetItems(homeItem);
         output += "Pages returned by ItemCrawler.GetItems():\n";
         foreach (Item item in items)
         {
             output += "- " + item.Name + "\n";
         }

         output += "\n" + GetTime() + " ItemCrawler.GetItems() done.";
     }

     string GetTime()
     {
         return DateTime.Now.ToLongTimeString();
     }

     private string Output {
         get {
             return Server.HtmlEncode(output).Replace(" ", "&nbsp;").Replace("\n", "<br />");
         }
     }
 </script>

  3. The first <script> block contains 2 variables you must set to the site name and site home item ID you want the sitemap to be generated for. Ensure you set those correctly.

     <script runat="server">
     // IMPORTANT
     // Change the values of those variables to match the Headless SXA site name and home item you want to generate the sitemap for.
     private const string SITE_NAME = "sxastarter";
     private const string HOME_ITEM_ID = "{BEE17B56-2406-4935-A730-CFE90356BE2C}";
 </script>

  4. Save the file. The container entrypoint script will copy the saved file to the CM container wwwroot folder.

  5. In your browser, load the https://xmcloudcm.localhost/generateSitemap.aspx page and follow the instructions.

    Sitecore XM Cloud sitemap developer tools interface with cache and sitemap generation options.

  6. Reload your sitemap route and enjoy a regenerated sitemap!

    Browser showing XML sitemap with loc, lastmod, changefreq, and priority elements.

Tips When the Sitemap Does Not Generate

  1. Ensure you edited the SITE_NAME and HOME_ITEM_ID variables values in the generateSitemap.aspx file and you saved your changes.
  2. Check the site’s sitemap settings (/sitecore/content/YourSiteCollection/YourSite/Settings/Sitemap)
    1. Crawler: The value should be either itemCrawler or indexCrawler
      1. Those crawlers are defined the Sitecore config under the <sitemapItemCrawler> node.
    2. Cache
      1. Refresh threshold: Set it to 0 to force a regeneration every time.
      2. Cache Type: You can play with the “Stored in cache” and “Stored in file” options.
        1. Both should update the sitemap media library item.
      3. Cache expiration: Set it to 0 to force a regeneration every time.
    3. Media items
      1. Generate sitemap media items: Ensure it is checked.
  3. Delete your site sitemap media library item (/sitecore/media library/Project/YourSiteCollection/YourSite/Sitemaps/YourSite/sitemap).
    1. Sitecore will inform you that there are links to that element. Chose to remove the links.
    2. The sitemap media library item will be recreated and relinked correctly the next time it will be generated.

Happy Sitecoring!


Related Links

Jeff L'Heureux

Jeff L'Heureux

Director of Technology | 4x Sitecore Technology MVP

Jean-François (Jeff) L'Heureux is an experienced leader in Sitecore and Coveo technologies, having worked in both organizations. He is a 4 time Sitecore Technology MVP, and 3 time Coveo MVP. He has 17 years of software development experience, including 11 years of Sitecore experience. He specializes in front-end, and he has experience in technologies like Next.js, React, Vercel, Netlify, Docker, Coveo Cloud, Coveo for Sitecore, Sitecore XP/XM, and the latest Sitecore technologies including XM Cloud, JSS, CDP, Personalize, OrderCloud, Discover, Send, Search, and Content Hub ONE. Outside of work, he is found outside practicing sports like rock climbing and hiking.

These are some related articles

Sitecore Personalization Deep Dive: XP vs XM Cloud vs Personalize

February 19, 2025

Designing Responsive Components in Headless Sitecore with Next.js for any Container

February 19, 2025

New Year, New Goals: A Sitecore Developer's Resolution for 2025

February 7, 2025

How to Integrate Google reCAPTCHA v3 in a Sitecore Next.js Headless App

January 29, 2025