In this article, we will discuss how to generate a sitemap in a MERN application. But before that what is a sitemap and why do we need it? A sitemap is a file that lists the pages, videos, and other files on your website, as well as their relationships. Search engines (such as Google, Duckduckgo, Bing, and others) use this file to help them crawl your site more efficiently. For more information on sitemaps click here.
Generating static sitemap: If the website you are working on has a fixed number of URLs, for example, your portfolio. You can generate a sitemap.xml file with the help of any online tool and place the folder in the src folder.
Step 1: Visit https://www.xml-sitemaps.com/ and generate the sitemap.xml file.
Step 2: Move the file into the public folder of your react app.
Step 3: Verify the changes by visiting https://ahampriyanshu.com/sitemap.xml.
Step 4: Finally, Add the Sitemap to your robots.txt file
User-agent: * Allow: / Sitemap: https://baseurl/sitemap.xml
Generating dynamic sitemap: So far we have discussed creating a sitemap with static URLs. But what if the number of URLs and the content of existing URLs change from time to time. Suppose we are creating a GFG clone. So, our sitemap should contain URLs of all the articles and the important pages.
For this, we will send a sitemap file from our backend by first looping through all the required records from our database. After this, we will manually add the URLs of other important pages like about, contact, etc.
Step 1: For demonstration purposes, We are considering a basic project with three files.
app.js
Javascript
const express = require("express"), mongoose = require("mongoose"), todoRouter = require("./routers/todoRouter"), app = express();app.use("/", todoRouter);const port = 3000,mongoose .connect(db) .then(conn => { console.log(`${db} connected`); });app.listen(port, () => console.log( `Server listening on ${port}`)); |
model.js
Javascript
const mongoose = require("mongoose"), todoSchema = new mongoose.Schema( { title: { type: String, unique: true }, }, { timestamps: true } );module.exports = mongoose.model("Todo", todoSchema); |
todoRouter.js
Javascript
const express = require("express"), router = express.Router();/* Todo Controller functions */module.exports = router; |
Step 2: Install the’ sitemap’ package to stream the sitemap buffer and write its data.
npm i sitemap
Step 3: Create the sitemapRouter.js file in your router’s directory. At this point, your folder structure might look like
Step 4: Import all the required dependencies.
Javascript
const express = require("express"), { SitemapStream, streamToPromise } = require('sitemap'), Project = require("../models/projectModel"), date = new Date().toISOString(), zlib = require("zlib"), router = express.Router(); |
Step 5: Declare an object for caching purposes and set header content for the given request.
Javascript
let sitemap;router.get('/', async function (req, res) { res.header('Content-Type', 'application/xml'); res.header('Content-Encoding', 'gzip'); // If we have a cached entry send it if (sitemap) return res.send(sitemap)});module.exports = router; |
Step 6: Now we will write all the required URLs to our sitemap.
Javascript
try { // Fetching project records and mapping it // the desired URL pattern const data = await Project.find(), projects = data.map(({ title }) => `/todo/${title}`), // Base url of our site smStream = new SitemapStream( pipeline = smStream.pipe(zlib.createGzip()); // Write project URL to the stream projects.forEach( item => smStream.write({ url: item, lastmod: date, changefreq: 'daily', priority: 0.7 })); // Manually add all the other important URLs smStream.write({ url: '/about', lastmod: date, changefreq: 'monthly', priority: 0.9 }) smStream.write({ url: '/contact', lastmod: date, changefreq: 'monthly', priority: 0.9 }) // Cache the response streamToPromise(pipeline).then(sm => sitemap = sm); smStream.end() // Stream write the response pipeline.pipe(res).on('error', e => { throw e });} catch (err) { console.error(err) res.status(500).end()} |
Step 7: In the end, sitemapRouter.js should look like
Javascript
const express = require("express"), { SitemapStream, streamToPromise } = require('sitemap'), Todo = require("../models/todoModel"), date = new Date().toISOString(), zlib = require("zlib"), router = express.Router();let sitemap;router.get('/', async function (req, res) { res.header('Content-Type', 'application/xml'); res.header('Content-Encoding', 'gzip'); // If we have a cached entry send it if (sitemap) return res.send(sitemap) try { // Fetching todo records and mapping // it the desired URL pattern const data = await Todo.find(), todos = data.map(({ title }) => `/todo/${title}`), // Base url of our site smStream = new SitemapStream({ pipeline = smStream.pipe(zlib.createGzip()); // Write todo URL to the stream todos.forEach( item => smStream.write({ url: item, lastmod: date, changefreq: 'daily', priority: 0.7 })); // Manually add all the other important URLs smStream.write({ url: '/about', lastmod: date, changefreq: 'monthly', priority: 0.9 }) smStream.write({ url: '/contact', lastmod: date, changefreq: 'monthly', priority: 0.9 }) // Cache the response streamToPromise(pipeline).then(sm => sitemap = sm); smStream.end() // Stream write the response pipeline.pipe(res).on('error', e => { throw e }); } catch (err) { console.error(err) res.status(500).end() }});module.exports = router; |
Step 8: Verify the changes by visiting {basename}/sitemap.xml.
Note: If experiencing any difficulties/errors in following the above steps, please have a look at the sitemapRouter.js
Step 9: Finally, Add the Sitemap to your robots.txt file.
User-agent: * Allow: / Sitemap: https://baseurl/sitemap.xml
