IndexNow and Sitemap Automation for Faster Indexing

By Yanni Papoutsis | 9 min read | 2026-07-06

TL;DR

IndexNow lets you notify Bing, Yandex, and participating search engines about new or updated URLs instantly -- no waiting for crawlers to discover them. Combined with automated sitemap.xml generation on every deploy, this is the fastest way to get new content indexed. This guide covers generating an IndexNow key, submitting URLs, automating submission via CI/CD, generating sitemaps in Astro and manually, and verifying everything in Google Search Console.

Prerequisites

A website deployed and publicly accessible (see the Cloudflare Pages or Vercel guides)
Command line access and Node.js 18+ for automation scripts
A Google Search Console account with your site verified

What Is IndexNow?

IndexNow is an open protocol supported by Bing, Yandex, Naver, Yep, and others. When you publish or update a URL, you send a single HTTP request to an IndexNow endpoint and the engines are notified within minutes rather than days.

Google has not yet adopted IndexNow, but submitting to Bing's endpoint notifies all participating engines simultaneously. Google indexing still relies on sitemaps and organic crawling -- the sitemap automation section of this guide addresses that.

IndexNow vs sitemaps: IndexNow is real-time but limited to engines that support it. Sitemaps are supported by all engines but are crawled on their own schedule (typically every few days to weeks). Use both.

Step 1: Generate Your IndexNow Key

IndexNow authenticates you by requiring a file at a specific path on your domain containing your API key.

Generate a key:

The key must be a string of 8 to 128 hexadecimal characters. Generate one with Node.js:

node -e "const { randomBytes } = require('crypto'); console.log(randomBytes(32).toString('hex'));"

Example output (use your own generated value, never this one):

a9f3c8d2e1b047f6a3c9d8e2f1b04785a9f3c8d2e1b047f6a3c9d8e2f1b0478

Create the key file:

Create a file named exactly <your-key>.txt in the root of your website's public directory. For example, if your key is a9f3c8d2, the file should be at:

public/a9f3c8d2e1b047f6a3c9d8e2f1b04785a9f3c8d2e1b047f6a3c9d8e2f1b0478.txt

The file content should contain only the key itself:

a9f3c8d2e1b047f6a3c9d8e2f1b04785a9f3c8d2e1b047f6a3c9d8e2f1b0478

Deploy your site and verify the file is accessible at:

https://yourdomain.com/a9f3c8d2e1b047f6a3c9d8e2f1b04785a9f3c8d2e1b047f6a3c9d8e2f1b0478.txt

Step 2: Submit URLs to IndexNow

Once your key file is live, you can submit URLs.

Single URL submission (GET request):

https://api.indexnow.org/indexnow?url=https://yourdomain.com/blog/my-new-post&key=a9f3c8d2e1b047f6a3c9d8e2f1b04785a9f3c8d2e1b047f6a3c9d8e2f1b0478

You can test this directly in a browser or with curl:

curl "https://api.indexnow.org/indexnow?url=https://yourdomain.com/blog/my-new-post&key=YOUR_KEY_HERE"

A 200 response means the URL was accepted. A 422 means the URL or key is invalid.

Bulk URL submission (POST request):

For submitting multiple URLs at once, use the JSON endpoint:

curl -X POST "https://api.indexnow.org/indexnow" \
  -H "Content-Type: application/json; charset=utf-8" \
  -d '{
    "host": "yourdomain.com",
    "key": "YOUR_KEY_HERE",
    "keyLocation": "https://yourdomain.com/YOUR_KEY_HERE.txt",
    "urlList": [
      "https://yourdomain.com/blog/post-one",
      "https://yourdomain.com/blog/post-two",
      "https://yourdomain.com/about"
    ]
  }'

The urlList array accepts up to 10,000 URLs per request. All URLs must share the same host as specified in the host field.

Step 3: Node.js IndexNow Submission Script

Create a reusable script at scripts/submit-indexnow.mjs:

import { readFileSync } from 'fs'
import { join } from 'path'

const INDEXNOW_KEY = process.env.INDEXNOW_KEY
const SITE_URL = process.env.SITE_URL || 'https://yourdomain.com'

if (!INDEXNOW_KEY) {
  console.error('INDEXNOW_KEY environment variable is required')
  process.exit(1)
}

// Read URLs from a file or define them inline
// Example: read from a generated list of published pages
const urls = process.argv.slice(2).length > 0
  ? process.argv.slice(2)
  : [`${SITE_URL}/`]

async function submitToIndexNow(urlList) {
  const payload = {
    host: new URL(SITE_URL).hostname,
    key: INDEXNOW_KEY,
    keyLocation: `${SITE_URL}/${INDEXNOW_KEY}.txt`,
    urlList,
  }

  const response = await fetch('https://api.indexnow.org/indexnow', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json; charset=utf-8' },
    body: JSON.stringify(payload),
  })

  if (response.ok) {
    console.log(`IndexNow: submitted ${urlList.length} URL(s). Status: ${response.status}`)
  } else {
    const text = await response.text()
    console.error(`IndexNow error ${response.status}: ${text}`)
    process.exit(1)
  }
}

submitToIndexNow(urls)

Usage:

# Submit specific URLs
INDEXNOW_KEY=your-key SITE_URL=https://yourdomain.com node scripts/submit-indexnow.mjs \
  https://yourdomain.com/blog/new-post \
  https://yourdomain.com/blog/updated-post

# Or set environment variables once
export INDEXNOW_KEY=your-key
export SITE_URL=https://yourdomain.com
node scripts/submit-indexnow.mjs https://yourdomain.com/blog/new-post

Step 4: Automate with CI/CD

Integrate IndexNow submission into your GitHub Actions workflow so new pages are submitted automatically on every production deploy.

# .github/workflows/deploy-and-index.yml
name: Deploy and Submit to IndexNow

on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    outputs:
      new-urls: ${{ steps.detect-changes.outputs.urls }}
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 2  # Need previous commit to detect changes

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '18'

      - name: Install and build
        run: |
          npm ci
          npm run build

      - name: Deploy to Cloudflare Pages
        uses: cloudflare/pages-action@v1
        with:
          apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }}
          accountId: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
          projectName: your-project-name
          directory: dist
          branch: main

      - name: Submit new/changed URLs to IndexNow
        env:
          INDEXNOW_KEY: ${{ secrets.INDEXNOW_KEY }}
          SITE_URL: https://yourdomain.com
        run: |
          # Get list of changed content files
          CHANGED=$(git diff --name-only HEAD~1 HEAD -- 'src/content/**' 'src/pages/**' | head -20)
          if [ -z "$CHANGED" ]; then
            echo "No content changes detected, skipping IndexNow submission"
            exit 0
          fi
          # Convert file paths to URLs (adjust pattern for your framework)
          URLS=$(echo "$CHANGED" | sed 's|src/content/blog/||' | sed 's|\.md$||' | sed "s|^|${SITE_URL}/blog/|")
          echo "Submitting URLs: $URLS"
          node scripts/submit-indexnow.mjs $URLS

Store your INDEXNOW_KEY as a GitHub repository secret. Never hardcode it in workflow files.

Step 5: Generate sitemap.xml

Astro (Recommended)

Astro has an official sitemap integration that generates sitemap.xml automatically on every build.

Install the integration:

npm install @astrojs/sitemap

Configure in astro.config.mjs:

import { defineConfig } from 'astro/config'
import sitemap from '@astrojs/sitemap'

export default defineConfig({
  site: 'https://yourdomain.com',
  integrations: [
    sitemap({
      // Optional: exclude specific pages
      filter: (page) => !page.includes('/admin/'),
      // Optional: add custom change frequency and priority
      customPages: ['https://yourdomain.com/special-page'],
      changefreq: 'weekly',
      priority: 0.7,
      lastmod: new Date(),
    }),
  ],
})

After building (npm run build), Astro generates dist/sitemap-index.xml and dist/sitemap-0.xml automatically.

Manual Sitemap Generation (Node.js)

For non-Astro projects, use this script at scripts/generate-sitemap.mjs:

import { writeFileSync, readdirSync, statSync } from 'fs'
import { join } from 'path'

const SITE_URL = process.env.SITE_URL || 'https://yourdomain.com'
const DIST_DIR = './dist'
const OUTPUT_FILE = join(DIST_DIR, 'sitemap.xml')

function getHTMLFiles(dir, base = '') {
  const urls = []
  const entries = readdirSync(dir)

  for (const entry of entries) {
    const fullPath = join(dir, entry)
    const stat = statSync(fullPath)
    const urlPath = `${base}/${entry}`

    if (stat.isDirectory()) {
      urls.push(...getHTMLFiles(fullPath, urlPath))
    } else if (entry === 'index.html') {
      const cleanPath = base === '' ? '/' : base
      urls.push({
        url: `${SITE_URL}${cleanPath}`,
        lastmod: stat.mtime.toISOString().split('T')[0],
      })
    }
  }

  return urls
}

const pages = getHTMLFiles(DIST_DIR)

const sitemap = `<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
${pages.map(({ url, lastmod }) => `  <url>
    <loc>${url}</loc>
    <lastmod>${lastmod}</lastmod>
    <changefreq>weekly</changefreq>
    <priority>0.8</priority>
  </url>`).join('\n')}
</urlset>`

writeFileSync(OUTPUT_FILE, sitemap)
console.log(`Sitemap generated: ${pages.length} URLs written to ${OUTPUT_FILE}`)

Run after your build command:

npm run build && node scripts/generate-sitemap.mjs

Add it to your package.json scripts:

{
  "scripts": {
    "build": "vite build",
    "postbuild": "node scripts/generate-sitemap.mjs"
  }
}

The postbuild hook runs automatically after build completes.

Step 6: Submit Sitemap to Google Search Console

Go to search.google.com/search-console.
Select your property (or add it and verify ownership).
In the left sidebar, go to Sitemaps.
In the Add a new sitemap field, enter the path to your sitemap: sitemap.xml Or sitemap-index.xml for Astro's generated index.
Click Submit.

Google will show the sitemap status -- typically Pending initially, then Success once crawled. The Discovered URLs count confirms how many pages Google found.

Automate sitemap submission via the Search Console API (advanced):

The Google Search Console API allows programmatic sitemap submission, but it requires OAuth 2.0 authentication which is complex to set up. For most sites, manual submission once is sufficient -- Google re-crawls submitted sitemaps on its own schedule after the initial submission.

Step 7: Verify in Search Console

After submitting your sitemap and running your first IndexNow submission:

In Search Console, go to URL Inspection.
Paste one of the URLs you submitted via IndexNow.
Click Request Indexing as a manual fallback for Google.
Check the Coverage report after 24-48 hours to see indexed page counts.

For IndexNow specifically, Bing Webmaster Tools (at bing.com/webmasters) shows an IndexNow section with submission history and response codes.

Troubleshooting

IndexNow returns 403 Forbidden The key file at yourdomain.com/<key>.txt is returning 403 or 404. Verify the file is deployed and publicly accessible. Some frameworks require the file to be in a specific public directory.

IndexNow returns 422 Unprocessable Entity The URL or key format is invalid. Ensure your URLs start with https://, match the host field exactly, and your key is exactly the string in the key file (no trailing newline).

Sitemap not showing in Search Console Wait at least 48 hours after submission. If it still shows an error, check that the sitemap URL is publicly accessible and returns valid XML with Content-Type: application/xml.

Google still not indexing new pages Google does not use IndexNow. Ensure your sitemap is submitted in Search Console, use the Request Indexing feature in URL Inspection for priority pages, and build internal links to new pages so Googlebot can discover them via crawling.

FAQ

Does IndexNow work with Google? Not currently. IndexNow is supported by Bing, Yandex, Naver, Yep, and others. Google has stated they are evaluating the protocol but has not adopted it.

How many URLs can I submit per day? IndexNow has no published daily limit, but the documentation recommends submitting only new or changed URLs, not your entire sitemap repeatedly. Submitting unchanged URLs adds no value and may result in rate limiting.

Does Cloudflare Pages support a sitemap.xml file? Yes. Any file in your build output directory is served as a static file. Place sitemap.xml in your output directory (dist/, public/, _site/, etc.) and it will be accessible at yourdomain.com/sitemap.xml. See the Cloudflare Pages deployment guide for details.

Can I use IndexNow with a dynamically rendered site? Yes. IndexNow is a server-side API call that you can trigger from any backend, CMS webhook, or CI/CD pipeline. It does not require static files beyond the key verification file.

Should I submit the sitemap URL or individual URLs to IndexNow? Individual URLs. IndexNow does not understand sitemap format -- it expects an array of specific page URLs. Your sitemap is for search engine crawlers; IndexNow is for real-time notification.

For AI tools to help you produce content faster so you have more to submit and index, explore 1000+ free AI tools. Once your site is fully indexed, review the Claude plugin guide for adding AI capabilities to your workflow.