`, `

` tags). AI crawlers depend on it. Properly semantic HTML helps AI systems understand which content is primary, which is navigation, and which is sidebar information. **3. AI Crawlers Need Explicit Metadata** Google can infer article dates, author information, and content type from various signals. AI crawlers rely more heavily on explicit metadata in `` tags. A missing publication date in your markup makes it harder for LLMs to assess freshness. **4. AI Systems Extract and Synthesise Differently** Google's algorithm creates a ranking score. AI systems extract information. This means your content needs to be extractable as distinct, quotable, attributable pieces. Long paragraphs of prose are harder to extract than structured lists or clearly delimited sections. ## Crawlability and Indexation Before any of your content can be included in AI responses, AI crawlers need to be able to find and read your pages. **1. Robots.txt Configuration** Your `robots.txt` file should explicitly allow AI crawlers. Create rules for: ``` User-agent: GPTBot Allow: / User-agent: CCBot Allow: / User-agent: Perplexitybot Allow: / User-agent: Googlebot-Extended Allow: / User-agent: * Disallow: /admin/ Disallow: /api/ Disallow: /private/ ``` If you block specific crawlers, you're preventing your content from being included in their generated responses. For most businesses, allowing these crawlers is beneficial. **2. Sitemap Configuration** Maintain an updated XML sitemap with: - All important content pages - Latest publication date (`lastmod`) for each page - Content priority (`priority` tags) - Language information (`hreflang` tags for international sites) The sitemap should be less than 50MB and contain fewer than 50,000 URLs per file. For larger sites, use sitemap index files. **3. Canonical Tags** Every page should have a canonical tag pointing to itself (or to the canonical version if you have duplicate content). This helps crawlers identify the authoritative version of content. ```html ``` **4. Noindex and Nofollow Usage** Selectively use `noindex` for: - Thin content pages (tag pages, archive pages) - Duplicate content that shouldn't rank independently - Pages in testing or development But do NOT `noindex`: - Your primary content pages - Pillar pages and cluster content - Pages you want included in AI Overviews Use `nofollow` sparingly. For internal links, it's rarely necessary. Use it for: - Links to paid or sponsored content - Links to user-generated content you don't endorse - Links to low-quality external sites **5. Meta Robots Tag** Your `` should include: ```html ``` The snippet and preview settings tell crawlers they can use your content generously in responses. ## Semantic HTML and Structure AI crawlers understand content hierarchy through both semantic HTML tags and heading structure. Your content should use: **6. Proper Heading Hierarchy** - Every page has exactly one H1 tag (the main title) - H2 tags break the page into major sections - H3 tags break H2 sections into subsections - Headings should be descriptive and follow a logical hierarchy (no skipping from H2 to H4) **Bad hierarchy:** ```

Ultimate Guide to Demand Generation

What is Demand Gen?

Demand Gen Tools

Marketo

``` **Good hierarchy:** ```

Ultimate Guide to Demand Generation

What is Demand Generation?

Demand Gen vs Lead Gen

Demand Generation Tools

Marketo

Marketo Pricing

``` **7. Semantic HTML Tags** Use semantic tags to mark up content: ```html

Title

Meta information

Section Title

Content...

``` **8. List Markup for Structured Content** Use `

First step in the process
Second step
Third step

Feature	Platform A	Platform B
Price per seat	$50	$75

` for emphasis and `` for strong emphasis: ```html
Marketing automation is essential for modern B2B teams.
``` Don't use CSS to style text as bold or italic; use semantic tags. ## Schema Markup and Structured Data Schema.org markup helps crawlers understand what type of content you're publishing and what it contains. **11. Article Schema** Every long-form article should include Article schema in JSON-LD format: **12. FAQPage Schema** If your article includes an FAQ section, use FAQPage schema: **13. BreadcrumbList Schema** Help AI systems understand your site structure: **14. HowTo Schema** For instructional content: **15. Organization Schema** Your homepage should include Organization schema: **16. Author Schema** Create dedicated author pages with author schema: ## Metadata and Open Graph Tags Your `` should contain explicit metadata that tells AI crawlers about your content. **17. Title and Meta Description** Every page needs: ```html Title | Your Company ``` The title should be SEO-optimised. The description should be a natural summary (not keyword-stuffed). **18. Open Graph Tags** Help AI systems understand what your content is about: ```html ``` **19. Twitter Card Tags** For consistency on social platforms: ```html ``` **20. Language and Charset Declaration** Every page should have: ```html ``` For international sites, use `hreflang` tags to indicate language versions. ## Page Performance and Core Web Vitals AI crawlers assess content quality partially through page performance. Slow pages signal lower quality to ranking and crawling systems. **21. Core Web Vitals** Optimise for Google's three Core Web Vitals: - **Largest Contentful Paint (LCP)**: First major content should load in < 2.5 seconds - **Cumulative Layout Shift (CLS)**: Page shouldn't shift unexpectedly as it loads; target < 0.1 - **First Input Delay (FID)**: Page should respond to input within 100ms (being replaced by Interaction to Next Paint in 2024) **22. Page Speed Optimisation** - Compress images (use modern formats like WebP) - Minify CSS and JavaScript - Implement lazy loading for images and video - Use a CDN to serve content from locations near users - Remove render-blocking resources - Optimise third-party scripts (analytics, ads) **23. Mobile Responsiveness** AI crawlers access your site from various devices and viewport sizes. Ensure: - Your site is fully responsive to all screen sizes - Text is readable without zooming - Buttons and links are touch-friendly (48px minimum) - Navigation works on mobile Test with Google's Mobile-Friendly Test and across various real devices. ## Content Accessibility and Readability AI crawlers benefit from content that follows accessibility standards (WCAG 2.1 Level AA). **24. Image Alt Text** Every image should have descriptive alt text: ```html ``` Alt text helps AI crawlers understand what's in images and serves blind and low-vision users. **25. Color Contrast** Text should have sufficient contrast against backgrounds (4.5:1 ratio for normal text, 3:1 for large text). **26. Focus Indicators** Interactive elements should have visible focus states for keyboard navigation. **27. Language and Readability Metrics** AI crawlers can assess readability. Use: - Clear sentence structure - Short paragraphs (3-5 sentences) - Active voice where possible - Jargon defined when first introduced Aim for 8th-9th grade reading level for B2B content (high school education level). Tools like Hemingway App and Readability Statistics can help. ## AI-Specific Technical Requirements **28. X-Robots-Tag HTTP Header** In addition to meta robots tags, you can use HTTP headers: ``` X-Robots-Tag: index, follow, max-snippet:-1, max-image-preview:large ``` **29. Allow AI Crawlers in Robots.txt** Explicitly allow major AI crawlers: - `GPTBot` (OpenAI) - `CCBot` (Common Crawl) - `Perplexitybot` (Perplexity) - `Googlebot-Extended` (Google's generative AI crawling) - `ClaudeBot` (Anthropic) **30. Monitor Crawl Patterns** Use Google Search Console to monitor: - Which crawlers are accessing your site - How frequently they crawl - Any crawl errors - Coverage issues (pages that aren't being indexed) Look for `GoogleExtended` and `GPTBot` requests in your server logs. **31. Content Freshness** Include publication and modification dates in: - Article schema JSON-LD - Your `` meta tags - Visibly on the page (helps both users and crawlers) Update modification dates when you revise content significantly. ## Audit and Implementation Checklist ### Crawlability (Priority: Critical) - [ ] Robots.txt allows GPTBot, Perplexitybot, Googlebot-Extended - [ ] XML sitemap exists and is updated monthly - [ ] Canonical tags on all content pages - [ ] No critical crawl errors in Search Console - [ ] Important content is not `noindex` ### Structure (Priority: Critical) - [ ] Every page has exactly one H1 - [ ] Headings follow logical hierarchy (H1 → H2 → H3) - [ ] Content uses semantic HTML (`
`, `
`, `
`) - [ ] Lists are marked up with `
` or `
` - [ ] Tables use `` with `` and `` ### Schema Markup (Priority: High) - [ ] Article schema on all long-form content - [ ] FAQPage schema if you have Q&A sections - [ ] Organization schema on homepage - [ ] Author schema on author pages - [ ] BreadcrumbList schema for site navigation - [ ] Validate all schema with Google's Structured Data Testing Tool ### Metadata (Priority: High) - [ ] Every page has unique title tag (50-60 characters) - [ ] Every page has meta description (150-160 characters) - [ ] Open Graph tags (og:title, og:description, og:image, og:url) - [ ] Publication date visible and in schema - [ ] Last modified date updated when content changes ### Performance (Priority: High) - [ ] LCP < 2.5 seconds - [ ] CLS < 0.1 - [ ] Mobile-Friendly Test passes - [ ] Images optimised and lazy-loaded - [ ] CSS/JS minified ### Accessibility (Priority: Medium) - [ ] All images have alt text - [ ] Text contrast ratio ≥ 4.5:1 - [ ] Interactive elements are keyboard navigable - [ ] Reading level appropriate (8th-9th grade for B2B) ### Content Quality (Priority: High) - [ ] No duplicate content without canonical tags - [ ] Thin content is `noindex`d or consolidated - [ ] Promotional content clearly marked - [ ] Citations for all non-obvious claims - [ ] Author credentials included ## Frequently Asked Questions ### Do I need to do all of this for every page? Priority is 1) your top 50 content pages (your money pages and pillar content), 2) your homepage and main category pages, 3) your blog archive. Secondary pages can follow best practices gradually. ### Will fixing these technical elements improve my search ranking? Probably, but these changes primarily affect AI visibility. Some elements (Core Web Vitals, mobile-friendliness) do impact Google ranking. Others (semantic HTML, schema markup) affect AI inclusion more directly than Google ranking. ### What's the fastest way to implement this checklist? Start with schema markup. Most of your existing content already has proper heading hierarchy and semantic HTML (or can be fixed with search-and-replace). Adding Article schema, FAQPage schema, and Organization schema takes 2-3 hours per 100 pages. ### Do I need to block Google's AI crawler (Googlebot-Extended)? Only if you explicitly don't want your content included in Google's AI Overviews. For most businesses, allowing it is beneficial. ### How often should I update my sitemap? For blogs and frequently updated content, daily or weekly. For static content, monthly is fine. Ensure your CMS automatically updates the sitemap when you publish new pages. ### Will this affect my existing Google rankings? Mostly not. These changes are neutral to positive for Google ranking. The only potential negative would be if you break heading hierarchy or introduce duplicate content issues. --- *Published by Fortitude Media. [Book a free AI visibility audit](https://fortitudemedia.com) to see what AI tools currently say about your business.*