<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[313 Automations Blog]]></title><description><![CDATA[313 Automations Blog]]></description><link>https://313automations-blog.hashnode.dev</link><generator>RSS for Node</generator><lastBuildDate>Mon, 02 Nov 2026 14:54:53 GMT</lastBuildDate><atom:link href="https://313automations-blog.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[How AI crawlers read your robots.txt (and why your * rules may not apply)]]></title><description><![CDATA[Most robots.txt files were written for Googlebot and Bingbot. Since then, AI companies have added several new crawlers, and they do different jobs. If you…]]></description><link>https://313automations-blog.hashnode.dev/how-ai-crawlers-read-your-robots-txt-and-why-your-rules-may-not-apply</link><guid isPermaLink="true">https://313automations-blog.hashnode.dev/how-ai-crawlers-read-your-robots-txt-and-why-your-rules-may-not-apply</guid><category><![CDATA[AI]]></category><category><![CDATA[SEO]]></category><dc:creator><![CDATA[Ghulam Mustafa]]></dc:creator><pubDate>Thu, 29 Oct 2026 16:57:52 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6ae048a20130645d01b34f42/4a2199f5-6916-4f92-b8d2-43a453e9462b.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Most robots.txt files were written for Googlebot and Bingbot. Since then, AI companies have added several new crawlers, and they do different jobs. If you have never looked at how your file treats them, you may be blocking AI search by accident, or letting training crawlers in without ever deciding to.</p>
<p>This post covers three things that trip people up: which AI bots exist, how a crawler picks the rules it follows, and why a well-meant <code>User-agent: *</code> block often does nothing for them.</p>
<h2>Three kinds of AI bots</h2>
<p>It helps to sort AI user agents by what they do, not by company:</p>
<ul>
<li><strong>Answer engines / AI search</strong> fetch pages so an assistant can cite them in answers. Examples: <code>OAI-SearchBot</code>, <code>Claude-SearchBot</code>, <code>PerplexityBot</code>.</li>
<li><strong>On-request fetchers</strong> load a page because a person asked the assistant about it. Example: <code>ChatGPT-User</code>.</li>
<li><strong>Training crawlers</strong> collect content to train models. Examples: <code>GPTBot</code>, <code>ClaudeBot</code>, <code>CCBot</code> (Common Crawl).</li>
</ul>
<p>Many site owners want the first two (they can send visitors) and want to opt out of the third. That is a business decision, but you can only make it if your rules name the right bots. Vendors also change how their bots behave, so check each one's own documentation.</p>
<h2>How a crawler picks its rules</h2>
<p>RFC 9309 (the robots.txt standard) and Google's documentation describe the same basic logic:</p>
<ol>
<li>The crawler looks for the group whose <code>User-agent</code> line matches its own name most specifically.</li>
<li>If it finds one, it follows <strong>only that group</strong>. The <code>*</code> group is a fallback for crawlers with no group of their own.</li>
<li>Inside the group, the <strong>longest matching path</strong> wins. If an <code>Allow</code> and a <code>Disallow</code> rule match with the same length, <code>Allow</code> wins.</li>
</ol>
<p>Point 2 is the one that catches people out.</p>
<h2>Why your * rules may not apply</h2>
<p>Take this file:</p>
<pre><code>User-agent: *
Disallow: /admin/
Disallow: /cart/

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /
</code></pre>
<p>It looks safe, but <code>OAI-SearchBot</code> now has its own group, so it ignores the <code>*</code> group completely. Your <code>/admin/</code> and <code>/cart/</code> rules no longer apply to it. The fix is to repeat the shared rules inside every named group:</p>
<pre><code>User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: ChatGPT-User
Allow: /
Disallow: /admin/
Disallow: /cart/

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /

User-agent: *
Disallow: /admin/
Disallow: /cart/
</code></pre>
<p>Several <code>User-agent</code> lines can share one group, which keeps the file short.</p>
<h2>Two honest limits</h2>
<ul>
<li><strong>robots.txt is a request, not a lock.</strong> Well-behaved bots follow it; others may not. If you need to enforce a block, do it at your host, firewall or CDN.</li>
<li><strong>Your CDN may already decide for you.</strong> Some CDNs and security plugins block AI crawlers by default. A perfect robots.txt does not help if the request never reaches it, so check your server or CDN logs.</li>
</ul>
<h2>Checking your own file</h2>
<p>You can test this by hand against each user agent, but it gets tedious with twenty-plus bots. We built a free checker that does it for 22 AI crawlers, for a live site or a pasted draft file, on any path. It also looks for an <code>llms.txt</code> file and writes a block like the one above that keeps your existing <code>*</code> rules (I run 313 Automations):</p>
<p><a href="https://313automations.com/free-tools/ai-crawler-access-checker">https://313automations.com/free-tools/ai-crawler-access-checker</a></p>
]]></content:encoded></item><item><title><![CDATA[Does Schema Markup Get You Cited by AI? What 730 Citations Show]]></title><description><![CDATA[Adding schema markup will not, by itself, get your pages cited by ChatGPT, Gemini or AI Overviews. That is the finding of the first serious attempt to test…]]></description><link>https://313automations-blog.hashnode.dev/does-schema-markup-get-you-cited-by-ai-what-730-citations-show</link><guid isPermaLink="true">https://313automations-blog.hashnode.dev/does-schema-markup-get-you-cited-by-ai-what-730-citations-show</guid><category><![CDATA[SEO]]></category><category><![CDATA[structured data]]></category><category><![CDATA[AI]]></category><dc:creator><![CDATA[Ghulam Mustafa]]></dc:creator><pubDate>Tue, 27 Oct 2026 20:13:51 GMT</pubDate><content:encoded><![CDATA[<p>Adding <strong>schema markup</strong> will not, by itself, get your pages cited by ChatGPT, Gemini or AI Overviews. That is the finding of the first serious attempt to test the claim: a cross-platform study published in February 2026 collected 730 AI citations across 75 commercial queries and found schema present on 43.1% of cited pages and 44.8% of pages that were never cited. The markup simply did not separate the winners from everyone else.</p>
<p>This matters because "add schema for AI" has become the most repeated piece of GEO advice on the internet, and a lot of budget is moving on the strength of it. Below is what the evidence actually shows, what structured data genuinely does for you, which types are still worth the effort, and what predicts citation instead. (Disclosure: I run 313 Automations, an AI automation and SEO agency.)</p>
<h2>TL;DR</h2>
<ul>
<li><strong>The finding:</strong> schema appeared on 43.1% of AI-cited pages and 44.8% of uncited ones (<a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=6284518">SSRN study</a>)</li>
<li><strong>The exception:</strong> attribute-rich Product and Review markup was cited at 61.7% against 41.6% for generic markup</li>
<li><strong>The real driver:</strong> organic rank position, with each place down cutting citation odds by about a quarter</li>
<li><strong>Still worth doing:</strong> rich results, entity identity, authorship and dates, unambiguous facts</li>
</ul>
<h2>What the evidence actually says</h2>
<p>The study worth knowing about ran 75 commercial queries across five categories through ChatGPT with browsing and Gemini with search grounding, captured the 730 pages those engines cited, and compared them against the Google top ten for the same queries as a control, 1,006 unique pages in total. Every page was checked for JSON-LD schema and for domain authority.</p>
<p>The first pass suggested schema was actively associated with fewer citations, which sounded dramatic; once the control set was corrected and errors clustered by query, the effect collapsed to nothing at all. That is the more useful result. Schema presence is so close to universal on commercial pages that it has almost no discriminating power. Everyone has it, so it cannot be what separates you.</p>
<p>One study is one study, and this one is a preprint testing correlation rather than a controlled experiment. But it lines up with what live testing keeps showing: when an answer engine fetches your page in real time, it largely reads the visible text. It is not hunting through your JSON-LD for a summary; it is lifting the sentence that answers the question.</p>
<h2>Why attribute-rich markup is the exception</h2>
<p>The one place structured data did move the numbers is instructive. Product and Review markup carrying populated pricing, ratings and specifications was cited on 61.7% of pages, against 41.6% for generic implementations. The likely reason is not that engines love schema; it is that those fields carry facts the prose often leaves out or buries. A price, a rating out of five, a release date and a set of specs are exactly the atoms an answer engine needs to build a comparison, and a page that states them unambiguously is a cheaper source than one where the model has to infer them. Empty markup that only says "this is an Article" adds no facts, so it adds no advantage.</p>
<p>The practical translation: schema earns its keep when it publishes data, not when it labels a template. If your Product markup has no price in it, you have written a comment, not a signal.</p>
<h2>So why keep doing schema at all?</h2>
<p>Because it still does four jobs nothing else does, and none of them are AI citations.</p>
<ul>
<li><strong>Rich results:</strong> star ratings, prices, FAQs where still supported, events, recipes and breadcrumbs come from markup, and they change how your listing looks in a results page that is already showing fewer of them</li>
<li><strong>Entity identity:</strong> Organization, Person and sameAs links tell every system that this business, this author and these profiles are the same entity, which is how you get treated as a known brand rather than an unknown URL</li>
<li><strong>Provenance:</strong> author and date fields tie content to real people and verifiable timestamps, which is increasingly what separates a source worth quoting from an anonymous page</li>
<li><strong>Clean extraction:</strong> prices, availability, opening hours and locations get read the same way by every parser, so nothing depends on a model guessing correctly from your prose</li>
</ul>
<p>Those are strong reasons. They are just not the reason it usually gets sold. In technical SEO work, schema is a hygiene layer you should expect to find working, in the same category as canonical tags and a valid sitemap, not a growth lever to bill as a strategy.</p>
<h2>What predicts citation instead</h2>
<p>Rank first. In the same study, organic position was the dominant predictor: pages at position 1 were cited in roughly 43% of queries against 5% at position 7, with each place down the page cutting the odds by about a quarter. Answer engines are, to a large extent, reading the same results you are.</p>
<p>Then comes recognisable authority, and the shape of that is visible in <a href="https://www.pewresearch.org/short-reads/2025/07/22/google-users-are-less-likely-to-click-on-links-when-an-ai-summary-appears-in-the-results/">Pew's data on Google AI summaries</a>, where Wikipedia, YouTube and Reddit alone supplied 15% of all listed sources. Engines lean on entities they already trust.</p>
<p>Third, extractability: a page that answers the question in its first two sentences, in plain declarative prose, gives a model something clean to quote, while a page that warms up for six paragraphs does not. Fourth, third-party mentions, because a claim repeated on sites the engine trusts is a claim it will repeat too.</p>
<p>That ordering is why GEO work that starts with markup usually disappoints, and why the same work starting with rankings, answer-first rewriting and earned mentions tends not to.</p>
<h2>A sensible schema plan for this quarter</h2>
<p>Spend an afternoon, not a retainer.</p>
<ol>
<li>Validate what you already have and fix the errors, because broken markup is worse than none.</li>
<li>Fill in the attributes on the types that carry facts, especially Product, Offer, Review, LocalBusiness and Event, and delete markup that declares a type and nothing else.</li>
<li>Get Organization and Person right once, site-wide, with accurate sameAs profiles.</li>
<li>Make sure every article has a real author and a real date.</li>
</ol>
<p>Then stop, and put the rest of the time into the things that actually move citation: better rankings on the queries you care about, answers written to be lifted, and being mentioned somewhere other than your own domain.</p>
<p><strong>Bottom line:</strong> schema markup is worth doing and is not a shortcut to AI citations, and the evidence now says so out loud. Publish real facts in your markup, get the fundamentals right, and treat any pitch that sells structured data as a generative-search growth strategy with the skepticism it has earned.</p>
<p><em>Originally published on the <a href="https://313automations.com/blog/schema-markup-ai-citations">313 Automations blog</a>.</em></p>
]]></content:encoded></item></channel></rss>