The Awkward Truth About Our Old Website

Here's an honest admission: until recently, auspexi.com was a single-page React app built with Vite. It looked good, it worked well as a product, and our dashboard was solid. But from the perspective of AI crawlers and search engines, it had a fundamental problem.

Single-page apps render in the browser using JavaScript. When a crawler — whether that's Googlebot, GPTBot, or ClaudeBot — visits a JavaScript-rendered page, they often see a near-empty HTML shell. The actual content is invisible to them because they don't run JavaScript the same way a browser does. The result: all our blog posts, all our product copy, all the detailed content that should be helping us rank and get cited — none of it was being indexed properly.

This is embarrassing for a company whose entire product is about helping other brands get cited by AI. We were preaching structured data and crawlability while running a site that was essentially opaque to the crawlers we were optimizing for.

So we rebuilt everything on Next.js, and we want to explain why that decision matters — not just for us, but as an illustration of the kind of infrastructure decisions that affect AI visibility for any business.

What Next.js Actually Changes

Next.js uses server-side rendering (SSR) and static site generation (SSG). When a crawler visits any page on the new auspexi.com, the server sends back fully rendered HTML — no JavaScript execution required, no waiting for hydration, no content missing. The crawler sees exactly what a browser would see.

That matters for two reasons.

First, indexability. Every single page on the site now has real, crawlable content. Our 31 blog posts are all individually indexed. The product pages, the about page, the pricing section — all of it exists as real HTML that search engines and AI crawlers can read and process. We went from effectively zero indexable content to a fully structured, crawlable site overnight.

Second, speed. Server-rendered pages load faster for humans too, which is a ranking signal for traditional search engines. And for AI crawlers that have rate limits and time constraints, a fast page means more of your content gets processed per visit.

The AI Crawler Permissions Layer

One of the first things we did after the migration was create a proper robots.txt that explicitly allows the crawlers that matter for GEO. By default, many platforms set conservative robots.txt rules or leave them unspecified. That's fine for preventing spam crawlers, but it can accidentally block the AI crawlers you actually want.

Our robots.txt now explicitly allows: GPTBot (ChatGPT), ChatGPT-User, anthropic-ai, ClaudeBot, PerplexityBot, and Google-Extended (Gemini's web data pipeline). These are the crawlers that feed into the AI engines your customers are using to find recommendations. Blocking them — even accidentally — is an own goal.

We also block the right things: /dashboard/, /api/, and the preview endpoints that shouldn't be publicly indexed. The principle is simple: be maximally open to AI crawlers for public content, and strict about protecting private routes.

A Sitemap That Covers Everything

The new site generates a dynamic sitemap at /sitemap.xml that covers every public page: the homepage, all product pages, the about page, the FAQ, and all 31 blog posts. Each entry has a last-modified date and a priority score (1.0 for the homepage, 0.9 for the blog index, 0.8 for individual posts).

The sitemap is automatically updated when we publish new blog posts — it's generated from the same data file that renders the blog, so there's no manual maintenance or risk of forgetting to update it.

Once we verify ownership in Google Search Console (waiting on DNS propagation as of this writing), we'll submit the sitemap and track indexation. More importantly, AI crawlers that support sitemap parsing will use it to discover content they might not find through standard link-following — especially important for a relatively new domain.

What This Means in Practice

For people building on similar stacks, the key lessons from our migration:

If you're on a JavaScript-only SPA, your AI visibility is probably lower than your traditional SEO metrics suggest. Even if you rank well in Google (which has gotten better at rendering JavaScript), the AI crawlers that feed Perplexity, ChatGPT, and Claude are more conservative. Server rendering is the safest path.

Robots.txt is not just about blocking bad actors. It's about actively signalling to valuable crawlers that your content is available and welcome. Explicitly listing the AI crawlers you want to allow sends a clear signal.

A sitemap is table stakes. If you're not submitting one, you're relying entirely on the crawler to discover your pages through link-following. For a site with dozens of blog posts, that's a slow and incomplete process.

We're Building on What We Preach

The core of Auspexi's value proposition is that your brand's digital presence should be structured, machine-readable, and explicitly optimised for AI engines. The infrastructure changes we've made to our own site are a direct application of that philosophy.

We moved to Next.js because it was the right call for our AI visibility, not because it was the trendy framework. The sitemap, the robots.txt permissions, the server rendering — these are exactly the kinds of technical GEO decisions we help our customers make. Now our own site is built on the same foundation.

If you want to understand whether your own site has similar crawlability issues, the Citation Probe in your dashboard is a good starting point. It'll tell you how you're currently being cited. And Citacious can walk you through the technical tab if you want to audit your schema and robots setup in more detail.