- Crawling and indexing
- Core Web Vitals and INP
- AI bot policy
Technical SEO · Crawling, speed and AI bots
Technical SEO audit: a site Google and AI read without friction
We run technical SEO audits that show how Google and AI crawlers crawl, render and index your site, then turn the findings into a prioritised fix plan your developers or ours can apply.
Personal reply, No obligation, English, Español
What it is
What a technical SEO audit is
A technical SEO audit checks whether search engines can find, crawl, render and index your site, and whether it loads quickly for real users. It surfaces what holds visibility back, from server errors and JavaScript-dependent content to conflicting canonicals and slow templates, and ranks it by impact.
You receive a plain-English report in four parts: an executive summary, issues prioritised by impact and effort, developer-ready specifications and a validation plan. Every point explains what is wrong, why it matters and how to fix it.
In 2026 your site has more readers than Googlebot: AI crawlers and agents browsing on users' behalf. So our technical SEO services cover their access too, as the foundation for our SEO and GEO work.
In depth
Crawling, rendering and indexing
If Google can't crawl a page or see its rendered content, it won't index it properly, and an unindexed page appears neither in classic results nor in AI Overviews or AI Mode. So we start here: which URLs Google discovers, processes and indexes, and why others fall away.
We cross-reference a full crawl with Search Console's indexing reports and server logs. For JavaScript SEO we compare initial and rendered HTML: if content or links only appear once scripts run, indexing becomes slow and unreliable. Common culprits:
- Key pages on noindex, blocked in robots.txt or without internal links.
- Canonicals that contradict the sitemap, links or redirects.
- Redirect chains, 4xx and 5xx errors, and URL-multiplying parameters.
- Filters and pagination wasting crawling on low-value URLs.
In depth
Core Web Vitals and INP: the speed that counts
Core Web Vitals measure your users' real experience against three thresholds: LCP (main content loading) at or under 2.5 seconds, INP (responsiveness to interactions) under 200 milliseconds and CLS (visual stability) under 0.1. INP replaced FID on 12 March 2024.
We work from field data in the Chrome UX Report and Search Console, not just a lab score, then trace the cause: heavy images or slow servers for LCP, third-party scripts for INP, elements without reserved space for CLS. Deeper fixes are coordinated with web development.
In depth
AI bots, CDNs and snippet controls
Controlling what AI does with your site starts by separating search bots from training bots. OAI-SearchBot (ChatGPT), Claude-SearchBot and PerplexityBot feed answers that can cite you; GPTBot and ClaudeBot collect training data. You can block training and stay visible in AI search.
robots.txt isn't enough: since 1 July 2025, Cloudflare has blocked known AI crawlers by default and asks new domains to choose. We also review CDN and firewall rules, and confirm in server logs that each bot gets the intended access.
Google-Extended governs Gemini's use of your content but affects neither Search nor AI Overviews; those are limited with nosnippet, data-nosnippet, max-snippet, noindex or Search Console's generative AI control, available worldwide as of 31 August 2026. llms.txt is optional: Google doesn't use it.
In depth
Structured data, AMP and semantic markup
Structured data still helps, but selectively. Google stopped showing FAQ rich results on 7 May 2026, HowTo rich results are gone, and in 2025 it retired others such as course info and estimated salary. Unused markup does no harm but justifies no new work.
We prioritise the types that still add value: Organization, LocalBusiness, Product and merchant listings, Article, Event and Video. AMP is no longer an advantage either: since July 2026 Google links straight to publishers' own AMP pages and ranks them like any other.
Semantics matter too: Google notes that browsing agents may read screenshots, the DOM and the accessibility tree, so proper headings, real buttons and labelled forms help people and agents alike. See UX/UI and accessibility.
In depth
Site migrations without losing rankings
To protect rankings in a migration, every old URL should 301-redirect to its closest equivalent, ranking content should survive and everything is validated before and after launch. Post-redesign drops usually come from forgotten redirects, trimmed content or an indexed staging site.
We step in before any code is written: a URL inventory with traffic and links, a redirect map, staging kept out of the index and a launch checklist. On multilingual sites, hreflang and canonicals must point to the new URLs; see international SEO.
After launch we monitor crawling, 404s, indexing and traffic against the baseline, use Search Console's Change of Address tool for domain moves and keep redirects long term.
What's included
What the technical SEO audit includes
Eight deliverables, scaled to your site and platform.
Executive summary
Site health, main risks and the highest-impact actions, without jargon.
Crawling and indexing
What the search engine sees and indexes, what it leaves out and why.
JavaScript rendering
Initial versus rendered HTML, revealing content and links crawlers miss.
Core Web Vitals diagnosis
LCP, INP and CLS from field data, with fixes per template.
AI bot policy
robots.txt, CDN and firewall reviewed, with a written decision on search and training.
Structured data plan
Which markup to keep, add or retire, validated with Google's tools.
Developer specifications
Tickets with acceptance criteria, ready for your developers or ours.
Validation and monitoring
Each fix verified, then tracked in Search Console and server logs.
How we work
How we run a technical SEO audit
In phases, as set out in our method; timing depends on site size and data access.
Phase 01 / 06
Scope and access
Goals, platform and languages; access to Search Console, analytics, CDN and logs.
Phase 02 / 06
Crawl and data
Full crawl, rendering checks, field data, indexing and bot activity.
Phase 03 / 06
Diagnosis and priorities
Each issue with its cause, impact and effort, in a phased plan.
Phase 04 / 06
Walkthrough
We review the report with your developers and agree first fixes.
Phase 05 / 06
Supported implementation
Your team applies the specs with our support, or we do it directly.
Phase 06 / 06
Validation and monitoring
Every change verified; later releases checked for regressions.
The SERMAN approach
Why choose SERMAN as your technical SEO agency
Built for Google and AI
We review AI crawlers, the CDN and snippet controls, not just Googlebot.
Priorities, not warning lists
Tools find hundreds of warnings; we tell you which matter and in what order.
From diagnosis to implementation
Developer-ready specs and a team that can apply them: the audit never ends as a PDF.
What it is not
What a technical audit is not
- Not a tool export sent without review.
- Not a ranking guarantee: it removes obstacles; content and authority still count.
- Not a chase for a perfect PageSpeed score: we optimise for real users.
- Not a myth seller: no llms.txt for Google, no AMP as an advantage, no FAQ markup for rich results.
What we measure
Signs of a technically healthy site
Index coverage
Key pages indexed and why others are excluded.
Field Core Web Vitals
LCP, INP and CLS from real users, by template and device.
Crawl errors
4xx and 5xx responses, redirect chains and surplus URLs.
Crawler activity
Visits from Googlebot, Bingbot and AI search bots in server logs.
Stability after changes
Indexing and organic traffic against the baseline, measured with reliable analytics.
Tools & platforms
Tools we work with
- Google Search Console
- Bing Webmaster Tools
- Screaming Frog SEO Spider
- Screaming Frog Log File Analyser
- Sitebulb
- PageSpeed Insights
- Lighthouse
- Chrome UX Report
- Rich Results Test
- Cloudflare
Who it's for
A particularly good fit if…
- Your organic traffic has dropped and you don't know why.
- You're redesigning, replatforming or changing domain.
- Your site runs on a JavaScript framework and Google struggles to index it.
- You run an online shop with filters, variants and thousands of URLs.
- You want to control which AI bots access your site.
Compliance
Rules and policies we check
Reviewed in every audit; not a substitute for legal advice.
Google spam policies
We flag practices Google penalises, such as back button hijacking (added in April 2026) or scaled low-value pages.
Accessibility
Since 28 June 2025, Spain's Law 11/2023 has required accessible consumer e-commerce (micro-enterprises exempt). We flag failures in the code.
Cookies and performance
Your banner must follow the AEPD cookie guide, with rejecting as easy as accepting, without hurting CLS or INP.
FAQ
Frequently asked questions
What does a technical SEO audit include?
A technical SEO audit reviews crawling, indexing, JavaScript rendering, internal linking, Core Web Vitals, structured data, language versions, security and AI bot access. You get a prioritised report, developer specifications and a validation plan.
How much does a technical SEO audit cost?
It depends on scope: URLs and templates, platform, languages, JavaScript, log access, and whether it includes implementation support or a migration. After a first call we send a tailored proposal detailing what we review and what you receive.
What are Core Web Vitals and what scores should I aim for?
They are the metrics Google uses to measure a page's loading, responsiveness and visual stability. A good experience means LCP at or under 2.5 seconds, INP under 200 milliseconds and CLS under 0.1.
How often should I run a technical SEO audit?
Before a migration or redesign, after a traffic drop and, as a general check, once a year. In between, Search Console monitoring and regular crawls catch what each release introduces.
How do I migrate a website without losing rankings?
With an inventory of URLs carrying traffic and links, one-to-one 301 redirects, equivalent content, staging out of the index, and updated hreflang and canonicals. Then monitor crawling, errors and indexing for weeks.
Should I block AI bots in robots.txt?
It depends on your goal. To be eligible for ChatGPT, Claude or Perplexity answers, allow OAI-SearchBot, Claude-SearchBot and PerplexityBot. Blocking training bots such as GPTBot or ClaudeBot doesn't cost that visibility. Also check your CDN, which may block them silently.
Related services
Pairs well with
SEO
Organic visibility on Google and in AI answers: a solid technical base, bilingual content and genuine authority, run as one measurable programme.
Web Development
Custom web engineering with Next.js and headless: measurable performance, integrations with your systems and care plans that protect the build.
Generative Engine Optimization (GEO)
GEO so ChatGPT, Gemini, Claude, Perplexity and Copilot can find, understand and cite your brand, built on solid SEO.
International & Multilingual SEO
International SEO that makes every language and market version of your site findable: architecture, hreflang, native keywords and country tracking.
Analytics, First-Party Data & Server-Side Tracking
GA4, first-party data, server-side tracking and Consent Mode v2 set up properly: reliable, lawful data you own, the base for campaigns and reports.
Insights
Go deeper
1 Oct 2026 · 12 min read · Design, web & motion
What is agentic commerce? How to make your website agent-ready
What agentic commerce means, what is actually live in Spain and how to prepare your website and store so an AI agent can read, compare and buy from you.
Contact
Start with a technical SEO audit
See what stops Google and AI crawlers reading your site properly, and what we'd fix first.
Personal reply, No obligation, English, Español
Start with SERMAN
Tell us about your project
First conversation
Diagnosis and proposal