SEO4Ajax web crawler

What is s4a?

If you've come to this page, then you're probably interested in learning more about our web crawler, identified by default as user-agent "s4a".

SEO4Ajax is a prerendering service. Our crawler renders the pages of our customers' websites in a headless browser and stores snapshots of them, so that search engines and social network crawlers can be served a fully rendered version of these pages.

The SEO4Ajax crawler runs on a cloud computing platform that consists of several thousand computers, which is why you may see our web crawler access your site from many different IP addresses.

Why is s4a crawling my website?

The crawler only visits websites on behalf of SEO4Ajax users. SEO4Ajax allows its users to run web crawls on their site in order to refresh cached pages. So, if s4a is crawling your website, it means that one or more SEO4Ajax users created a web crawl that went (eventually) to your website.

User agent

By default, the crawler identifies itself with the following user agent: 

Mozilla/5.0 AppleWebKit (compatible; s4a/1.0; +https://www.seo4ajax.com/webscraper)

SEO4Ajax users can customize the user agent used to crawl their own websites (for example to emulate a mobile device). In that case, the requests can still be identified by their signature, see Verifying our requests below.

What does the crawler fetch?

The crawler fetches the pages of the websites configured by SEO4Ajax users, along with the resources needed to render them (scripts, stylesheets, images, API calls, etc.), and the sitemaps declared in the configuration of these websites. It follows the links found in the pages according to the crawl settings defined by the user.

robots.txt

The crawler does not read robots.txt files. The pages to crawl, the crawl rate and the exclusion rules are configured by the website owner directly in their SEO4Ajax account. Besides, robots.txt rules are meant for search engines: applying them would prevent us from prerendering the very pages that the website owner wants search engines to index.

Verifying our requests

All requests sent by the crawler are signed with Web Bot Auth (HTTP Message Signatures, RFC 9421) with an Ed25519 key. Each request carries the following headers:

  • Signature-Agent: "https://www.seo4ajax.com"
  • Signature-Input and Signature, with the tag web-bot-auth

The public keys are published in our key directory: https://www.seo4ajax.com/.well-known/http-message-signatures-directory.

Since the signature does not depend on the user agent, this is the reliable way to identify our crawler, including when a custom user agent is used.

Stopping or slowing down the crawler

If you are an SEO4Ajax user, you can pause the crawl of your website at any time in the SEO4Ajax console, and adjust its settings (crawl rate, pages to crawl).

If you don't have an SEO4Ajax account, or if you feel that s4a is crawling your website too quickly, please contact us at contact@seo4ajax.com and we will stop or slow down the crawl of your website.

Blocking us by IP address

Blocking our web crawler by IP address will not work. Due to the distributed nature of our infrastructure, we have numerous constantly changing IP addresses. Please rely on the request signatures described above to identify our crawler, or contact us if you want us to stop crawling your website.

Learn more

To learn more about SEO4Ajax, please check out the rest of the site. If you'd like to ask us any questions, don't hesitate to contact us.