Web Automation

Enter your domain — a real crawler follows internal links, discovers every page and generates a valid sitemap.xml with last-modified dates. No URL pasting.

Crawl a site and export every discovered URL as a plain text list — for audits, migrations and screening.

Fetch your sitemap.xml and automatically check that every listed URL actually loads — catch 404s before search engines do.

Crawl up to 80 pages and grade title tags, meta descriptions, H1s, canonicals, thin content, mixed content and more — with a prioritized fix list.

Crawl a site, test every internal and external link, and report exactly which pages are broken and where they're linked from.

Crawl the site and audit every page's title and description length, plus missing and duplicate pairs.

Find every image without alt text across a site — required for WCAG accessibility and image SEO.

Map the H1/H2 outline of every crawled page and flag pages with missing or multiple H1s.

Word-count every page, average it, and flag pages under 300 words.

A complete crawl table: status codes, redirects, HTML size, image and link counts per page.

Crawl a site and summarize every outbound domain it links to, ranked by link count — great for link audits.

Crawl a website and collect every published email address and social profile link with the pages they were found on.

Detect the CMS, frameworks, CDN and server tech behind any website from headers, meta tags and known paths.

Automatically discover RSS/Atom feeds on any site — checks page headers and all standard feed locations.

Discover every favicon and touch icon a site publishes, with sizes — grab the best one for embedding.

Fetch and analyze any site's robots.txt: rules, blocked paths, sitemap declarations and common mistakes.

Probe a site for llms.txt, security.txt and other well-known files — the emerging standards AI crawlers and researchers look for.

Follow alias records to find where a hostname really points.

See which nameservers are authoritative for a domain.

List a domain's mail servers with priorities.

Inspect the start-of-authority record: primary NS, admin, serial and timers.

Check which certificate authorities are allowed to issue certificates for a domain.

One lookup that returns A, AAAA, CNAME, NS, MX, TXT, SOA and CAA records together.

Resolve an IP address back to its hostname — essential for mail server setup.