Security Headers and robots.txt Check: graded, in bulk

Give it a list of URLs and it returns, for each one, the security headers the server sends, what each value means, a score out of 100 with a grade from A to F, whether the plain http:// address moves visitors to HTTPS, how the page's cookies are flagged, and the site's robots.txt parsed into rules.

What is checked

The URL is requested with redirects followed, and the final page's headers are read. Only the headers are read; the page body is not downloaded.

• Strict-Transport-Security: present, max-age, includeSubDomains, preload. A header sent over plain HTTP is reported and scores 0.
• Content-Security-Policy: the directives it sets, and weaknesses in script-src (or default-src when there is no script-src): 'unsafe-inline' without a nonce, hash or 'strict-dynamic', 'unsafe-eval', and wildcard or scheme sources such as * or https:. A report-only policy is listed separately.
• X-Frame-Options: DENY or SAMEORIGIN. A CSP frame-ancestors directive counts in its place.
• X-Content-Type-Options: nosniff
• Referrer-Policy: the policy that applies, the last known one in the list
• Permissions-Policy: present, and the features it names
• HTTPS: whether the final page is HTTPS, and whether the http:// address of an HTTPS URL ends on HTTPS
• Cookies: name, Secure, HttpOnly and SameSite for each Set-Cookie header. Cookie values are never stored.
• Disclosure: the Server, X-Powered-By, X-AspNet-Version, X-AspNetMvc-Version and X-Generator headers, with an issue when one carries a version number
• Other headers, listed without scoring: Cross-Origin-Opener-Policy, Cross-Origin-Embedder-Policy, Cross-Origin-Resource-Policy, X-XSS-Protection

The grade

The six headers share 100 points.

Header	Full points	Partial points
Strict-Transport-Security	25 with a max-age of one year (31,536,000 seconds) or more	15 with a shorter max-age above 0
Content-Security-Policy	25 when enforced with none of the weaknesses above	15 when enforced with a weakness, 5 when report-only
X-Frame-Options	15 for DENY, SAMEORIGIN or a CSP frame-ancestors directive	n/a
X-Content-Type-Options	10 for nosniff	n/a
Referrer-Policy	15 for no-referrer, same-origin, strict-origin, strict-origin-when-cross-origin, origin or origin-when-cross-origin	5 for unsafe-url or no-referrer-when-downgrade
Permissions-Policy	10 when present	n/a

A missing header scores 0. The grade follows the score: A from 90, B from 75, C from 60, D from 40, and F below 40. A page whose final URL is not HTTPS is graded F whatever its score. Cookie flags and disclosure headers are reported as issues but do not change the score.

robots.txt

The robots.txt file of the final URL's host is fetched and parsed into groups of user agents with their allow and disallow rules and crawl delay, plus the sitemaps it names and any lines it does not recognize. For each user agent you list (default *, Googlebot, Bingbot and GPTBot) you get whether that crawler is allowed to fetch the URL and the rule that decides it.

Matching follows RFC 9309: user agents match without regard to case, groups for the same agent are combined, the longest matching rule wins, allow wins a tie, * matches any characters and $ anchors the end [1]. A 4xx answer means crawlers are free to fetch everything, and a 5xx answer means they must treat the whole site as disallowed [1]. Issues flag a robots.txt that disallows the whole site for all crawlers, answers with an HTML page, names no sitemap, has lines that are not recognized, or is larger than 500 KiB, the smallest parsing limit RFC 9309 allows [1].

Input

• URLs (required). An address without http:// or https:// is checked as https://.
• Fetch and parse robots.txt (default on) and the user agents to test
• Check the HTTP to HTTPS redirect (default on)
• Timeout per request (default 20 seconds), URLs checked at once (default 10), user agent

Output

Dataset: one item per URL with url, ok, status, finalUrl, redirectCount, https, httpRedirectsToHttps, httpRedirectError, grade, score, scoreBreakdown, headers (one entry per scored header with its value and findings), otherHeaders, disclosure, cookies, cookieIssues, issues, issueCount, robotsTxt and checkedAt. A URL that cannot be reached gets an item with ok false and the reason in error. Export as JSON, CSV or Excel, or read it over the API.

Pricing

Pay per event: one event per URL checked. URLs that cannot be reached and entries that are not valid URLs are reported and not charged.

Scope and conduct

It makes at most three requests per URL: the page, its http:// address, and robots.txt. It sends no forms, tries no attacks and does not log in.

Support

Questions and problems: the Issues tab on this page. Replies are written by an AI assistant on behalf of the developer and reviewed by the developer.

References

[1] RFC 9309, Robots Exclusion Protocol, sections 2.2.1, 2.2.2, 2.2.3, 2.3.1.3, 2.3.1.4 and 2.5. https://www.rfc-editor.org/rfc/rfc9309.html