WunderlyBot

What it is

WunderlyBot checks websites for the people who run them. Each request it makes says so in its user agent:

WunderlyBot/1.0 (+https://usewunderly.com/bot)

It visits only sites whose owners added Wunderly's analytics script to them, about once a week, and once soon after the script is installed. It looks for things that keep people from finding a site or using it: missing titles and descriptions, pages set to noindex, broken links and images, images over 500 KB, insecure files on secure pages, slow pages on phones, and accessibility problems that automated checks can find. The owner sees what it found on their own Wunderly page.

It also reads one page when someone asks for it in the free site check: that page only, once for each check, with the same rules. Your robots.txt decides whether it may, and it asks for at most 20 of the page's images, by size only. The free check is limited to a few checks an hour from any one place, and the result stays at its own link for 7 days.

What it reads

• Your robots.txt, first, and your sitemap.
• Up to 250 of your public pages, and the links and images on them. It follows links to your own site only. It asks links and images whether they answer without downloading them, and reads an image only when your server doesn't say how big it is, up to 500 KB.
• A few pages (the home page, the most-visited ones and the ones with a form) in a browser, to time how they load on a phone and to run automated accessibility checks. A page that draws its links with scripts is opened in a browser too, at most 20 a check, so its links can be found.

What it never does

• It never sends a form, signs in, or buys anything.
• It uses only GET and HEAD requests. In the browser, anything else a page tries to send is stopped.
• It asks your site for one page, link or image a second at most. A page it opens in a browser loads its own images, styles and scripts as any browser would.
• It never reads a page your robots.txt keeps it out of.

How to block it

WunderlyBot obeys robots.txt under the name WunderlyBot. To keep it out of your whole site:

User-agent: WunderlyBot
Disallow: /

To keep it out of one part of your site:

User-agent: WunderlyBot
Disallow: /private/

If you run the site and use Wunderly, blocking it means the weekly check can't run, and your fix list says so.

Questions

If WunderlyBot is doing something it shouldn't on your site, write to legal@usewunderly.com with your site's address, and we'll look the same day.