Skip to main content
DiffNexa

DiffNexaBot

DiffNexaBot is the page fetcher behind DiffNexa's web tools. If you run a website and see it in your logs, this page explains what it is doing and how to control it.

How to recognise it

DiffNexaBot identifies itself with this user agent:

DiffNexaBot/1.0 (+https://diffnexa.com/bot)

What it does

DiffNexaBot fetches a page only when a person using Website Change Detector, Policy & Terms Monitor, Competitor Monitor or Price Monitor asks DiffNexa to capture or check that page. Each request reads one page, so that person can see what the page says and, later, what changed on it.

  • It fetches the single address it was given. It does not follow links or crawl a site.
  • It does not run on a schedule. Every fetch is started by a person, when they choose.
  • It reads the HTML only. It does not run JavaScript or load images, stylesheets, scripts or other files.
  • It does not log in, submit forms or accept cookies.
  • It follows at most three redirects, and only to public addresses on the standard web ports.
  • It stops after 20 seconds, or after reading 5 MB, whichever comes first.
  • DiffNexa limits how often each of its visitors can capture or check pages.

Blocking or limiting it with robots.txt

DiffNexaBot reads your site's robots.txt before fetching a page and follows the rules for the DiffNexaBot user agent, or the rules for all robots (*) when there are none for it. To stop it fetching any page on your site:

User-agent: DiffNexaBot
Disallow: /
  • A robots.txt that returns a server error (5xx) is treated as “do not fetch” until it recovers.
  • A missing robots.txt (for example a 404) is treated as having no rules.
  • DiffNexaBot remembers your robots.txt for up to 15 minutes, so a change may take that long to apply.

When a page is blocked, the person who asked for it is told the address can't be checked because some sites ask not to be read automatically.

What happens to the page

The page's text and structure are returned to the person who asked for it. DiffNexa does not keep a copy, build an index or search engine, or use your pages to train AI models. The Privacy Policy explains how DiffNexa handles information.

Questions or problems

If DiffNexaBot is causing a problem for your site, or you have a question about it, email info@diffnexa.com with your domain and the times of the requests you are seeing.