Note that Crawl-delay is not part of the original robots.txt specification. But it’s no problem to include them for those parsers that understand it, as the spec defines: Unrecognised headers are ignored. So older robots.txt parsers will s…