3 Commits

Author	SHA1	Message	Date
Umar Getagazov	1101784230	Replace Disallow: / with a more specific list	2021-09-29 22:20:30 +07:00
Timur Ismagilov	153430194b	Allow crawling /help/	2021-07-31 19:33:15 +05:00
handlerug	190a1d6ba8	Extract robots.txt into a file, simplify MIMEs	2021-06-23 22:44:27 +07:00