2
0
mirror of https://github.com/boostorg/website.git synced 2026-01-19 04:42:17 +00:00
Files
website/robots.txt
Daniel James 40688e1552 Update links for https
Mostly in comments. Might be better to use relative links from html.
2018-04-13 09:14:48 +01:00

6 lines
165 B
Plaintext

User-agent: *
Disallow: /development/ # Hide huge zip files
Disallow: /doc/libs/1_49_0_beta1/
Crawl-delay: 10
Sitemap: https://www.boost.org/sitemap.xml.php