User-agent: * Disallow: / Disallow: /search-page/* Disallow: *.pdf # Block pdf files. Non-standard but works for major search engines. Sitemap: https://staging.rias.co.uk/robots.txt