I want to be able to access /robots.txt
from a variety of sites using JavaScript. This is for a side project that tests the availability of sites, not all of which are under my control. I've tried this:
$.get(robotsUrl, function() {
console.log('success!');
}, "text")
.fail(function() {
console.log('failed :(');
});
However, this fails with
XMLHttpRequest cannot load https://my.test.url/robots.txt. Origin http://localhost:8000 is not allowed by Access-Control-Allow-Origin
MDN's page on Same-Origin-Policy says that it's possible to embed content with some elements, such as <script>
, <iframe>
<embed>
. Could I load /robots.txt
from an arbitrary site with any of these? Is there any other way I can access this file on other domains?