vicos
Member
There is a problem with some sites not unFURLing:
Some of these new ultra-paranoid firewalls/gateways test unrecognized bots by using cookies and/or javascript tests.
I have Perl scripts that do similar functions and they get past the checks since I have cookie support enabled via:
## This will allow us to use Cookies in our requests
use HTTP::Cookies;
use HTTP::Cookies::Netscape;
use HTTP::Response;
## ---
Not sure if PHP has similar support, but maybe something the devs can consider for a future version.
We also have the same issue with some sites using the RSS Feed Importer. When we try to import from one site that sits behind Incapsula, we get this code back (I used wget to see what they were returning).
Some of these new ultra-paranoid firewalls/gateways test unrecognized bots by using cookies and/or javascript tests.
I have Perl scripts that do similar functions and they get past the checks since I have cookie support enabled via:
## This will allow us to use Cookies in our requests
use HTTP::Cookies;
use HTTP::Cookies::Netscape;
use HTTP::Response;
## ---
Not sure if PHP has similar support, but maybe something the devs can consider for a future version.
We also have the same issue with some sites using the RSS Feed Importer. When we try to import from one site that sits behind Incapsula, we get this code back (I used wget to see what they were returning).
Code:
<html>
<head>
<META NAME="robots" CONTENT="noindex,nofollow">
<script src="/_Incapsula_Resource?SWJIYLWA=5074a744e2e3d891814e9a2dace20bd4,719d34d31c8e3a6e6fffd425f7e032f3">
</script>
<body>
</body></html>
Last edited:
Upvote
1