Lack of interest Cookie support for unFURLing and RSS Feed Importer to get thru firewalls testing/blocking bots

This suggestion has been closed automatically because it did not receive enough votes over an extended period of time. If you wish to see this, please search for an open suggestion and, if you don't find any, post a new one.

vicos

Member
There is a problem with some sites not unFURLing:


Some of these new ultra-paranoid firewalls/gateways test unrecognized bots by using cookies and/or javascript tests.

I have Perl scripts that do similar functions and they get past the checks since I have cookie support enabled via:

## This will allow us to use Cookies in our requests
use HTTP::Cookies;
use HTTP::Cookies::Netscape;
use HTTP::Response;
## ---

Not sure if PHP has similar support, but maybe something the devs can consider for a future version.

We also have the same issue with some sites using the RSS Feed Importer. When we try to import from one site that sits behind Incapsula, we get this code back (I used wget to see what they were returning).
Code:
<html>
<head>
<META NAME="robots" CONTENT="noindex,nofollow">
<script src="/_Incapsula_Resource?SWJIYLWA=5074a744e2e3d891814e9a2dace20bd4,719d34d31c8e3a6e6fffd425f7e032f3">
</script>
<body>
</body></html>
 
Last edited:
Upvote 1
This suggestion has been closed. Votes are no longer accepted.
Back
Top Bottom