A search engine spider running on XenForo framework (iolabs.io)

digitalpoint

Well-known member
So this is an interesting story...

As some may know, I went through the federal legal system awhile ago... It started in 2007, and in 2013 the federal prosecutors finally offered me a plea deal (over 6 years of dealing with court and lawyers and the whole legal system). With what they charged me with, I was looking at up to 20 years in prison and $87M restitution. Long story short is I didn't actually do what they accused me of, but the technical prowess of federal prosecutors made it so they barely even understood what I was being charged with. It was my word against someone else's, with neither side able to prove it either way (which is why my case dragged on for so long).

In 2013, I had an epiphany... if I made a special kind of search engine that could spider the entire web, fully render it/execute JavaScript, render CSS, etc., then keep track of all the cookies being set, keep backtraces of how everything renders and executes, I would be able to show it was mathematically impossible to have done what they charged me with. I presented the results to the prosecutors and they finally "got it". They offered me my first plea deal (where I went to camp for 5 months and had $0 restitution)... they got a "win", I could be done with the legal system and stop bleeding money to lawyers and get my life back. So I took it.

I ended up leaving the search engine running all these years because I don't know why... I think data is cool and I had the infrastructure to do it (didn't cost me anything). One day I had another epiphany... my search engine that was able to show I couldn't have been cookie stuffing/committing click fraud, had all the data (going back to 2013) of every website that WAS (for every ad network and affiliate program out there). That might be useful to ad platforms or affiliate programs. Save money by not paying publishers committing click fraud.

Later added the logging of JavaScript objects which can answer other questions for people, like what technology, services or frameworks are being used by websites.

Decided to put a user interface on it where someone can get realtime alerts (via push notifications, XF alerts, Telegram channels, Discord channels, Slack channels), and turn it into a service for those businesses that would be interested in that info.

Some examples:
For companies that want to know more than "who", but also "how", the spider generates full backtraces of the spidering. For example, this was a real example where a website was using a 0x0 image to cookie stuff eBay: https://iolabs.io/logs/example

Originally the spider was built on XF1 framework on digitalpoint.com (it's what generates the Cookie Search stuff that some people are familiar with). When I decided to put a UI on it for alerts and stuff, it was moved to XF2.

Anyway, if anyone is bored, this is it: https://iolabs.io/

Maybe (or maybe not, I don't actually know) the only global search engine running on XenForo? 🤷‍♂️

Oh yeah, it's also the new home for the Cookie Search system. For example commercial forum marketshare chart: https://iolabs.io/stats/commercial-forum-platforms/
 
Last edited:
Top