Scanning Site for Bad Links

A

Alfuzzy

Well-known member
Hello everyone. I'm trying to scan my site for bad/old outdated links.

Issue is...if the home page URL is used as the starting point for a scan...the site is too large to do this all in one go (scans can take hours & hours...and this is when only a portion of the site is being scanned).

What I would like to do is limit the scope of each scan by only scanning one forum node at a time.

For example (using Xenforo.com as an example)...let's say I wanted to only scan the "Xenforo pre-sales questions" node. If the "Xenforo pre-sales questions" node is clicked on...the URL = https://xenforo.com/community/forums/xenforo-pre-sales-questions.5/

Xenforo Home Page.webp

If this URL is used for a scan...only the thread titles in this node are scanned...individual threads & thread posts are not scanned. I've tried other URL's...but nothing I've tried will scan all the thread posts in a single forum node.

Again my goal is to scan all thread posts in a single Xenforo forum website node (as described above).

Obviously I'm not getting something correct. If anyone knows what the URL syntax should be for scanning all the thread posts in a single forum node...I would greatly appreciate it!

Thanks:)
 
Alfuzzy said:
Hello everyone. I'm trying to scan my site for bad/old outdated links.

Issue is...if the home page URL is used as the starting point for a scan...the site is too large to do this all in one go (scans can take hours & hours...and this is when only a portion of the site is being scanned).
Click to expand...
Are you trying to replicate what this addon does? …with a different tool?
 
Thanks for the link...yes I'm aware of that tool/add-on.

Using a different method that simply scans the site for bad links...get a list...then fix them one by one manually. This gives greater control over what I want to do with each individual link (fix it, remove it, replace with a working link).

What I'm looking for is help with how to scan smaller portions of the site (individual nodes for example)...so the scans don't take quite as long.

Thanks
 
With Screaming Frog (paid) you can set delays and url restrictions. You can set the parent forum as the URL to spider and then in the settings set the number of links deep to follow to 4. You can also use the Segments settings. The software costs €245 a year.
 
Thanks much Andy...I know you have lots of good stuff!:)

Sounds like your add-on mostly handles 404 errors (a very common error of course). 404 errors probably account for about 5-10% of the bad link errors I'm dealing with.

I need to re-new my subscription at your site...so I can get some updated goodies!

Thanks:)
 
To repeat what I'm looking for:

If I use the home page URL of the website for the scans...everything works fine...it just takes a very long time due to the size of the website.

What I'm looking for to reduce the size of scans...is what URL I would need to use to scan a single forum node...such as this:

Xenforo Home Page.webp

Thanks for any help.:)
 
You must log in or register to reply here.
Back
Top Bottom