Link Checker by AddonsLab [Paid] 1.17.0

[AddonsLab]

Hello, @Court !

In general, the add-on does not execute any checks or any time-consuming processes during page load, neither the first page of the session nor later. All actual checks are done in the background.

It seems the issue you have is related to the moderation of broken links - XenForo builds moderation cache and counts the number of items pending for moderation during the page load, and if the cache is out of date, it will take quite long time to build it. If this is the case, only admin/moderator accounts will be affected by this issue (e.g. open the website as a guest or a user who does not have moderator permissions and you will not see increased TTFB), also you can see an increased number of items pending for moderation. If this is the case, the only way to speed up the page would be clearing up the items from moderation queue (e.g. removing the broken links).

Please let us know if you still have any issues.

Thank you!

 

[Court]

Hello, @Court !

In general, the add-on does not execute any checks or any time-consuming processes during page load, neither the first page of the session nor later. All actual checks are done in the background.

It seems the issue you have is related to the moderation of broken links - XenForo builds moderation cache and counts the number of items pending for moderation during the page load, and if the cache is out of date, it will take quite long time to build it. If this is the case, only admin/moderator accounts will be affected by this issue (e.g. open the website as a guest or a user who does not have moderator permissions and you will not see increased TTFB), also you can see an increased number of items pending for moderation. If this is the case, the only way to speed up the page would be clearing up the items from moderation queue (e.g. removing the broken links).

Please let us know if you still have any issues.

Thank you!

Wonderful thank you! I re-enabled the addon and tested while not signed in and sure enough it loads fast every time. We do have a lot of broken links to work through! Since it's working fast for our regular users all is well.

Appreciate the fast response, and the great addon

 

[Gossipy]

Hi - we just purchased this. It's not letting us select a status code for a batch operation. Am I overlooking something obvious? Thank you.

 

[AddonsLab]

Hi - we just purchased this. It's not letting us select a status code for a batch operation. Am I overlooking something obvious? Thank you.

View attachment 190785

Please follow the instructions at https://xenforo.com/community/resources/link-checker-by-addonslab.5579/field?field=faq to get started and it should work.

Thank you!

 

[Gossipy]

Please follow the instructions at https://xenforo.com/community/resources/link-checker-by-addonslab.5579/field?field=faq to get started and it should work.

Thank you!

We overlooked those.

Thank you for your fast response.

 

[AddonsLab]

We overlooked those.

Thank you for your fast response.

You haven't overlooked anything The instructions were created for Link Checker for 2.x but we forgot to add them for XenForo 1.x version.

Just let us know if you need anything else.

Thank you!

 

[Gossipy]

You haven't overlooked anything The instructions were created for Link Checker for 2.x but we forgot to add them for XenForo 1.x version.

Just let us know if you need anything else.

Thank you!

Oh, thanks again.

I've noticed with certain settings on the "Check Dead Links" process, I get this:

An exception occurred: Call to undefined function AddonsLab\Core\Service\mb_detect_encoding() in /home/xxxxx/public_html/forum/library/AddonsLab/Core/Service/Utf8Converter.php on line 44

AddonsLab\Core\Service\Utf8Converter->convertToUtf8() in AddonsLab/LinkChecker/Library/Service/UrlProcessor.php at line 80
AddonsLab\LinkChecker\Library\Service\UrlProcessor->getDecoded() in AddonsLab/LinkChecker/Model/Tag.php at line 269
AddonsLab_LinkChecker_Model_Tag->prepareTag() in AddonsLab/LinkChecker/Abstract/Model/Tag.php at line 225
AddonsLab_LinkChecker_Abstract_Model_Tag->prepareTags() in AddonsLab/LinkChecker/DataProvider/TagDataProvider.php at line 228
AddonsLab\LinkChecker\DataProvider\TagDataProvider->getTags() in AddonsLab/LinkChecker/Library/Service/JobRunner.php at line 204
AddonsLab\LinkChecker\Library\Service\JobRunner->runDeadLinkCheckJob() in AddonsLab/LinkChecker/Deferred/DeadLink.php at line 42
AddonsLab_LinkChecker_Deferred_DeadLink->_execute() in AddonsLab/LinkChecker/Deferred/AbstractTask.php at line 42
AddonsLab_LinkChecker_Deferred_AbstractTask->execute() in XenForo/Model/Deferred.php at line 295
XenForo_Model_Deferred->runDeferred() in XenForo/Model/Deferred.php at line 429
XenForo_Model_Deferred->_runInternal() in XenForo/Model/Deferred.php at line 374
XenForo_Model_Deferred->run() in XenForo/ControllerAdmin/Tools.php at line 159
XenForo_ControllerAdmin_Tools->actionRunDeferred() in XenForo/FrontController.php at line 369
XenForo_FrontController->dispatch() in XenForo/FrontController.php at line 152
XenForo_FrontController->run() in /home/xxxxx/public_html/forum/admin.php at line 13

I haven't been able to isolate which settings trigger it. Sometimes it does it with the same settings that worked previously. Other times if I change the timeout or items per-page, it gives the error once, but then works the second time. Any ideas?

 

[AddonsLab]

Oh, thanks again.

I've noticed with certain settings on the "Check Dead Links" process, I get this:

An exception occurred: Call to undefined function AddonsLab\Core\Service\mb_detect_encoding() in /home/xxxxx/public_html/forum/library/AddonsLab/Core/Service/Utf8Converter.php on line 44

AddonsLab\Core\Service\Utf8Converter->convertToUtf8() in AddonsLab/LinkChecker/Library/Service/UrlProcessor.php at line 80
AddonsLab\LinkChecker\Library\Service\UrlProcessor->getDecoded() in AddonsLab/LinkChecker/Model/Tag.php at line 269
AddonsLab_LinkChecker_Model_Tag->prepareTag() in AddonsLab/LinkChecker/Abstract/Model/Tag.php at line 225
AddonsLab_LinkChecker_Abstract_Model_Tag->prepareTags() in AddonsLab/LinkChecker/DataProvider/TagDataProvider.php at line 228
AddonsLab\LinkChecker\DataProvider\TagDataProvider->getTags() in AddonsLab/LinkChecker/Library/Service/JobRunner.php at line 204
AddonsLab\LinkChecker\Library\Service\JobRunner->runDeadLinkCheckJob() in AddonsLab/LinkChecker/Deferred/DeadLink.php at line 42
AddonsLab_LinkChecker_Deferred_DeadLink->_execute() in AddonsLab/LinkChecker/Deferred/AbstractTask.php at line 42
AddonsLab_LinkChecker_Deferred_AbstractTask->execute() in XenForo/Model/Deferred.php at line 295
XenForo_Model_Deferred->runDeferred() in XenForo/Model/Deferred.php at line 429
XenForo_Model_Deferred->_runInternal() in XenForo/Model/Deferred.php at line 374
XenForo_Model_Deferred->run() in XenForo/ControllerAdmin/Tools.php at line 159
XenForo_ControllerAdmin_Tools->actionRunDeferred() in XenForo/FrontController.php at line 369
XenForo_FrontController->dispatch() in XenForo/FrontController.php at line 152
XenForo_FrontController->run() in /home/xxxxx/public_html/forum/admin.php at line 13

I haven't been able to isolate which settings trigger it. Sometimes it does it with the same settings that worked previously. Other times if I change the timeout or items per-page, it gives the error once, but then works the second time. Any ideas?

This is a built-in function available in most PHP installation but seems to be missing in yours. Please check with your host provider and see if it possible to enable these functions - http://php.net/manual/en/mbstring.installation.php Let us know if this is impossible and we will try to implement some workaround.

Thank you!

 

[Gossipy]

This is a built-in function available in most PHP installation but seems to be missing in yours. Please check with your host provider and see if it possible to enable these functions - http://php.net/manual/en/mbstring.installation.php Let us know if this is impossible and we will try to implement some workaround.

Thank you!

That seems to have fixed the problem. Thanks again!

 

[AddonsLab]

Hello!

This is an official announcement that the price for the add-on will be updated on 1st of March and it will be 54.99 USD.

The add-on started as a relatively simple automation tool to check for broken links but has grown since to include many advanced features and the sales we get don't cover the expenses we had developing the solution, so this change is already long overdue.

The change does not affect existing customers in any way.

Thank you!

 

[DavidXLD]

Hello,

I wish to switch my french forum from normal URLs to "Romanize Titles in URLs".
My forum is 4 years old and there are links to threads in messages with accents in URLs.

So I would like to use your add-on in order to replace URLs with accents by URLs without accents.

I will :
1 - Activate "Romanize Titles in Urls" XenForo Option
2 - Use your add-in order to replace URLS having accents inside by latin characters

For example I have this URL :

[URL='https://www.mysite.com/threads/r%C3%A9solu-recherche-windows-%C3%A0-partir-dexcel.20003552/#post-20026995']https://www.mysite.com/threads/résolu-recherche-windows-à-partir-dexcel.20003552/#post-20026995[/URL]

That I wish to replace by :

https://www.mysite.com/threads/resolu-recherche-windows-a-partir-dexcel.20003552/#post-20026995

Is there a way to do that with the add-on using regexp, maybe in 2 times?

Thanks
David

 

[AddonsLab]

Hello, @DavidXLD

Unfortunately, the scenario described is not possible to handle with our add-on, as it is too much complex transformation of the URL. Regular expressions are not much help in this case, the URL needs to be decoded first to it original form like https://www.mysite.com/threads/résolu-recherche-windows-à-partir-dexcel.20003552/#post-20026995 using "urldecode", then accents should be removed using "iconv" function. This will require a custom "Replacement" criterion in Admin Panel UI. As this is a rare use-case for the add-on we can implement the enhancement as a funded task. Please contact us at https://customers.addonslab.com/submitticket.php and we will discuss the terms of implementation.

Thank you!

 

[DavidXLD]

Hello,
Thanks for your prompt reply.
I will think about your proposal as I am not yet 100% sure to switch to "Romanize Titles in URLs".
I think it could have a bad impact for pages which are already indexed with accents in URL.
Google might not understand such a big change an negatively indexed my website.
An opinion about that?
David

 

[DavidXLD]

Hi again,
I have stopped the "Check dead links" operation before the end of the process.
I had around 18 000 links to analyse.
When, I stopped around 9 000 links had been analyzed.
I would like to restart the process only for the links which have not been analysed but I find only 1379 not checked URL which seems very low.
It seems that the 200 status code seems more realistic regarding the number of URL not analysed.
Could you please confirm what is the status of links not analyzed when a "Check dead links" analyse is canceled during the data processing?
[ATTACH type="full" width="463px" alt="196660"]196660[/ATTACH]
Many thanks in advance

 

[AddonsLab]

Hello, @DavidXLD

Google might not understand such a big change an negatively indexed my website.

This could indeed have an unexpected effect on the site's ranking. I think Google would identify them as new URLs in the content (and old ones would be considered as deleted) and I am not sure if it would be a good idea to have thousands of new URLs added and removed in the content. I think the safest option would be to enable "Romanize Titles in URLs" to ensure, that older URLs redirect to newer ones using "301 Permanent Redirect" header. In this case, I don't see any special reason to have the actual URLs updated in post content itself.

I would like to restart the process only for the links which have not been analyzed but I find only 1379 not checked URL which seems very low.

As I see from your screenshot there are indeed only 1379 unchecked links, the remaining is already checked (maybe via additional background jobs, not sure). You should consider checking the Invalid URL group especially, as it seems to be a major part of broken URLs.

Thank you!

 

[Alpha1]

I have some questions. After building the index for the link checker this is the result:

As you see out of 600k links there are:

• 5k links with 403 status. These are all internal links that require usergroup permission. Is there any way to check links to xf pages that require permission?
• 5k links with 520 status. These are all valid internal links that require usergroup permission. Any idea what is causing this to come back as 520 instead of 403?
• 584k links that are unchecked. How do I get these checked?

 

[AddonsLab]

I have some questions. After building the index for the link checker this is the result:
View attachment 246443
As you see out of 600k links there are:

• 5k links with 403 status. These are all internal links that require usergroup permission. Is there any way to check links to xf pages that require permission?
• 5k links with 520 status. These are all valid internal links that require usergroup permission. Any idea what is causing this to come back as 520 instead of 403?
• 584k links that are unchecked. How do I get these checked?

There is no special handling of XenForo pages and permission-related checks, so 403 status is what a browser client would see and that's why you see by link checker.

520 error seems to be sent to us by Cloudflare I am not sure what exactly was the cause.

Indexation just finds the links, it does not check their status. Use the Rebuild data -> Check dead links section to check those links.

 

[Alpha1]

XenForo_Exception: The existing data required by the data writer could not be found. - library/XenForo/DataWriter.php:1359
Generated By: Unknown Account, Today at 3:19 AM
Stack Trace

#0 //library/XenForo/DataWriter.php(1401): XenForo_DataWriter->_haveErrorsPreventSave()
#1 //library/MetaMirror/DataWriter/DiscussionMessage/Post.php(24): XenForo_DataWriter->save()
#2 //library/SV/DeadlockAvoidance/XenForo/DataWriter/DiscussionMessage/Post.php(11): MetaMirror_DataWriter_DiscussionMessage_Post->save()
#3 //library/AddonsLab/ContentHandler/Traits/PostTrait.php(271): SV_DeadlockAvoidance_XenForo_DataWriter_DiscussionMessage_Post->save()
#4 //library/AddonsLab/LinkChecker/Library/Content.php(97): AddonsLab_LinkChecker_ContentHandler_PostHandler->updateContentMessage(Array)
#5 //library/AddonsLab/LinkChecker/Library/Service/TagReplacementProcessor.php(265): AddonsLab\LinkChecker\Library\Content->persist()
#6 //library/AddonsLab/LinkChecker/Library/Service/TagReplacementProcessor.php(320): AddonsLab\LinkChecker\Library\Service\TagReplacementProcessor->persistTagReplacementsForContent(Object(AddonsLab\LinkChecker\Library\Content), Array)
#7 //library/AddonsLab/LinkChecker/Library/Service/JobRunner.php(312): AddonsLab\LinkChecker\Library\Service\TagReplacementProcessor->persistTagReplacementList(Object(AddonsLab\LinkChecker\Library\TagReplacementList))
#8 //library/AddonsLab/LinkChecker/Deferred/BatchUpdate.php(42): AddonsLab\LinkChecker\Library\Service\JobRunner->runBatchUpdateJob(Array)
#9 //library/AddonsLab/LinkChecker/Deferred/AbstractTask.php(42): AddonsLab_LinkChecker_Deferred_BatchUpdate->_execute(Array, Array, 8, '')
#10 //library/XenForo/Model/Deferred.php(295): AddonsLab_LinkChecker_Deferred_AbstractTask->execute(Array, Array, 8, '')
#11 //library/XenForo/Model/Deferred.php(429): XenForo_Model_Deferred->runDeferred(Array, 8, '', false)
#12 //library/XenForo/Model/Deferred.php(374): XenForo_Model_Deferred->_runInternal(Array, 8, '', false)
#13 //deferred.php(23): XenForo_Model_Deferred->run(false)
#14 {main}

Request State

array(3) {
["url"] => string(36) "/deferred.php"
["_GET"] => array(0) {
}
["_POST"] => array(3) {
["_xfRequestUri"] => string(58) "/threads/151917/"
["_xfNoRedirect"] => string(1) "1"
["_xfResponseType"] => string(4) "json"
}
}

I have been encountering a lot of deadlocks when batch updating links through the immediately option.
I get this on the cron option:

 

[Alpha1]

The link checker only sees the exact domain set in xenforo as internal. My domain is https://mydomain.com
Links to www.mydomain.com http://mydomain.com and http://www.mydomain.com are erroneously listed as external domains.
This makes it a daunting task to manage dead links when there are 100k+ external links that have a certain status. Internal dead links is something we manually try to fix. So we cannot batch process external links if there are internal links mixed in. Please address this.

 

[Alpha1]

The link checker chokes on this. I have 210k errors:

ErrorException: Undefined index: type - library/AddonsLab/LinkChecker/ContentHandler/PostHandler.php:109
Generated By: Unknown Account, Sunday at 5:10 PM
Stack Trace

#0 /library/AddonsLab/LinkChecker/ContentHandler/PostHandler.php(109): XenForo_Application::handlePhpError(8, 'Undefined index...', '/home/nginx/dom...', 109, Array)
#1 /library/AddonsLab/LinkChecker/Library/Service/TagReplacementProcessor.php(309): AddonsLab_LinkChecker_ContentHandler_PostHandler->applyReplaceCriteria(Object(AddonsLab\LinkChecker\Library\Replace\ReplaceCriteria))
#2 /library/AddonsLab/LinkChecker/Library/Service/JobRunner.php(312): AddonsLab\LinkChecker\Library\Service\TagReplacementProcessor->persistTagReplacementList(Object(AddonsLab\LinkChecker\Library\TagReplacementList))
#3 /library/AddonsLab/LinkChecker/Deferred/BatchUpdate.php(42): AddonsLab\LinkChecker\Library\Service\JobRunner->runBatchUpdateJob(Array)
#4 /library/AddonsLab/LinkChecker/Deferred/AbstractTask.php(42): AddonsLab_LinkChecker_Deferred_BatchUpdate->_execute(Array, Array, 8, '')
#5 /library/XenForo/Model/Deferred.php(295): AddonsLab_LinkChecker_Deferred_AbstractTask->execute(Array, Array, 8, '')
#6 /library/XenForo/Model/Deferred.php(429): XenForo_Model_Deferred->runDeferred(Array, 8, '', false)
#7 /library/XenForo/Model/Deferred.php(374): XenForo_Model_Deferred->_runInternal(Array, 8, '', false)
#8 /deferred.php(23): XenForo_Model_Deferred->run(false)
#9 {main}

Request State

array(3) {
["url"] => string(36) "/deferred.php"
["_GET"] => array(0) {
}
["_POST"] => array(3) {
["_xfRequestUri"] => string(111) "/forums/377/?prefix_ids[0]=184&prefix_ids[1]=168&prefix_ids[2]=201&prefix_ids[3]=185&prefix_id=208"
["_xfNoRedirect"] => string(1) "1"
["_xfResponseType"] => string(4) "json"
}
}

And:
Error Info
ErrorException: Undefined index: type - library/AddonsLab/LinkChecker/ContentHandler/PostHandler.php:109
Generated By: Unknown Account, Sunday at 4:19 PM
Stack Trace

#0 /library/AddonsLab/LinkChecker/ContentHandler/PostHandler.php(109): XenForo_Application::handlePhpError(8, 'Undefined index...', '/home/nginx/dom...', 109, Array)
#1 /library/AddonsLab/LinkChecker/Library/Service/TagReplacementProcessor.php(309): AddonsLab_LinkChecker_ContentHandler_PostHandler->applyReplaceCriteria(Object(AddonsLab\LinkChecker\Library\Replace\ReplaceCriteria))
#2 /library/AddonsLab/LinkChecker/Library/Service/JobRunner.php(312): AddonsLab\LinkChecker\Library\Service\TagReplacementProcessor->persistTagReplacementList(Object(AddonsLab\LinkChecker\Library\TagReplacementList))
#3 /library/AddonsLab/LinkChecker/Deferred/BatchUpdate.php(42): AddonsLab\LinkChecker\Library\Service\JobRunner->runBatchUpdateJob(Array)
#4 /library/AddonsLab/LinkChecker/Deferred/AbstractTask.php(42): AddonsLab_LinkChecker_Deferred_BatchUpdate->_execute(Array, Array, 7.9999990463257, '')
#5 /library/XenForo/Model/Deferred.php(295): AddonsLab_LinkChecker_Deferred_AbstractTask->execute(Array, Array, 7.9999990463257, '')
#6 /library/XenForo/Model/Deferred.php(429): XenForo_Model_Deferred->runDeferred(Array, 7.9999990463257, '', false)
#7 /library/XenForo/Model/Deferred.php(374): XenForo_Model_Deferred->_runInternal(Array, 8, '', false)
#8 /deferred.php(23): XenForo_Model_Deferred->run(false)
#9 {main}

Request State

array(3) {
["url"] => string(36) "/deferred.php"
["_GET"] => array(0) {
}
["_POST"] => array(4) {
["_xfRequestUri"] => string(29) "/admin.php?links/batch-update"
["_xfNoRedirect"] => string(1) "1"
["_xfToken"] => string(8) "********"
["_xfResponseType"] => string(4) "json"
}
}