Add-on A Look At The Upcoming Public Release of SEO & Index Tools

Nulumia

Well-known member
eCjMDtF.jpg

t1XfgiK.jpg


I'd like to offer an in depth look at the upcoming public launch of SEO & Index tools, an extremely powerful addon which has been in development for over a year.

One important disclaimer:
This article is intentionally verbose, giving very detailed explanations now to gather feedback before information is "compressed" later for convenience. If this is too much for you to read, feel free to skip around!

Author's Note​

I have been working in the SEO space for over 10 years, with much experience in contractual management of SEO campaigns, traffic monitoring, on-site technical improvement, and even site speed optimization. I've been through the trenches of white-hat and black-hat bulletin boards, webmaster forums and other sites dedicated to solving Google's ever perplexing algorithm roll-outs.

Further, I have deep understanding of technical/on-page SEO, from everything down to the nitty gritty of "alt" attributes to how link "title" tags pass Page Rank (PR) onto URLs.

With this background, SEO & Index Tools aims to be the most powerful and advanced SEO addon for Xenforo, hands down. Virtually every idea and method collected from years of experience has been going into this product, with no shortcuts taken.

Overview​

This addon is not a "meta title and description" hack, it is not checkbox to index content or not, or anything other that targets one type of content on Xenforo. It is a comprehensive and deep-reaching system which affects all manner of on-site technical SEO across nearly all types of content, pages, and entities on Xenforo.

What are the major features?
  • Powerful indexing management with advanced rules & criteria
  • Meta tags with Google snippet previews
  • Open Graph management
  • Page title formatting
  • Image and link tag management, alt and title attributes
  • rel="nofollow" handling
  • Deep 3rd-party support for addons and other systems
  • Recommendations
Those are just some of the big ones, but trust me there's much more under the hood. Now let's take a closer look at the highlights!

Index Management​

This is SEO & Index Tool's most powerful functionality with over a year's evolution and work. It adds deep control, rules and criteria to all different kinds of content in Xenforo, with category inheritance.

What is indexing?
Search engine indexing is whether or not pages are included in search results, such as in Google or Bing, or blocked to these search engines. "Noindexing" a page means it will not be included in search results. It also affects your site's ranking and search placement.

Why should you "index" or "noindex" content?
Typically, you don't want "junk" content or pages which offer no valuable text content or keywords, to be shown in Google. Rather, the goal is to only include valuable pages which you want people to click to visit your site. Think threads, your homepage, useful articles, etc. If a user makes a @#$%-post about "Look at this funny cat", chances are you don't want this in Google search results!

Can indexing affect my site's rank?
This is complicated, but YES! At the very simplest, by including tons of junk URLs in your search results, you're diluting the authority & positioning of your important pages. This can ultimately affect the overall Domain Authority of your website itself.

By blocking search engines to junk pages via the "noindex" or "nofollow" directives, you theoretically should cause a bump in the rank of your important content.

Why the need for powerful indexing in Xenforo? Does this already affect me?
By using Xenforo, the answer is already yes. This is not a knock or slight against Xenforo which is a fantastic system, but by nature the platform can produce countless low-value URLs and redirects which can and should be blocked from search engines. Is this a problem with Xenforo? Not really. Remember that the standpoint of this addon is from a hyper-aggressive and in-depth approach to technical SEO, for site owners looking for that power and control. I don't personally believe such an aim rests on Xenforo as a general community platform, especially when so many of these SEO decisions should ultimately be on a case-by-case basis.

Brand New Indexing System​

Right now in Xenforo, we have the ability to noindex threads and posts based on either a toggle "Allow" or "Deny", or by setting some criteria such as "Thread is X days old", "Reaction score", etc. That's it. And outside of threads, there is no further control over indexing our site's content across all manner of pages.

SEO & Index Tools introduces an incredibly in-depth system for index management. First, let's see how it overhauls Xenforo's thread indexing - so it will make sense when explaining how this system extends to other parts of Xenforo!

First, index settings and criteria have been migrated off the 'Forum' entity and onto the 'Node' entity itself, which means other type of nodes such as Categories and Pages are now affected! This alone has huge benefit. Not only can you control indexing of category pages, but the indexing of all content under the category (including subforums) by setting criteria on the category itself! Further, you can noindex Page nodes, which is very crucial for sites with many custom pages for non-forum based content.

Setting criteria on the category? Yup, that means INHERITANCE
I'm personally a big fan of Xenforo's nested set functionality, and use it in several addons. Nested set allows settings for certain Xenforo entities to be extended down child items, such as forums, skipping only when there's a setting already made. A good example of this is "Override style ID" for forums or categories, which extends down all the subforums.

I'm also a fan of automation and making things simple with addons, so index settings are now inherited. For example, you can simply set one of your top-level categories to "noindex", and it will affect all subforums, pages and threads within it! The same applies to whether you've set criteria, such as "Minimum reaction score". That criteria is also passed down the chain of nodes.

Advanced note:
This method uses an "index_criteria" and "effective_index_criteria" combo on the node entity, where the effective value is the definitive after checking parent values
Advanced note #2:
Index settings are copied off the Forum entity during install and translated onto the Node entity where it makes sense to do so. If nothing is set, it will default to the new 'Inherit' value which will typically mean "Index". Settings are back-translated where possible onto the Forum entity on uninstall.

Seeing the index system visually:
For convenience, the addon shows the index rules for each node on the Admin -> Forums -> Nodes list:

XiBPjvH.jpg


As you can see, rules set at the category level are inherited unless given an explicit value.

So what else can we do here?

Extending the index criteria system​

Firstly, I believe that the given criteria are not enough for proper SEO strategy (Minimum reaction score, thread is X days old etc). A new ruleset has been added, "Words per page", which is far more important than the available options:

rccpRRj.jpg


What's more, you can even choose to count the total words per page from either the entire page (all posts counted together!) or by limiting from the first X posts of a thread. SEO & Index Tools will count the words from the message content of posts (not the overall page), after stripping BBCode and HTML tags.

Ok, so that's pretty cool. But what about if you have a large number of nodes? Don't worry, that's been handled.

Global forum criteria​

I said that there's no shortcuts in this addon. And I hate performing repetitive tasks. So you will now have access to global forum indexing criteria which is set in the addon's options:

wXXdq3g.jpg


Now with these handy settings done, we can go back to our forums/nodes and find that there's a new option added, to use the global criteria!

aqK95FE.jpg


This means that after installing the addon, you can simply set your global criteria and then assign all top-level categories to use this setting, effectively taking care of all your forums in a few clicks. No wasting time! There will likely even be a "Rebuild / assign global criteria" entry in the Tools section at release, to take care of this for you!

So we've looked at how indexing is overhauled and extended using the addon. But that's just the beginning..

And now for the big whammy​

Once the overhauled index system was put in place, it has been extended to all other systems on Xenforo (including addons) that use hierarchical content and categories. Yes, this means Xenforo Media Gallery, Resource Manager, Product Manager, XenPorta, shout/chat addons, DragonByte eCommerce, and other systems.

This includes the inheritance system, application of the criteria system onto these entities, category settings, etc, including the visualization of the index inheritance on Admin category listings.

Indexing for Resource Categories:

sXPuJcs.jpg


Criteria also mimics the forum settings, but takes into account differences for the content. For example, Media items will have "Minimum X words" taken from the media description, Products from their descriptions, etc.

But what if you apply pretty good index criteria, but there's still content which meets the rules that you don't want indexed? You can also manually control individual content from the mod menu! For example, here we change a single thread:

sQmBdng.jpg


You can even manage this from the forum list using inline moderation!

2TCiw5O.jpg


Whether Media Gallery photos, products, resources or news articles, SEO & Index Tools applies its powerful indexing features across the board!

Managing indexing via options​

You can also simply switch indexing on or off for various sections via the options page:

ZLLxa2y.jpg

This feature alone can have enormous potential in reducing the amount of low SEO value URLs on your site, particularly when having a large number of members, product or resource update pages, etc.

The Google Snippet Live Preview​

4Nmfq1Z.jpg


The next powerful feature in the addon is the live Google snippet preview, which shows an approximation of how your page will appear in search results, and updates as you type in the fields! By default, the preview will use the regular title and description for the content - whether a forum or category, resource category, XenPorta section.

The live preview also adds two optional fields to all supported content, an SEO meta title and description. This allows you to specify custom content to display in the <head> section of your page (which will be visible to Google and search results), while keeping the regular title and <h1> tag on your page for visitors.

Also, when entering text into the SEO fields, it will override the standard values!

View attachment Sequence 01.mp4

Further, you can supply a custom Open Graph image, which will display to the left of the Google snippet. And of course, several handy labels will change colors from green to red and alert you for optimal character count as you type!

Advanced note:
The advised lengths for optimal title and description are also set in the Options, with default values being recommended lengths by Google). Again, no shortcuts, always full control.

Extending the Google Live Snippet across Xenforo​

But in true fashion to this addon, it doesn't just work for threads. This feature has been extended everywhere where content can allow, including but not limited to the following:
  • Forums
  • Pages
  • Forum list
  • XFRM overview
  • XFRM categories & resources
  • XFMG overview
  • XFMG categories & media pages
  • Chat/shout pages
  • eCommerce software (XR Product Manager, DBTech eCommerce, XFA Marketplace etc)
  • XenPorta overview, categories & articles
  • XenMedio
  • Other news/article addons
You can also edit the SEO fields directly for content (such as threads, resources etc) from the moderation tools, or when editing the item! There's also settings to automatically use the first attachment from threads or resources, product images etc as the Open Graph image, with more support to be added!

Page Title Formatting​

A very important aspect of technical SEO and another focus of this addon, you can control the formatting of page titles by assigning categories, sections and custom suffixes. For example, let's say you have a thread called "My favorite books", and your Admin -> Basic Options -> Board Title is set to "All About Books - A Book Lover's Haven". Your page's <title> tag, along with the tab shown in your browser, would look like this:
My Favorite Books | All About Books - A Book Lover's Haven

This can get a bit cumbersome when your site name is lengthy, and in fact uses a lot of character count and possibly keyword repetition in ALL of your page titles. For example, a handful of your threads could look like this:
Nice Poems | All About Books - A Book Lover's Haven
Best of the month | All About Books - A Book Lover's Haven
Fun reads | All About Books - A Book Lover's Haven

With SEO & Index Tools, you can assign custom suffixes for each area of your site - and even inject the name of the category for SEO boost!

5AXAqMS.jpg


The live preview will show how your browser tab would appear. This can be very handy for using a shortened suffix when you want the page's title to take the most space, or showing the category name.

For example, we could change the thread from earlier to simply:
My Favorite Books | All About Books
..or with the forum category..
My Favorite Books | What Are You Reading | All About Books

Further, you can configure the options to limit the entire page title to a certain amount, ie 70 characters. This method is intelligent enough that it will retain the suffix while cropping the content title with ellipsis (...).

Remember, this addon is all about giving you FULL control.
 
Last edited:

Link Title Formatting​

Another useful feature is the ability to format the title="" attribute of links. This is not only advised from an accessibility standpoint, but is another form of technical SEO.

A properly coded link would appear like this in the HTML:
HTML:
<a href="/forums/cats/" title="A Place To Discuss Cats, For Cat Lovers!">Cats Forums</a>

Without SEO & Index Tools, you have no control over the title="" attribute for links, and in fact, the vast majority of links in Xenforo do not have a title="" attribute. This includes countless popular addons. Now, you can format this attribute nearly everywhere!

pKjNvYq.jpg


In the above example, you could add the title="" attribute and automatically use the thread preview snippet, or by supplying a custom title using variables!

Using the custom method above, you'd find this in links to your threads - whether from the forum list, widgets etc:
HTML:
<a href="/threads/my-favorite-books/" title="My Favorite Books Posted by John S. at What Are You Reading forum">My Favorite Books</a>

Remember, NO SHORTCUTS!​

This, along with the other features, is extended to countless other content types, from media items, resources, to products. And because there are SO MANY occurrances of direct links to content across Xenforo and addon templates, SEO & Index Tools must catch ALL of these and modify them!

That actually sounds kinda crazy..
It is, but it's necessary to deliver results for this feature. And SEO & Index Tools stands by its claim to be the most powerful SEO addon, hands down.

Sitemap Overhauls​

With so many new changes to indexing, it's crucial to rework how Xenforo and addons handle the sitemap. Therefore, sitemap inclusion rules have all been modified to reflect how the addon indexes content. This happens instantly upon install, so you don't have to do anything!

Further, the addon also adds inclusion of many missing types of content not supplied by addon authors, such as categories or other sections.

Other Features​

These are some of the major highlights, but there's plenty more including:
  • Always point paginated content back to the main URL
  • Remove username/avatar links for guests (Block crawl spiders from accessing members!)
  • Full permission system extended across all supported addons, ie:
    • Can set thread meta tags ("Own" or "All" for moderators)
    • Can set resource meta tags
    • Can set media meta tags
    • etc
  • Enable/disable the Google Live Preview (with settings for character limits)
  • Set default Open Graph image to a custom URL, or, let the style decide
  • Use first attachments as the Open Graph image for various content
  • <meta> tag inputs for various Site Verification: Pinterest, Bing, Yandex, Baidu, Facebook Pixel, etc
  • Inputs for tags/scripts in the head and below body areas (no need to edit templates)
  • Enable/disable the indexing preview column in Admin
  • Noindex extra templates (large text field, one per line)
  • Nofollow extra templates
  • Add rel="nofollow" directly to navigation tabs from Admin -> Public Navigation with a simple checkbox!
  • Numerous advanced settings, small fixes, recommended tweaks etc that fix up the nitty gritty in Xenforo

But, this sounds like so much! I don't understand SEO / Where would I begin?​

The Recommendations page will be there to help guide admins not only after install, but will give up to date alerts about recommended settings, changes to make after installing addons, etc.

There's currently no preview for this section as it's highly in development, but the Recommendations will utilize a snazzy UI to group areas of concern together wityh Completed (in green), Warnings (in orange, or Errors/Issues (in red).

This covers everything from whether your board uses SSL, to whether you've configured indexing, to whether Google has access to such and such content.

This section will evolve over time to reflect feature changes and progress in Xenforo.

Summary​

I hope you've enjoyed this early preview, and look forward to releasing the stable public version of SEO & Index Tools. It's been a long road (over a year of solid development) to get it to this point, and it's still nowhere near full form. There's so much that wasn't shown in this sneak peek.

As it stands, this addon's goal is to be the definitive and most powerful SEO solution for Xenforo, second to none. However further usage and feedback will be required to shape the addon's evolution.

Feel free to reply with questions or suggestions!
 
Last edited:

What's planned but not in development?​

There's some cool ideas for later:
  • Front-end scanner for images - detect images with missing SEO attributes, alt text etc
  • Front-end scanner for links - detect broken links, missing SEO attributes etc
  • Google Analytics or Search Console integration (Admin charts/previews)
  • Rule conflict detection
  • Live keyword/topic system - add site keywords and show/recommend them when creating new content (Think Yoast SEO Wordpress plugin)

FAQ​


Q: Why is the addon taking so long?
A: Simply by looking at the features, this may be easy to see. The biggest issue is that this addon extends all functionality across all of the major addons and systems for Xenforo, in being a comprehensive solution. As Xenforo and addons change, development has to stay caught up.

Q: Is it close to release? What's the ETA?
A: The single issue holding public release right now, is finishing extending the new index system. Xenforo 2.2 changed the way threads are indexed, by adding its own criteria and toggles in the Admin. This immediately made similar functionality in the addon irrelevant, so that functionality had to be dropped and morphed around Xenforo's new method. Once this was complete, it had to be extended to all other addons and systems!

Q: Can you release it now and just patch those in?
A: No, as the addon's indexing method makes changes to the database, and I don't feel comfortable implementing the legacy methods for some content, and then making even more database changes later when features are caught up.

Q: Will it include AMP?
A: Unlikely no. For the simple reason that I've worked with @AddonsLab AMPXF, and it's a fantastic product. I don't see competing in mind and marketshare against an existing solution as a good route, and in fact a true AMP solution (such as AddonsLab's) would be far outside the scope of this addon (and price range!)

Q: What versions of Xenforo will it support?
A: This addon is one of my few that will only develop on a running current version of Xenforo, due to the many and rapid changes. It would be impossible to support this addon on older or legacy Xenforo, so a very recent margin of versions will be officially supported at any time (at least for addon updates).

Q: Will it actually improve my site's SEO rank?
A: If you configure it correctly, yes.

Q: Will there be documentation?
A: Yes, full documentation at public release will at least cover the major features. Various settings in the admin have very detailed explanations. Further, option sections for the addon in Admin have handy links to relevant articles and blogs from authoritative sources for the settings at hand.

Q: How much will it cost?
A: Tbd. I am torn between possibly a lot (to justify the insane development needs), or very cheap, out of my desire for everyone possible to have this addon. That is a tough call.

Q: Will it work with X addon or style?
A: There's already comprehensive addon support, and more will of course be implemented with requests. Styles are more tricky, as customizations to templates in a style can cause the addon's automatic Template Modifications to fail. However, this can also be patched.
 
Last edited:
BTW, this looks really promising and is very much needed IMO.

Are you, open to feature requests yet?

Image and link tag management, alt and title attributes
Will there be something that prompts users to add useful alt tags? In fact anything that prompts users to keep their content useful (I like the way Yoast gives you hints on writing, e.g. avoiding over long sentences and/or paragraphs)
 
Last edited:
BTW, this looks really promising and is very much needed IMO.

Are you, open to feature requests yet?


Will there be something that prompts users to add useful alt tags? In fact anything that prompts users to keep their content useful (I like the way Yoast gives you hints on writing, e.g. avoiding over long sentences re paragraphs)
Yes you can give as many suggestions as you want! And yes that's a good idea, I think front end hints and recommendations could be very useful.
 
This additional module seems incredible, a little scary also considering the number of adjustments that will have to be made, or not.
Obviously we would prefer a single checkbox but SEO is so complex!

I wish you very good success and hope, like my comrades, an affordable price, perhaps like AMPXF with a price range depending on the size and notoriety of the customer forum.

Well done to you anyway!
 
Last edited:
If a large addon is to be updated, enhanced, maintained and kept alive, it needs to generate good income.
To elaborate further: Once we would start using this addon, we would be locked in. If for some reason development stopped, anyone who uses this addon would have a significant problem because they would loose all the settings, meta data, etc. It would be a similar situation as when vbSEO or NuSEO pulled the plug back in the vb days. What I mean is that this offers an intrinsic value that once used for some time content and value will be lost.

Therefore its important to know that the addon has a long future. IMHO, offering a large addon on the cheap does not make it likely that the addon has a solid future. Too many addons here on xenforo.com have went the way of the dinosaur.

And I believe that hard work deserves reward and appreciation.
 
Last edited:
Obviously we would prefer a single checkbox but SEO is so complex!
That's true. And any addon which would only add a single or several checkboxes would not be a real SEO solution. I don't mean that against other potential addons as there's a place for free and lightweight options. But rather in promotion of this addon.
This additional module seems incredible, a little scary also considering the number of adjustments that will have to be made, or not.
At the very least the Recommendations page will help guide through necessary changes after install, along with documentation and hints. All recommended settings will also likely be enabled by default.
 
To elaborate further: Once we would start using this addon, we would be locked in. If for some reason development stopped, anyone who uses this addon would have a significant problem because they would loose all the settings, meta data, etc. It would be a similar situation as when vbSEO or NuSEO pulled the plug back in the vb days. What I mean is that this offers an intrinsic value that once used for some time content and value will be lost.

Therefore its important to know that the addon has a long future. IMHO, offering a large addon on the cheap does not make it likely that the addon has a solid future. Too many addons here on xenforo.com have went the way of the dinosaur.

And I believe that hard work deserves reward and appreciation.
Thanks for your opinions and I'm definitely weighing that.

I'd like to mention at this point that the addon has been privately running on a major website for a long time, which has also helped shape its development. It's a lot of work but obviously I'd like the addon to reach a point of stability or plateau after a while where it's easy to maintain. That being said there will always be maintenance to be done.
 
Last edited:
Top Bottom