XF 2.3 What's new with Enhanced Search, Resource Manager and Media Gallery in XenForo 2.3?

Today is the day we sadly have to report that this is the final 'Have you seen...?' thread for XenForo 2.3. While there are still changes planned, these will almost exclusively be changes to existing features, bug fixes and miscellaneous improvements that aren't significant enough for their own thread. But this only brings with it the implication of very good news to come as we start wrapping up the beta process for XenForo 2.3 and rocket towards a stable release as soon as possible.

There's still a lot of work ahead but at the very least we will still be aiming for (almost) weekly Beta/Release candidate releases from this point forwards. Of course our very next release will be Beta 5 and that's what we're here to talk about today. The changes in this release will very much focus on new features added to our suite of official add-ons and improvements to them that have come by virtue of the new functionality we have added to XenForo 2.3 itself.

First, let's take a look at Enhanced Search, shall we?

What's new in XenForo Enhanced Search 2.3?​

Minimum requirements​

For reasons that will become clearer below, we have made a somewhat last-minute decision to increase the minimum version requirement for Elasticsearch even further. We had previously raised the minimum requirement from Elasticsearch 2.0 (released 2015) to Elasticsearch 7.0 (released April 2019). We have now raised the minimum version to Elasticsearch 7.2 (released June 2019). We don't envisage that this would impact many people as we expect that most users who needed to upgrade from a much older version probably upgraded to a much more recent version. However, if you are running Elasticsearch below 7.2.0 please take extra care when upgrading to XenForo 2.3.0 Beta 5 to ensure you're on a suitable Elasticsearch version.

Official OpenSearch support​

Since the Elasticsearch license changes were announced in 2021, and concerns about the impact that might have on the licensing of Elasticsearch now and into the future, Amazon Web Services have launched OpenSearch. This fork of Elasticsearch is an open source distribution with promises of providing ongoing development and support under a more permissive license. Enhanced Search didn't previously officially support OpenSearch but we're pleased to announce that starting with 2.3 this changes.

1713282629384.webp


Search suggestions​

With the slight further increase in minimum requirements brings with it some additional functionality to facilitate "Search suggestions" AKA "Auto complete" for search results! Simply type your query as normal and, if enabled, relevant results (with respect to the title of the content) based on your search query will be displayed:

1713382953347.webp


This also works with any content type constraints you might have set as seen with this example inside the "XenForo suggestions" forum.

1713366339475.webp


And of course this works in the advanced search form too!

1713366758462.webp


This is all made possible behind the scenes thanks to Elasticsearch's search_as_you_type mappings with shingle and edge n-gram filters. The eventual query we run also supports fuzzy matching to provide some tolerance for typos.

The concept of auto completion for search results has also been implemented into the core of XenForo itself. This enables other developers to implement search suggestions with third party search add-ons using other search platforms by simply implementing an autoComplete method in their source handler and indicating their search source supports it.

And for add-on developers in general wishing to add auto complete support to their own content types, that's really easy by simply implementing our AutoCompletableInterface in your search data handlers (we also have AutoCompletableTrait which is optional but recommended for most cases).

Give it a try! Ahead of its appearance in Beta 5 later this week, you can test it out right here. Let us know your thoughts!

What's new in XenForo Resource Manager 2.3?​

There is nothing entirely brand new to show you today in XenForo Resource Manager 2.3 but throughout the development of XenForo 2.3 we have introduced the split-score review graph along with a slew of changes by virtue of the general framework improvements in XenForo 2.3. Let's have a look at them.

Split-score review graph​

We've had this live here since Beta 1 but in case you don't venture over to Resources very often, this is what it looks like:

1713375637482.webp


This gives us an at-a-glance view of the distribution of ratings for each resource. On resource lists, hovering over the rating stars will display the full split graph in a tooltip:

1713377405742.webp



Featured resources changes​

Featuring content in XenForo initially got its debut in XenForo Resource Manager many years ago! With the inclusion of featured content as a generic feature in the core software we have of course now replaced the legacy implementation of featured resources over to the featured content system in XenForo 2.3 and benefits from the updated feature set there.

Image optimization​

Along with the image optimization features in XenForo 2.3, resource icons are also optimized at the point of upload. You can also rebuild the existing resource icons to convert them to WebP either through the admin control panel or through the xf-rebuild:xfrm-icon-optimization command.

Auto-complete search​

The aforementioned search suggestions/auto complete feature which is new in Enhanced Search 2.3 also supports resource items. Here's a specific example which shows resources mixed with thread results, though you can of course, where supported, use the search menu to filter on specific content types and the search suggestions will be constrained to that.

1713382250507.webp


Webhooks​

We have added resource manager events for various resource content types which trigger outgoing webhooks!

1713378933783.webp


To learn more about webhooks, checkout the HYS thread here.

What's new in XenForo Media Gallery 2.3?​

Our primary focus for XenForo Media Gallery 2.3 is performance. Now, the Media Gallery is performant for most use cases, but there's an upper limit in terms of the size of your gallery which can determine just how performant it is. The queries that Media Gallery executes to return media items in various contexts are necessarily complex. We have to take into account a variety of different states including general permissions, category permissions, album privacy settings and individual media and album states.

Over the years we have added features which can mitigate the issues to an extent including the ability to limit the media lists to a maximum time period, for example, only displaying the most recent 12 months of media items.

Aside from the general complexity of the unit of work each query does, one of the biggest performance degradations is caused by MySQL's handling of large offsets. In a recent test consisting of over 45,000 pages of media (1 million media items!) trying to load page 45,252 resulted in an over 30 second delay to load the page.

Deferred joins​

Enter "deferred joins"! This is actually a new feature in the core of XenForo, albeit currently only used within Media Gallery. For the developers amongst you, all it really means is you will call $finder->fetchDeferred() rather than $finder->fetch(), and for the non-developers amongst you, what it means is "slow query go fast" 😉

Behind the scenes what actually happens is rather than executing a single query, it executes two! Which may seem counter-intuitive but bear with me. The first query simply aims to fetch the applicable IDs for the conditions and page you are on which is fast because it uses the primary key, whereas the second query fetches all of the data applicable with all of the joins but matching those specific IDs fetched from the first query.

To cut a very long story short, if you take the page 45,252 example from above, this same query reduced from over 30 seconds to only 2 seconds. While 2 seconds might still seem a little on the high side, this is a 93% decrease from before, and it's only as high as it is because of the high page number. The upshot is that the sheer majority of those 45,000+ pages will now be remarkably faster to load than before. This will not only massively decrease frustration of your users, but it will significantly increase the likelihood of these pages being crawled by search engines and dramatically decrease the load on your database.

What else is new?​

Much like Resource Manager, Media Gallery too benefits from the addition of featured content, image optimization, webhooks and search suggestions.


And that's about all the time we have. We are aiming to have Beta 5 in your hands by the end of this week and we will then continue to implement fixes and improvements with a view to bringing the beta process to a close as quickly as possible. In the meantime, on behalf of everyone at XenForo, we very much appreciate all of the feedback, ideas, bug finding (and patience!) through this process.

We're excited to get this in your hands and continue working towards XenForo 3.0.
 
Last edited:
Click to expand...
Chris D said:
Enhanced Search didn't previously officially support OpenSearch but we're pleased to announce that starting with 2.3 this changes.
Click to expand...
Will you be offering a minimum OpenSearch version requirement for Enhanced Search as well? I'm OK with ElasticSearch (works perfectly for us) but I may give OpenSearch a try on one of our forums to see how well it works.
 
A little disappointed as I expected real improvement and some new features in Resources Manager. However, it’s good to know that we become closer to the stable version :)
 
Chris D said:

What's new in XenForo Media Gallery 2.3?​

Our primary focus for XenForo Media Gallery 2.3 is performance.
Click to expand...

It would be highly appreciated if other performance issues (that are can't be mitigated with deferred joins) could find some love as well:

K

Thread 'FFMPeg handling is very inefficient'

\XFMG\Attachment\Media::onNewAttachment
PHP: 
    public function onNewAttachment(Attachment $attachment, \XF\FileWrapper $file)
    {
        /** @var \XFMG\Service\Media\TempCreator $tempCreator */
        $tempCreator = \XF::app()->service('XFMG:Media\TempCreator');
        $tempCreator->setAttachment($attachment);
        if ($file->getExif())
        {
            $tempCreator->setExif($file->getExif());
        }
        $tempCreator->save();
    }

When this is called, the attachment upload has been completed, the file has been copied to data and the original uploaded...
  • Like
K

Thread 'Thumbnail / Poster handling seems inefficient'

\XFMG\Attachment\Media::onAssociation
PHP: 
if ($tempMedia->thumbnail_date)
{
    $tempThumbPath = $tempMedia->getAbstractedTempThumbnailPath();
    $mediaThumbPath = $container->getAbstractedThumbnailPath();

    $tempFile = \XF\Util\File::copyAbstractedPathToTempFile($tempThumbPath);
    \XF\Util\File::copyFileToAbstractedPath($tempFile, $mediaThumbPath);

    $container->fastUpdate('thumbnail_date', $tempMedia->thumbnail_date);

    \XF\Util\File::deleteFromAbstractedPath($tempThumbPath);
}
if ($tempMedia->poster_date)
{
    $tempPosterPath =...
K

Thread 'Unnesessary query for non-mirrored media items'

In XF\Pub\Controller\Media::actionIndex
PHP: 
$mirrorAttachment = $mediaItem->MirrorAttachment;

This does gall get getMirrorAttachment on the entity which moreless unconditionally issues a query:
PHP: 
public function getMirrorAttachment()
{
    $attachment = $this->Attachment;
    if (!$attachment)
    {
        return null;
    }

    return $this->_em->findOne('XF:Attachment', [
        'data_id' => $attachment->data_id,
        'xfmg_is_mirror_handler' => 1
    ]);
}

This query seems useless for non-mirrored media items as it will never return a result.
Especially if...
K

Future fix Thread 'Media Gallery Index causing slow query'

Code: 
SELECT COUNT(*)
FROM `xf_mg_media_item`
LEFT JOIN `xf_mg_album` AS `xf_mg_album_Album_1` ON (`xf_mg_album_Album_1`.`album_id` = `xf_mg_media_item`.`album_id`)
WHERE (IF(`xf_mg_media_item`.`album_id` > 0, `xf_mg_album_Album_1`.`album_state` = 'visible', 1=1))
AND ((`xf_mg_media_item`.`category_id` IN (X, Y, Z)) OR (`xf_mg_media_item`.`category_id` = 0 AND `xf_mg_album_Album_1`.`view_privacy` = 'public'))
AND ((`xf_mg_media_item`.`media_state` IN ('visible')))

This query is slow as it can't use any existing index in xf_mg_media_item and thus causes a full table scan.

The...
  • Like
 
Chris D said:
Along with the image optimization features in XenForo 2.3, resource icons are also optimized at the point of upload.
Click to expand...
Thanks for the update. Just a quick question about the quote above, is the ability to optimize RM icons a separate option from optimizing post attachments or is it a 'all or nothing' situation?
 
cdub said:
Was hoping for native custom field indexing and filtering. 😥 Maybe in 3.0?

Seems like a great and needed feature.
Click to expand...
Everyone has their own priorities.

For us, support for
  • Synonyms
  • Stemming exceptions
  • Decompounding
  • Exact search / exact hit boosting
  • Better ranking algorithm
would be much more important than custom fields (which I wouldn't even consider needed at all).
 
Nice developments. Will the deferred join optimizations make their way into the core XF (page/thread rendering) too?
 
It might be worth considering more broadly, though deep pagination on threads largely avoids this problem by calculating post positions up front.
 
DagTime said:
please tell me approximately when to expect the xf 3 and the new style?
Sorry!
Click to expand...
There's an app...er...thread for that discussion.

BikeGremlin

Thread 'XF 2.3 / 3.0 release date'

Is there a “pessimistic” timeline (with all the possible worst-case disclaimers) for the 2.3 (or 3.0 for that matter) release?

I plan to start a forum, and would rather wait a month or two than hassle with updates from the start (does that make sense?).

Do you know which PHP version will be optimal (8.1 is for the current version if I understand correctly)?

Relja
  • Sad
  • Like

Of the first party add-ons, I have contemplated getting Enhanced Search. The additional features it added even in earlier versions were nice. And the changes here just add to that. But my shared hosting doesn't have ElasticSearch and moving would be more cost and effort than the site is really worth right now. But something I have filed away for future enhancements.
 
beerForo said:
I like it all I was just hoping the comment editor in MG would finally allow attachments 😭
Click to expand...
This is a good start. I am new here so I am looking forward to the Integration of the 10 "Most voted suggestions" are made real Upgrade.....
For XF / XFMG / & XFRM

Maybe 2.3.1?
 

Similar threads

L
XF 2.1 Adding new permissions to the resource manager and media gallery add-ons
Replies
5
Views
807
Lawrence
L
IgorG
  • Question Question
XF 2.1 Correct upgrade sequence
Replies
3
Views
615
jeb35
J
Top Bottom