[MC] Search Analytics 1.1.2

This is primarily a maintenance update focused on performance and bugs, with a couple improvements.

Important change that will affect every add-on:

All of our add-ons are now using Core Library in order to maintain shared helpers and utilities across add-ons for easier maintenance and to keep add-ons cleaner. Core Library also covers our branding for all add-ons in one switch without having to turn it off in every add-on.

This update will require Core Library 1.0.1.

Improvements

• A new admin permission, "Manage search analytics denylist", controls who can edit the denylist. Admins with only the view permission can still see the analytics but can no longer change what gets logged. The Denylist tab is hidden from admins without it.
• Admin lists and the CSV export now show "Peak daily searchers" instead of a figure that added daily counts together.

Privacy updates

• A query now only appears on public widgets once enough different people have searched it. Before, one member searching the same thing several times was enough.
• Members who opt out of public widgets now stay out even when "Attribute searches to users" is off.
• Adding a denylist entry, or a member opting out, now removes the affected queries from public widgets straight away, instead of at the next scheduled refresh.
• Click tracking now follows the same rules as search logging: guest logging, excluded user groups and the denylist.

Performance updates

• The daily rollup now works through large days in batches, so very busy boards no longer run out of memory or hit database limits.
• The admin overview loads its headline figures in one query instead of several.
• Admin query lists run their main query once per page, and skip the count when everything fits on one page.
• A new database index speeds up lists filtered by search type.
• Search engine diagnostics stop at the first failure when Elasticsearch is down, instead of retrying every cluster.

Bugs

• Searches that XenForo answers from its own short-term cache are now counted. Popular queries were undercounted, especially for guests, and the zero-result rate was too high.
• A day that was only partly rolled up is no longer lost if the next scheduled run is delayed by more than a day.
• The backfill job no longer gets stuck retrying a day that fails.
• Clicks are now recorded against the day of the search, not the day of the click, so results pages opened around midnight are counted correctly.
• Guests who see the same cached results page now each have their clicks recorded.
• Clicks can no longer be recorded for positions beyond the number of results.
• Only clicks on a result's title count as result clicks. Clicks on usernames or forum links no longer do, and middle-clicks are now recorded.
• Links in the public widgets no longer count as new searches, so the top searches list no longer boosts itself. Search engine crawlers are no longer logged either.
• The clustering language setting now applies the selected languages. It always used English before.
• Queries that differ only by accents, such as "muller" and "müller", now cluster correctly.
• Cluster counts stay accurate after reclustering and after old data is pruned. Empty clusters are removed.
• Unusually large trending spikes no longer stop the whole trending update.
• Invalid regex denylist entries are now rejected when saved. Before, they saved and then silently never matched.
• A \# typed in a regex denylist entry now works.
• A regex denylist entry that is too expensive to check now blocks the query from being logged, instead of letting it through.
• Paging or sorting a custom date range in the admin lists no longer jumps back to a preset range.
• Invalid or extreme custom dates no longer slow down or time out admin pages.
• Sparklines for a custom date range now show that range.
• The "Top zero-result queries" box and the cluster member list show one row per query instead of one per day.
• The overview charts and headline figures now cover the same days.
• Exact and contains denylist entries now also hide matching queries from admin lists and the CSV export.
• Public widgets now show the full number of queries when some are hidden by the denylist.

We have released a new maintenance update that fixes a critical issue with XenForo 2.3.13. Please update to the latest version before upgrading.

• XenForo 2.3.13 introduces new security measures that caused a fatal error regarding our shared copyright footer.

We have released a major new version with new features and improvements.

New features

• Click-through capture is now supported. Top queries and query details now gain a click-through rate and average click position.
  • Turned off my default, must be enabled manually.
  • Queries with nothing recordered return "No data" rather than 0% and the column doesn't render at all until there's actual click data.
  • It is advised to update your privacy policy if you choose to capture result clicks.
• CSV exportis now supported on zero result queries, top queries, trending, and clusters.
  • Honors the active date window, search type, trending period, and sort.
  • Attribute searcher data is excluded from every report

Improvements

• Dashboard date range now persists when set from any tab, not just the overview

A new version has been released that contains improvements and bug fixes:

• Improvement: Cache the search-type scan so getPresentSearchTypes() doesn't re-run on every dashboard load.
• Improvement: Slight adjustments made to overall responsiveness of dashboard
• Bug fix: Wrap clustering writes in a transaction so an error can't leave clusters half-assigned.
• Bug fix: Clamp all-stopword signatures through the same varchar 64 column path, via a shared clampSignature() helper.
• Reset $inUserSearch after capture so later passes in the same request aren't logged twice.