Where do you think this will all end up? (Google AI Overview)

Garfield™

Active member
Licensed customer
AI bots scan the site, and if the content is organic, natural, and useful, they incorporate it into their system. If it's AI-generated or outdated and not useful to the user, they ignore it. Consequently, they show our organic content at the top of their search results without asking us. I won't even get into the source issue; why would a user bother checking a source site after getting the information they need? Besides, there's practically no source culture in our country.

As a result, traffic to all sites has dropped significantly, some sites have even crashed. Did Google have to do this? In my opinion, yes, because people have started asking AI questions instead of search engines.

In this sense, Google's AI overview has been a lifesaver, and it's not just Google using it; Yandex and others have also started using it.

Two things come to mind here: 1- Useless sites will be cleaned up (even if this happens, AI will continue to feed on large, authoritative sites, nothing will change). 2- All sites will eventually disappear, and then where will AI get its feed from? (Because in this sense, there's no point in putting in effort, spending time, or working on projects anymore; we're currently working on AI, and for free, as volunteers.)

Even if we progress through question-and-answer sessions and user experience interactions in forums, AI takes all of that and displays it.

The situation is the same in organic, categorical forums operating in their own fields; nothing has changed.

For 3-5 months, we've been silently watching like fish out of water, but I have no idea what needs to be done. How sensible is it to invest and produce content? Getting rankings and traffic from Google has become almost impossible; the already monopolized market will become even more monopolized.

Honestly, I don't know what we're going to do or what we should do. I've started thinking about how we'll survive, or should we even survive, in terms of projects, let alone grow.
 

Attachments

  • Screenshot_1.webp
    Screenshot_1.webp
    70.3 KB · Views: 48
I'll just copy + paste my reply to a related discussion on another forum:

A user said:
More and more, zero-click searching is the new trend on Google.

But no matter what happens, if your content helps people with their pain points, that's how you rank. All the other stuff isn't that important.

And I replied:
But it is important. Very important actually.

When users get what they're looking for with zero-click AI summarized results, they never actually end up on your forum. Even when the zero-click result is from your forum.

One forum I Admin is 19 years old and literally the only place most of its info exists, serving a very niche topic. I already see a lot of zero-click AI summarized answers in Google search, all linking to our site.

But as I said, the AI summarized answers mean people have no need or reason to actually click into our site.


We're well on the path of paying for servers, that host info, for AI to scrape & serve up in Google's AI summarized search results. Its not sustainable.
 
Soon, there will be less and less good content, and more and more stupid content.

People with AI will be much more stupid than people without AI. AI content is more and more garbage.

Future is clear. :D
 
For 3-5 months, we've been silently watching like fish out of water, but I have no idea what needs to be done. How sensible is it to invest and produce content? Getting rankings and traffic from Google has become almost impossible; the already monopolized market will become even more monopolized.

Honestly, I don't know what we're going to do or what we should do. I've started thinking about how we'll survive, or should we even survive, in terms of projects, let alone grow.
Depends honestly how you are targeting users. Many of us simply changed strategies. We started targeting user devices and feeds. So that we rely less on Google search and but get clicks from feeds more. Those who are going to work on authority more in coming time will not go down even when AI Overview is coming up. Work on Authority of your site, so that in Google Overview your website is used as source and people do click to check source. Reputation of website matter so work on building partnerships. That is the way forward.
 
AI bots scan the site, and if the content is organic, natural, and useful, they incorporate it into their system. If it's AI-generated or outdated and not useful to the user, they ignore it. Consequently, they show our organic content at the top of their search results without asking us. I won't even get into the source issue; why would a user bother checking a source site after getting the information they need?

For 3-5 months, we've been silently watching like fish out of water, but I have no idea what needs to be done. How sensible is it to invest and produce content?
What I've done quite a while ago already is to aggressively lock out training bots from my forum (but not the user triggered search bots). With Google this is not possible, but it is a start. Locked out the scrapers as well as good as I can. And finally, a couple of months ago, I changed the settings of my forum that w/o being logged in one will only be able to see the first post of any given thread (and in some sub forums like the ones with reference/knowledge base character only the thread title but not the content plus a bunch of sub forums are not visible to guests at all).

I've recently checked what the various AIs have to say about my forum topic when asked through the free, search engine based interface (Google, Bing etc.).

• When asked something like "where do I find the best information about $forum topic?" or similar my forum is always ranked on top and described very positively, basically as authority, including mentioning that, due to scraping AI bots, one has to register to view the full content (which is written prominently on the forum homepage when visiting as a guest). :)
• When asked actual questions around my forum topic the answers are in the range between ok-ish, mediocre, half-right and plain wrong, depending from the question. For an innocent person hard to decide which is true for the actual question asked. Backreference is rarely given (the various AIs differ a bit here) and the sources they mention are more or less random web pages that clearly do have no authority about the topic (and often to even relevant amounts of or correct information) and a lot of reddit. Which is no wonder, given the deal that google has made with reddit. Reddit on the other hand is, regarding my forum topic, a swamp of disinformation and half-true information.

My learning from that was that Google and other AIs recognize, accept and mention the relevance of my forum for the topic and are not shy to elaborate this when asked directly for references or sources of knowledge. So locking them out of most of the content has no negative impact regarding that as they can derivate the authority from the bits they can see. For actual questions on the other hand my forum is, even when used or referenced, not standing out in their answers in a way that anyone would identify it as authority. Rather is it hidden among the lot of strange and mediocre sources that the AIs mention while especially Reddit is pushed quite a bit and mentioned very frequently. But in general the AIs seem more eager to provide what sounds like a comprehensive answer (but is none) while referencing and back linking is more of an alibi and rather hidden in the footnotes than inviting to users to click, no matter what the source may be.

As a consequence I stick to my strategy to lock the AIs out from high quality information as good as I can (so that they cannot exploit the exclusive, high quality content of my forum) while giving them just enough snippets that they can't ignore my forum. This strategy is obviously tied to the fact that I do have loads of exclusive an high quality content that cannot be found elsewhere on the net. I even deliberately do often include "bait" bits in the first posts of a thread I am starting myself but the actual relevant information only starting with the second post of the thread (that the AIs and guests are locked out from viewing). The AIs recognize that as it seems and even honor it to a degree despite being locked out from seeing it. If your forum does not have that level of authority or is rather generic than specific for a niche topic another strategy may work better.

PS: The "first post only" for guests together with the note why this policy is in place led to a dramatic rise in registrations. Many users that have been silently reading in the past have now registered (and a bunch of them do post as well), the majority of registrations do still come via Google.
 
I think it is dumbing us down, and doesn't understand the quality of the source it is quoting. The user also does not think to verify bias of the source that was quoted, and takes it as fact. I see that with political discussions, people thinking because they found it on the search results - it is validated.

We set up a llms.txt on our forum root, to give AI bots some rules for our forum. One of them is requiring attribution, when quoted as a source of information. It has brought us a lot of new members, because of that.

AI puts a real drain on most servers, with intensive crawling. I think the whole thing is still evolving, and hasn't found its true purpose yet.
 
As social sites which many forums are, we are better placed than sites that just contain information, they will be the ones that get gobbled up by google et al and there will be no need to visit them - I am guilty myself of asking AI and not visiting the sources as I am sure many of us are.

The social/community side is what we really need to push on our forums to give people a reason to keep visiting.
 
Last edited:
As social sites which meany forums are, we are better placed than sites that just contain information, they will be the ones that get gobbled up by google et al and there will be no need to visit them - I am guilty myself of asking AI and not visiting the sources as I am sure many of us are.

The social/community side is what we really need to push on our forums to give people a reason to keep visiting.
I had not thought of that, but yes I agree it really be what makes a community out of what might otherwise just be a support site. We now have quiteb a large off topic section with ongoing threads like what are you up to, show us your pets, wildlife, word games etc. It's encouraging to hear you post this and I'll do a bit more work on that side of the forum
 
We set up a llms.txt on our forum root, to give AI bots some rules for our forum. One of them is requiring attribution, when quoted as a source of information. It has brought us a lot of new members, because of that.
That is an interesting one - would you mind sharing it? Do you see that AIs are following the rules you set? I've created an ai.txt a while back - as alms.txt it is a proposed standard:



Hae not checked in a while, but ai.txt was not read even a single time while al kinds of AI bots cent berzerk on my forum (or tried to do so). So while it does not hurt I do not see that it would have any effects.
 
We set up a llms.txt on our forum root, to give AI bots some rules for our forum. One of them is requiring attribution, when quoted as a source of information. It has brought us a lot of new members, because of that.

Interesting... Could you provide more information about how we can do it?
 
Interesting... Could you provide more information about how we can do it?

I actually asked AI to help me create it, and it made a decent example that was very customized for the public areas of my forum. I tweaked it to my liking after consulting sources like:



I did this last month because I was getting bombarded by AI bots, and the sql server couldn't handle it. upwards of 17k ai bots at a time. I researched this in an effort to make them focus more on what they need, instead of scraping like crazy. Then I added CLOUDFLARE and XF BOT GUARD 3 weeks ago, and havent had an issue since. I think all 3 changes helped. :) This is for a forum that completes its 29th year in a few weeks.

An llms.txt file is an emerging, machine-readable standard designed to help Large Language Models (LLMs) and AI agents navigate, summarize, and understand your website's content. Think of it as an AI-focused reading list that directs models to your most valuable pages, bypassing complex navigation menus. [1, 2, 3]

Why It Matters
  • Agent Guidance: It helps AI platforms (like ChatGPT, Claude, and Perplexity) and AI-driven search engines accurately retrieve information about your business, products, or documentation. [1, 2]
  • Generative Engine Optimization (GEO): It acts as an AI equivalent to traditional SEO, ensuring your content isn't missed or hallucinated during AI-powered searches. [1]
  • Efficiency: It reduces token usage and prevents models from having to parse cluttered HTML or bloated navigation. [1]

How to Structure It
The file is written in plain Markdown and is hosted in your site's root directory (e.g., https://yourwebsite.com). A standard file includes: [1]
  1. Title: An H1 indicating the site's primary purpose.
  2. Summary: A short description of your website or business in a blockquote.
  3. Key Links: Grouped H2 sections containing curated, high-value Markdown links. [1, 2]

Advanced Extensions
Depending on your needs, the standard suggests two additional files: [1, 2]
  • /llms-full.txt: A comprehensive, single-page compilation of all your documentation or core content.
  • .md alternative: Appending .md to original URLs to serve a clean, text-only version of a page. [1, 2]

Should You Use It?
Opinions are mixed across the web. While many consider it a low-effort, future-proof step for GEO, some experts point out that major LLMs have not publicly committed to universally supporting the format, and mainstream search engines (like Google) still primarily rely on traditional sitemaps. [1, 2, 3]
You can read the full official proposal or generate templates using the llmstxt.org specification. Popular SEO plugins like Yoast SEO also offer built-in features to generate this file automatically. [1, 2, 3, 4]

Mike
 
Last edited:
Back
Top Bottom