Mike Fara
Active member
As I have been exploring the capabilities and possibilities of using large language models to generate all sorts of things, for instance the capability to generate a simple cosign similarity between two terms or articles, it seems like a huge advancement/leap in what is available to most end users is happening. If you can generate a high or even basic quality embedding (using local machine resources or a large language model), you can apparently incorporate these into elasticsearch. How this can be utilize by elasticsearch is something I am looking into, but apparently it could improve the search quite a bit in terms of finding like terms?
An "AI powered" search if you will. So that is why I am suggesting to give the option to add local embeddings or even OpenAI generated embeddings to the Enhanced Search product. You can generate embeddings using Python fairly easily to messages in any database. The only problem is you have to store those entries somewhere, in another table, or column, or whatever.
I'm not sure if what I'm saying sounds like I'm in outer space with this or if this is something other forum owners would want to utilize.
For example, if I search on a term "NT" and "New Technology" or even "porn" and "sex", with something that uses a sequence matcher, it may not know the similarity. It would not find the similarity. But embeddings would find the similarity...
Thoughts?
An "AI powered" search if you will. So that is why I am suggesting to give the option to add local embeddings or even OpenAI generated embeddings to the Enhanced Search product. You can generate embeddings using Python fairly easily to messages in any database. The only problem is you have to store those entries somewhere, in another table, or column, or whatever.
I'm not sure if what I'm saying sounds like I'm in outer space with this or if this is something other forum owners would want to utilize.
For example, if I search on a term "NT" and "New Technology" or even "porn" and "sex", with something that uses a sequence matcher, it may not know the similarity. It would not find the similarity. But embeddings would find the similarity...
Thoughts?
Upvote
2
