So I looked into this:
Xenforo_Model_Feed::_checkProcessedEntries does not take duplicate entries in the original feed into account:
I've created a patch to fix this:
--- library/XenForo/Model/Feed.php
+++ library/XenForo/Model/Feed.php
@@ -399,7 +399,11 @@
foreach...
When processing an RSS Feed that has duplicate entries with the same id, no duplicate thread is posted but a new thread and post are indexed. This happens everytime the feed is processed.
The result of this is that the search index grows insanely much (2 new index entries for every duplicate...