Only performance drawback I can see is loading those thousands of fields on every page load. Not sure if a multiselect script like the following helps on that front, but it does give you a search bar to make finding what you are looking for easy at least. Requires custom development to use of course.
Select2 is a jQuery based replacement for select boxes. It supports searching, remote data sets, and pagination (infinite scrolling) of results.
select2.org