If you know which one you set XenForo to use in config.php, then I'm assuming it's fine to remove the other one if you're positive that it isn't being used by any other software.
XenForo only uses one database -- the one in config.php -- so the other would have been created through actions outside of XF. I'm not sure what data is in it, but it won't be "connected" to the main database.