Sure, unless there is a XF2 version of the addon to upgrade to. If You delete it, you will loose any database information that may be associated with the addon.
Say you have a "likes" addon. It is popular in your site, If you delete the addon, then install the XF2 version of the add o , any previous likes members had or have will be gone, you would be starting from scratch.
This is why it is important to evaluate each addon and then decide which ones to delete and which ones to keep.