Generally, no, it shouldn't.
If an add-on was particularly poorly coded it can impact your site, though. A particular scenario involves when a developer changes the database schema e.g adding a new column without setting a default value. That can cause big problems.
There's other scenarios too but that's usually the most common.
In this case it is usually better to uninstall completely -- assuming the developer has actually coded the uninstaller properly...