There may be a few dependent modules like usergroups and user fields that you will have to run before it lets you import users. But basically yeah, just run the modules you want and skip the rest. Guide here with pictures:
As Jake Bunce said, you will need to still improve user groups, user fields, permission settings, and other user dependences ..... But confirmed it can be done
I wouldn't suggest it though. I think it would harm any community no matter where you're importing from or to. But you could do it.