Yeah that's right. The xf_attachment record should be deleted straight away, then the attach_count will be decreased by one. 99 times out of 100 that is essentially setting it to 0 (as we don't do a lot of attachment re-use at this point).
If the album contains lots of images then a job will be queued to delete the media items in the background.