The editor's smilie menu orders the smilies by id number. You can either remove / re-add the smilies in the order you want. Or you can directly edit the records in the xf_smilie table to change the ids. If you directly edit the records then you need to edit/save a smilie in the Admin CP afterwards to prompt a rebuild of the smilie cache.