The editor's smilie menu orders the smilies by id number.  You can either remove / re-add the smilies in the order you want.  Or you can directly edit the records in the xf_smilie table to change the ids.  If you directly edit the records then you need to edit/save a smilie in the Admin CP afterwards to prompt a rebuild of the smilie cache.