/* Increase width of smiley dropdown menu */
.xenForoSkin .mceSmiliesMenu div {
width: 398px !important;
}
Change the width to suit.
The smiley dropdown is now populated using divs instead of tables, so if your smilies are different sizes you will have varying quantities on each row.