The source code 'inspect' in google worked and got it to the right size, but I don't see any template in Xenforo to actually edit and save.
The code is this:
.redactor_box .redactor_smilies
.smilieContainer {
max-height: 150px;
overflow: auto;
}
Thinking it's something to do with extra.css, but i'm not sure how to enter the code correctly so the smilie box container becomes 500px height instead of 150px currently.