If you have the default avatar system enabled, there's not a way to pick certain colors for it. The color used for the user is pseudo-randomly generated. The color is based on a hash of the username, so the only way to change the color would be to change the username (and even then it's just a different random color). You could of course disable the default avatar system and just use normal avatars that are image based (and whatever image the user uses).
If you really wanted to go down the road of excluding certain colors, you could make an addon that modified the coloring method: XF\Template\Templater::getDefaultAvatarStyling
. That of course will require some coding ability, but then you can make the color logic whatever you want.