And some people enjoy being at the mercy of others.
Point is - if you are responsible for your server (and it sounds like you aren't) then a panel is a panacea that won't help when the poop hits the fan. I'd rather know how to take care of it directly instead of "oh, my button doesn't do what it should I wonder what's wrong".
I guess when you use a panel it is automatically transfers the knowledge to you on what the different options are?
I realize that Linux CLI intimidates a lot of people... but so did DOS and OS/2 (if you really wanted to do anything with it). Some people are happy living in a protected shell (GUI's).