One thing I've noticed about so called 'old school' RPGs is that they liked punishing you for what you didn't know. The most egrecious of these punishments was 'you didn't see the trap, you die.' Even so, there was a more insidious punishment baked into those systems: system mastery. Lack of knowledge of what made an effective character would cost you over the life time of the character. In those olden dayes it wasn't so much of a problem as character mortality was fairly high and the average life of a character comparatively short.
Times change, though. Level caps expand, character survivability rises, and focuses change. That said, system mastery and its passive-aggressive punishment continues. Some games, though, have started challenging it. And I'm not talking low complexity RPGs like Dread, I mean things like Dungeons and Dragons.
It is my firm belief that system mastery should make the game better, without a lack of system mastery making the game worse. My primary example of bad is Dungeons and Dragons' toughness feat from the 3rd edition. This feat was good in literally 1 combination of class and race. It consituted a trap of the worst sort. Now, a note from the developers, and Monte Cook has mentioned this himself, would have prevented this trap from being a trap.
Now, system mastery that makes a game better is like Go, or Chess. Learning to play and playing an even opponent makes for a fun game. But then you learn how to read two turns ahead, then three, and the game only gets more interesting as you begin to reason with the future you're realizing could come about from your actions.
So as I'm sitting down to work on my complex game (The unnamed mecha game) I'm realizing I need to make the choices as obvious as possible as I begin designing the choices each player needs to make. Let the depth of the battle map take the mastery, I want to make character creation as easy as possible.
Another thing to help ease out of the system mastery problem is the 'retraining' that has been built in to some newer systems. That ability to fix mistakes made during creation. 4th Edition Dungeons and Dragons has a good example of this, as you can switch out one feat, power, or skill training each level. This prevents a conditional choice from being a permanent flaw on your character while your buddies get to enjoy their full abilities.
I'm considering taking retraining a step further, what if after every adventure/mission you can retrain your characters ability choices?
You tell me, readers: How much system mastery is too much? And how much retraining is too flexible?