If you were an adventurer, crusader, fighter, mercenary, whatever colorful terms gets your character out and exploring in a fantasy RPG, new enemies would present a challenge. You’ve never seen a bullywug or beholder before (unless it’s in your background that you have), and you’re probably not going to be sure about how it fights, what powers it has, etc.
Now, to a certain extent, forbidding metagaming in these situations is impossible. Players may very well know trolls have an adverse reaction to fire, and they may have read up on what to expect from a bullette. What should be an exciting experience against a new foe instead turns into a two-round kill, as the poor creature suffers Death By Metagaming.
Is there a good mechanical representation to represent the first fight against a new type of enemy? Certainly the easiest solution might be to assign a -1 or -2 to hit the first time a player fights orcs, dragons, flumphs, etc. That penalty drops off next time you fight them. Eventually, you could even mark an enemy you fight the most as favored, with bonuses not unlike that of the Ranger’s ability in D&D 3.5.
I know I’ve seen some folks do something like this around the blogosphere, but for the life of me, I can’t remember where.
Of course, Game Masters should not be afraid to “tweak” stats and add & subtract abilities to make new encounters more uncertain and intense. Players should not have a tactical advantage because they had access to your Monster Manual.