Over the weekend I ended up talking with my oldest friend about some of the people we used to know, hang out with, and game with in the past. While the conversation didn't turn towards thoughts of gaming and the tablecraft of how games work my own head did and it was something I wanted to bring up. Often times we are in games that we want to go one way or be about one thing but they end up not and we're left wondering just what the eff is going on or where we went wrong. The thing is, people are fairly simple creatures when it comes to guiding behavior, especially in games, so if you're game is going awry it might be worth looking into just what your game rewards.
The XP Conundrum
Nearly a decade ago at this point I was in a D&D 3rd ed game that ran around here. The game was fun, and was the only table top gaming I had at the time so it was very special to me. At first the game was just a simple dungeon crawl. The DM had a few dungeons he wanted to run folks through so we did that. However, after completing the first dungeon the DM wanted to do a more RP heavy game with people going for more RP concepts than power/dungeon crawl concepts and playing that. Everyone made characters, but it only took a few sessions before everyone was back to the old "pure power" basics for their character.
The reason? Because even though the GM said he wanted RP the game itself had no reward for RP. We were still playing D&D. We were still doing regular combats based on the Challenge Rating level of the group and all of our XP and loot was based off those encounters with only a very small bonus for good RP. In other words, the people whose RP Concept enabled them to do well in combat still thrived while the more complex RP concepts that sacrificed some power had a greater chance of death or otherwise. Simply put, whatever way the GM said he wanted the game to go the actual fact was he was rewarding playing in the same manner as the dungeon crawl. It was only natural the players would follow suit shortly after.
On the other hand there is the way I hand out XP rewards in my own game right now. I run L5R every other Friday and in general folks get 5 xp for a session, sometimes more depending on the situation. It doesn't matter if the players have combat or not they still get five XP. In essence, the reward here is for showing up to the game and playing. Even then there are flaws. While it hasn't come up yet, technically there are no obvious lack of rewards for someone phoning it in one day as opposed to coming to play. There is no reward for going over the top or really into character. At least...not with XP.
Some game systems don't even use XP this way for rewards. FATE, for example, doesn't do progression with XP but with milestones. Generally the group hits milestones after adventures and story arcs. This means that the reward of character progression comes irregularly and at the end of arcs. The idea is to represent how the story changed a character, and progression itself is more a matter of modifying a character more than a flat growth in power.
However, in FATE there is another reward mechanic to encourage people to play in character, to act towards the narrative, and for making themselves vulnerable. The reward mechanic in question is Fate Points. Fate Points give power to the player. They allow the player to re-roll failed rolls. They allow the player to boost rolls when they need to. And they allow the player to modify the scene or do other things in a way that is favorable to them. However, the only way to get Fate points - aside from a small grant at the beginning of a session - is for your characters Aspects to bring up obstacles and interesting situations that players might otherwise try to reward.
For example, a character could have the aspect "Aggressive Fighter" marking them as someone who stays on offense in a fight and is good at doing that. However, because of that Aspect the GM can offer the player a Fate Point to be aggressive in a social situation, thus turning things negative and making it harder for the group as a whole. Not the kind of thing a player normally wants, but the incentive is that for taking the downturn here the player then has more control in a later situation. The game then gets to follow the normal "narrative flow" of things getting worse and worse with setbacks until the players finally have enough fate points to solve everything.
Finally there are rewards that aren't mechanical in nature, or are more on the periphery. For example, a highly lethal system encourages players to not fight and when they do fight to do so from ambush or with overwhelming force so they aren't taken out on the second turn.
However, there is also encouragement in how easy it is to solve a problem. This is, partially, what leads to standard player tendencies. Why not threaten and cajole your way through everything, keeping violence always an open option, if not only are the consequences for it minimal but it is also the fastest way through the problem? Why not approach a situation directly when it gets results and has no real down side for it coming out?
This doesn't mean to constrain your players, but find a balance. People naturally gravitate to the easiest course of action. They break from this when there is either greater ease or greater reward down another path. This means that the trick to breaking standard player tendencies is to make the other pass more lucrative. Doesn't matter if this is "better" rewards for taking the harder path, more fun for taking the harder path - and fun is always relative. Someone playing a fighter isn't likely to have much fun if they're not getting in fights - than the direct path.