More Than A Game Mechanic
One of my favorite quotes about the NPCs in a videogame comes from the creator of the Xbox action game Ninja Gaiden. When talking about his game, the creator said "in other games the monsters are there for the player to kill. In Ninja Gaiden they are there to kill the player." It's a simple statement. Stupid really. And yet, as I look at it it has purpose and it tells me a lot about the intention of the game. Ninja Gaiden is a game where you survive and prevail despite the best efforts of the other characters you see in the game. Partially because of this, I took more joy and pleasure in beating Ninja Gaiden than I have in numerous other games of the same genre. Yes, other games were hard or challenging, but in Ninja Gaiden they did everything they could to kill me and that meant something.
Now apply this to your own game. Why are those goblins/bandits/muggers/space pirates actually there in your game? I mean, sure, you need a combat to spice up the night, and you rolled an encounter on the table, and if the PCs don't fight they don't get XP (in some systems) but all of those things can exist without touching on the story part. Are those goblins/whatever there just so the PCs can kill them? Or are they there for something else? Something more?
Less Does Not Mean Bad
To be clear, if those monsters or whatever are there just to give the PCs something to kill that isn't necessarily a bad thing. Whatever your group has fun with is fine. However, as someone who enjoys the power of a story I feel it is a missed opportunity. The reasons those
A Simple Bandit Encounter
Let's go with a simple bandit encounter in your setting of choice. Now, with the simple motivation of "players get to kill them" we end up with a situation where the PCs roll up, the bandits threaten them, initiative is rolled, a fight breaks out, and the bandits fight to the death without a care for how many of their friends have just died to the whirly death blades/bullets of the PCs. It's a tried and true encounter and capable of being fun.
However, what if we thought about it for a bit. So we know we want an encounter with bandits and PCs, right? So let's set it up. What do the bandits want? We'll keep it simple and say money. Why do they want money? So they can eat, and because they're greedy buggers trying to make it rich.
Well alright, now we know why the bandits are there and what they want. The greedy buggers part even explains why they would accost the PCs (who tend to walk around heavily armed in battle armor and stuff like that.) So the last question is: how do the bandits accomplish their goal despite the PCs?
Twisting the Encounter
How many of you came up with the idea of ambushing the PCs? It is a common one and used in games all over. It can work and that's great. But what about other options. Let's go with a fantasy setting and let's say the bandits have control over the only bridge across the river. They're less being bandits now and more being illegal toll takers. But it changes the encounter.
Now when the PCs roll up a bandit comes out and says "two silver each to cross the bridge, or you can lose a day walking to the nearest fording point to the south." The PCs can pay. They can bribe/cajole/threaten. They can fight. We have given the PCs a situation with options open to them to approach it.
If the PCs try to force their way across, maybe the bandits cut the ropes on the bridge and destroy it. Now everyone loses. If the PCs fight, the bandits can fight but they want money so how likely are they to stick around if the fighting goes bad? or maybe the bandits are smart. Maybe when the PCs threaten they let them cross...and then attack the PCs while they're bottled up on the bridge.
Regardless of how you've done it you've taken a simple combat encounter and made it into something more. It has chances for RP. It has room for problem solving. It puts the agency with the PCs. It also makes the bandits into npCs (yay emphasis.)
What About Non-Combat Encounters?
Even with non-combat encounters it works the same. The guy running the store might be there because he runs the store to make a living, and htat's fine. But for quest givers, why are they giving the quest? What does it mean to them? Why is hiring a group of potential murder hobos the solution they've settled on?
Knowing what motivates the character, and keeping it in mind, will make them more than just a random stat block for murder or giving out jobs. It will let them be a character, and one that acts with consistency.
So ask yourself the next time you're about to drop an NPC into your game: why are they there?