I just listened to an excellent Age of Persuasion on the role of villains in advertising. The role is very similar to what it is in fiction and gaming, but it's all a little bit closer to the surface. The villain's role is not to provide opposition (though it may appear so), but rather it is to highlight the attributes of the hero. It's a useful bit of perspective, and certainly speaks to some of the things you want to think about when you design a villain. Specifically, you want to think about what it is you want to showcase - a villain that lets a player show off his character's cool abilities is probably going to be different than one whose actions cut right to the emotional core of the character, though a villain may well be able to do both of these.
Whatever you use them for, villains are tremendously useful in gaming, and with that in mind I want to talk about a few things that are important to remember when you craft up your villains.
Response is Better than Nullification - This is a big one, because it applies equally strongly to the big picture as it does to round-to-round combat. The villain should never make things not happen. In the context of a fight, he should not be immune to the player's capabilities. In the larger context, he should not cancel out a player's goals or motivations. In both cases, the villain's interaction with these things should be dynamic.
The immunity to powers matter is a classic problem in D&D and similar games, and in large part it's because the designers felt that it would be cool to give the players access to abilities like Charm Person and then never let them use it when it matters. This is, of course, intensely lame, but it does acknowledge a real problem. If your game has these potential scene-breaking powers, you need to be ready to deal with it in some way more useful than "nuh uh". Consider that the villain knows that these things are potential options, and consider what contingencies he might put in place to prevent them. If a character can't use his signature Thundering Death Fist because the villain has Death Fist Immunity, then that's lame. If the character can't use the Thundering Death Fist because the villain has put hostages in a glass-bottomed cage over lava, and the thundering would also shatter the glass, that's fantastic. Not only has it added an extra element to the scene, it has demonstrated that the villain respects the character's abilities.
If you find yourself in the position where you have too many of these things to deal with, and your villains are engaging in truly preposterous levels of preparation, then I'll be frank - take a long hard look at your system. If it's so easy for players to steamroller opposition then either you're doing something wrong or the game is.
In the larger context, there is a temptation to have a villain nullify things in the name of drama. It's rare that the GM thinks of it as nullification, but when you decide to kill the girlfriend and stuff her in the fridge you are announcing that
A) You could not think of anything interesting to do with the girlfriend and
B) That you're kind of a tool.
Destroying things seems like a cheap and easy path to drama, but in reality it is far less satisfying than merely threaten an thing. As I said in a previous post, killing someone's dad is lame. Introducing a character who's trying to kill someone's dad is awesome.
Layers Have Limits - I have a great respect for the progression of villains but it needs to be handled carefully. It seems like a trick that makes your life easier - after all it allows for the players to kill more villains and for you to introduce a wider range of villain types- but it's actually much more work for you because you need to make sure that each successive villain creates as much emotional response as the one before it. It is distractingly meta when you realize that you're basically fighting a throwaway villain or two solely because the plot won't let you fight the main bad guy yet. It's just clumsy design. If you are going to do a progression of villains, make sure that the progression is the one the players choose. That is to say, the guy they think should be the next villain damn well better be the next villain.
When Not to Fight - If you're ever stumped for a villain, write one up who the players can crush in a fight. When the Villain is big and tough, it's a pretty easy crutch to fall back on a confrontation, but when the villain is a bureaucrat or a clerk, you need to think much harder about how he can be a real threat, and you end up using much more interesting and sophisticated dangers than new piles of mooks.
Villains are the Hero of their Own Story - This one is old enough to be a chestnut, but it's worth digging out just to remind people. Villains are doing things that make sense to them, and if you can get your head around what they want and what they're willing to do for it, thye come across as much more believable. This is not an invitation to invest heavily in the character so much as a trick to provide verisimilitude.
But That Doesn't Mean They're Not Dicks - It is also easy to get swept up in the sympathetic villain, and while there's certainly some room for that, try to keep it to a minimum. Sometimes, maybe even often, people are kind of jerks without requiring deep motivations or profound plotting. Your villains does not have to be someone who you would want to have dinner with. If you lack for inspiration in this, then just look to all the people who piss you off daily. Yes, putting yourself in their shoes is an important empathetic activity to understand why they thing "no right turn on red" doesn't apply to them, but sometimes empathy just helps you conclude that it's because they're kind of a dick.
The Speech is Real - I've written about this before, so the short of it is this: a good villain has every reason to lay out a good villainous monologue, and it is likely to sound a lot like an Internet rant.
A Villain's Story is Also Called and Adventure - If you find yourself strongly invested in a villain, use that to your game's advantage. Villains are a proactive lot, and they tend to have stories that are something like '[ANTAGONIST] wants [A THING] and [DOES SOMETHING] to get it, overcoming [OPPOSITION] in the process.' Think about your villains story, then think about the places where it might go wrong - "might go wrong" is a fantastic code phrase for "Where players show up". The thing that makes this trick most useful is that if you have a villain who is clear in your head, adventures write themselves. You answer questions about what the villain needs to do to accomplish his goal, what resources he brings to bear, who he hires or recruits, and pretty soon you have all the fixings of an adventure. The only hard part is that you may have to jump through hoops to figure out how a gigantic subterranean lair fits into it…again.
1 - I deliberately say hero rather than protagonist because, unlike in fiction, advertisements about people cannot be as easily pawned off as art.
2 - And a point in 4e's favor is that it's pretty much shed itself of this problem, though certain tradeoffs had to be made for that.
3 - If you're hard up for ideas, steal the contingencies your players use to protect themselves
4- If you do not get this reference, you are one of the lucky ones.
5- The idea that the first villain you meet works for the second one, who works for the third one and so on. This is almost a necessity for D&D or any other level-based game because it allows the villain organization to "level up" with the players. Unfortunately, it also ends up stretching credulity, as you end up with the World of Warcraft problem of Town Guards capable of killing everything on the planet.