Five Things You Would Think Would Make Players Hate a Villain But Actually Make Them Hate You
1. Steal Their Stuff
This is a classic, and it will absolutely result in some rage, but it has all sorts of problems. First, it's very hard to do without cheating or otherwise making it look like an obvious GM force, and that puts the blame on you. Second, if you steal the character's gear (esp. in a gear-heavy game like D&D) the players will feel like they're being penalized, and they may have to do extra bookkeeping to address the changes in their sheets. This is not to say that nothing can get stolen—plot items, macguffins and loose treasures are all in play—but think long and hard before you take anything that is written on the character's sheet.
Making It Work: To make a theft work, you must spend some time thinking about how to make it recoverable. That means that the villain should not steal things he intends to keep unless you expect things to end with the players looting his corpse. Instead, the villain should be the means by which the item stolen gets transmitted to a new owner or location. This way the villain opens the door to further play rather than just being a black hole that valuable things disappear down.
The balance here is that you definitely want to hit things that are important to the characters and players (and their gear definitely qualifies) but you want the pain to be measured in emotional intensity, not in added bookkeeping.
2. Make Them Feel Stupid
One common situation to encounter when GMing occurs when you present your players with a problem which you can clearly see the solution to, but for one reason or another it completely baffles your players. This is normal enough, but it becomes a problem if the villain then solves the same problem in the way that GM envisioned it. For all intents and purposes this is telling your players they were too dumb to figure out what they "should have done" and then rubbing their nose in it. Not only is this a jerk thing to do, there's a good chance the original disconnect was more your fault than theirs, and that just makes it all the more insulting.
Making It Work: Let your players see the villain have similar frustrations to their own. If the PCs really had to work to get past a challenge, that work really feels rewarded when they see that yes, it really was hard. Hard enough that it’s stopping other people. Plus, every time you show the villain being stymied, you earn a little more credibility in the eyes of your players—if they feel your villain can't just breeze past things with the power of GM fiat then they are less likely to assume you're cheating the next time you do something clever with him
3. Second Verse, Same as the First
The idea of a chain of villains is an old classic—kill off one villain and now you have to face the new, more powerful villain that was behind the first one. This is a good, usable trick, but you need to make sure that each successive villain is distinct. If the new villain is basically just the old villain with more powers, bigger weapons and a more ominous cape then it's not much different than just re-using the old villain, except now you've undercut any sense of victory the players got out of defeating the old villain. Even bringing back the original villain is preferable to introducing a knock-off.
Making It Work: This is pretty easy to fix—just take a little time and flesh out your villains. You absolutely should connect you villains so that the new villain has a motive tied to the old villain, such as avenging him or fulfilling an oath to him, but otherwise make sure he's different. One easy way to underscore that difference is to change the type of villain—if the old villain was a brawler, the new one might be thinky or sneaky. The mechanical changes will emphasize the other changes.
When one of your characters really drills down to excel at something, like a skill or particular combat style, there is a temptation to make the villain better than him, so as to inspire the character to new levels of performance. That works to a point, but remember this: a villain who outshines the character has one purpose—to be defeated by the character in reasonably short order. When that happens it’s pretty satisfying. The problem comes when a recurring villain does this, and is only made worse if he outshines multiple characters. When this happens the GM is basically communicating to the players “It doesn't matter how hard you work, I'm the GM and I can trump your efforts with a whim.”
Making it Work: There are two good solutions to this. The first is to simply have the villain excel in an arena that none of the characters overlap with. That gives him a gimmick that's recognizable, and can create problems the characters are not as able to tackle directly, but which don't make them feel inept. For example, if you have a group that is not at all sneaky, a high stealth villain creates interesting problems.
The second is possibly even more fun, especially for a rival. Pick an area of expertise and make sure the villain is good enough to make the character sweat a little—even if the villain is not quite as good as the character, he should be good enough that the character can never take success for granted. As a bonus, make sure the villain understands and respects their respective level of talent in a way that other people who are less well versed in it do not. If you have seen or read The Princess Bride, the dialog during the big fencing scene is a great example of this.
5. Bulletproof Sheep's Clothing
It is not hard to fool your players. This is not because they are not smart, or because you are particularly clever, but because they trust you to do your job as DM.
This trust means that they are willing to overlook small inconsistencies and make decisions that lead to fun play rather than spend their time studying every detail for the potential twist. One common way that this trust gets abused is to introduce a villain as an ally and then proceed to cause all sorts of bad things to happen to your players while the villain remains hidden in plain sight. Often, this is accompanied by the GM expressing disbelief that his players can't see it when it's "so obvious."
While the breach of trust alone is very bad for a game, this is more subtly toxic over time. Players expect bad things to happen to their characters as part of adventuring, and if the GM is using the hidden villain in this fashion, players are likely to just assume he's cheating (because he is) and that will taint the whole game something fierce.
Making it Work: The purpose of hidden villains is to be discovered, so it is your job to make sure they're found out. This does not mean there won't be complications or that it won't take time, but it needs to be doable. You can start with clues that there is a traitor, and build investigation from there. For a nice twist, you can have the characters know the villain's nature but lack the evidence to convince those around him. Whatever you do, keeping that goal of discovery in mind changes the hidden villain from a way for the DM to feel clever into an engine to drive play.
This idea also has a lot of overlap with the betrayer, an ally who sells out or changes sides. The difference is that the betrayer should be a genuine ally for a time, and when he switches sides, then he plays out much like a hidden villain.