The GURPS Basic Set gives some good guidelines in the "Game Mastering" section. There was an important point brought up in the section entitled "Keeping the Characters Alive" that I would like to expand on a little bit. I will excerpt a portion for context: (I think this could be considered "fair use".)
There is a basic contradiction on RPGs. The players are all looking for adventure and adventures are dangerous. On the other hand, nobody wants to get killed! The GM must walk a fine line between a "giveaway" adventure - where nobody is in real danger - and wholesale massacre.
This excerpt makes an important point, one I would like to expand on. All of the players have to play their role, but the Game Master has to play two very large roles (this is seperate from the numerous roles of the NPCs that the GM will have to play as well): 1) the bad guy, and 2) the player's advocate. Thus, we have another basic contradiction.
It is a foregone conclusion that the GM has to play the bad guy. Who else is going to play him? (or them). Moreover, the GM has to play a challenging bad guy. After all, if the villain isn't making the players feel tense, making them sweat a little, the heroes won't have a chance to save the day and be heroic! The GM can't go too far in the bad-guy role however (i.e. ruining the chance for success, giving the players crippling injuries just for fun or killing them off), otherwise he'll end up alienating himself from the players and really bring the whole game down, making it no fun for anyone and discouraging people from playing in the future. So, in terms of walking a fine line, the GM has to play a tough bad guy, but not too tough.
So, on the one hand, the GM has to play the bad guy, but on the other hand, the GM also has to be the player's advocate. He's the one who drops clues for the players to follow, and nudges them (gently!) in the right direction. There really is no substitute for good planning on the part of the GM; the GM should take the time to design an adventure with a flexible storyline that allows the players to drive the plot (rather than the GM trying to drive the players through the plot) and allows several, redundant paths to success. The GM should be mindful of the PCs advantages and disadvantages and use those to help the players find and decipher the clues that will lead them to success (writing the players more salient (dis)advantages down on a crib sheet beforehand can help the GM keep track of the more important (dis)advantages the PCs have).
With the above in mind, let's talk about some things that a GM could do to play the bad guys, without doing undue harm to the PCs.
There is a very controlling style of GMing that goes something like this: "You are standing at the head of a trail. There is an impenetrable wood to your left, a bottomless ocean to your right, and the cliffs of insanity behind you. What do you do?" Occasionally, in such situations, the player might try to test his boundaries with an answer like: "I attempt to forage through the wood," to which the GM replies "Asmodious appears from the dust and points his sword of flame toward the trail." The player might try again to test his boundaries by saying: "Okay, I try to climb the cliffs," to which the GM replies "Orcus appears at the foot of the cliff and points his sword of flame toward the trail."
If you just read this and can't figure out what the GM is doing wrong, you have no business being a GM.
Having just talked about the the importance of avoiding compulsory tactics, once the players have voluntarily walked into the heart of darkness, anything goes. The PCs might just get inside Jabba's desert palace when all of a sudden the enormous portcullis slams shut behind them. They might be escaping from the ape-beasts, running for their canoe on the shore, only to find that the canoe has drifted off with the tide. Be sparing about how many traps you lay, though. If the players are expecting to have poison darts shoot from the walls at them every time they walk into a room, it's going to drag down the adventure to a 1 foot-per-hour drudgery as the PCs meticulously search for traps on every square inch of ground they cover. Making occasional, infrequent, and clever traps can keep the tension up while keeping the tempo up as well.
You don't want the baddies to be complete pushovers, but you don't want them to be deadly foes either. Gague the difficulty level of the enemies according to the skill level of the players. If it's a group of beginning characters, send the small-time thugs after 'em. If it's a group of hardened, experienced warriors, they're gonna have to fight dragons and griffons. Ideally, you want to have baddies that will take off a few hit points and worry the players a little, maybe even make them have to rest and heal up a bit, but don't haul out the supervillain death-knights that will knock them into a coma at the first blow.
If you are not sure how tough to make the baddies (for whatever reason), try feeling out the players' skill level by sending some minor thugs after them. If the players wipe out the thugs in one round, you'll know that the later baddies will need to be beefier. If the players have a tough time tackling the minor thugs, then maybe this should be a non-combat adventure instead...
The PC is flying his P-12 and--after a hard fought dogfight--gets shot down. The plane is spinning out of control and crash lands into a gully. The impact is so tremendous that the PC gets knocked unconcious. As he's blacking out he sees swarms of enemy soldiers coming at him; there's no escape.
So what happens? Do the soldiers feed his bloody body to the dogs? No! They take him to prison, of course! They keep him locked up in there, interrogate him, treat-him-like-the-scum-that-he-is, but above all they give him a chance to heal up and escape! While the PC is inside the prison, he might learn a valuable clue, find what he's looking for, or encounter his evil arch-nemesis and have a duke it out with him. Heck, for all the player knew, getting trapped in this prison was just a part of the adventure that the GM had planned from the beginning! (wink wink)
A failed roll does not have to spell catastrophe. Let's say the characters are traversing a flimsy rope bridge that stretches high above the volcano crater below. They are inching their way toward their final goal on the other side: the Amulet of Anitsirk. One of the characters fails his climbing roll--badly.
What do you do? Have him plummet to his death, at the cusp of his moment of triumph? Have the player lose a character that he's put a lot of time and effort into? Bring down the whole game?
Here's an alternative solution: "Ooo, failed the climbing roll, eh? Let's roll on the 'rope bridge failure table'..." Note: This table does not exist; you just made it up. So you roll the dice behind a screen and--regardless of the roll--you say: "Okay, as you loose your balance, the anchoring ropes on the far side of the crater snap and the entire aparatus swings down and crashes into the side of the crater." At this point there are some potential consequences: everyone must make a ST or DX roll to hold onto the dangling bridge as it crashes into the crater wall; everyone takes damage from the crushing impact; everyone must make a HT roll to resist the effects of the tremendous heat; everyone must make a climbing roll (at a penalty) to get back to the top. At the end of it, the players are bruised, panting, twitchy, maybe even a little upset, but they are not dead. Plus, they now need to find some other way to get to the Amulet of Anitsirk...
The Compendium I lists an advantage called "Extra Life" that is worth 25 points. Let's say a PC gets in trouble -- and by "trouble" I mean the kind of trouble that kills him. If it is obvious that the player really wanted to carry on with that character and felt that his life was cut too short, you could allot the character an Extra Life right on the spot, but put the PC into "character point debt", i.e. any character points the player earns go toward paying off the Extra Life you awarded until it is paid off in full.
It is possible that, even after trying all the techniques listed above, the PCs will execute a sequence of events so colossaly stupid that there is no other reasonable alternative than to kill them. In such a case, kill them, but make it very clear along the way that they're being dumb and when the death knell tolls, make it clear that they've had all the chances they're going to get. You wanna shoot them? Sure. Stab them? Go ahead. Torture them? Okay, within reason. Maim them? This is a touchy one; maiming a character might actually be a fate worse than death as far as the player is concerned. You want to kill them? Not at all, except as a last, completely final resort; you should not kill them at any point short of this final resort.
Here's a little platitude that I think sums up this point nicely: If the players accomplish all the goals of the adventure with resounding success, they should feel like they worked pretty hard to do it. If the players make stupid blunders and get themselves killed horribly, they should feel like they worked pretty hard to do that too.
Something else that the players need to understand is that, just like any other game, it is possible to lose in this game. The stakes can be a lot higher in an RPG, too: if you're playing a board game and you lose a pawn or a draught, it's no big deal, but if you lose your character (especially one that you've spent a lot of time playing) it can hurt. Even if your character doesn't die but does suffer a crippling wound, it can feel like your character's dead (or has suffered a fate worse than death). It's at these moments that the player needs to remember all those lessons we were supposed to learn as kids about being a good loser. It might be a good idea to play out some worse-case scenarios in your mind before something tragic befalls your character so you can be better prepared if the dark day comes.