The relationship between story and RPGs is a complicated and often painful one. It's been popping up on my radar lately since I've been hearing "story" used as if it means "Railroading" or "Fudging Dice Rolls" and that bothers me a little. Not that I have any objection to people having strong opinions about specific topics like that - it's just jarring to me to see them lumped under story, a categorization which is almost nonsensical to me.
The thinking is something like this - there are players out there who view RPGs as a means to tell the story. In some cases it's the GM telling a story to the players, in others it is everyone at the table collaborating to create the story. Whatever the model, telling a story is the priority, and unless the rules explicitly support the creation of a story, then the rules take a back seat to the needs of story creation. As such, behaviors like dice fudging, railroading and other tricks are fair game because everyone is on board with the priority of story.
If this was the only role of story in games, I'd understand the confusion. While these "pure story" games are (in my experience) a small but vocal minority, it is easy to buy into their principals and try to treat story as something additive - that by taking one of these techniques like railroading, you can improve your game through the inclusion of story. The fact that numerous published adventures have bought into this idea has contributed to muddling the waters further, but I'll state it as plainly as I can: that is not all that story is. And I'll go a step further to add that's not even the important part of story, at least with regards to games.
"Story" (as differentiated from a story) is a set of techniques used to tell, you guessed it, a story. They're used in every media imaginable, and while some elements are specific to specific media (such as the act structure of television), others seem universal (such as rising tension and the necessity of conflict). This is a huge toolbox, and it can be used for everything from structuring an 27 book epic just as easily as it can a 55 word short story. There are more tools in this box than you will ever use, and no two people's toolboxes are the same. You find things that work or sound interesting, even things which are contradictory, and you add them in over time. Some become well-worn form use, others never come out of the plastic wrap.
So, given this, let me look at RPGs. There is a school of thought that asserts that the GMs role is solely that of referee, which is to say that the GM should effectively just serve as an intelligent, dynamic rendering engine for the setting. In this worldview, the fairness of the GM is the most critical issue - the GM has material (adventures, setting etc) which he needs to present and use as written to the best of his abilities. If there is a gap, he is expected to fill it, but he should hew as closely to the original material as possible, and his creative satisfaction should come from running antagonists and playing existing NPCs.
Setting aside that I would go watch TV or read a book before I'd want to run a game under those sorts of strictures, that thinking simply doesn't hold up outside of tournament play or other specialized environments. GM creativity and judgment calls are not fringe elements to an RPG and there is no way to open that door just a little. And I think that's great, since the alternative is by-the-numbers play, which might make for a decent fight scene in 4e, but makes for an unsatisfying framework.
And this is where we come back to story. See, once the GM starts creating his own material (from a single encounter or NPC, to a huge mega-campaign), he is in a position to start using tools from the story toolbox. This does not mean he's telling a story, rather that he's using elements that make a story interesting to make play interesting. Making characters sympathetic or unsympathetic? Story. Having villains take things from bad to worse? Story. Menacing things that are important to the character? Story. Many of the important decisions and designs that will go into creating satisfying play can be improved by understanding the tools of story.
Now, it's obviously going to be a personal decision on how much this shapes play. You might just use the tools of story to craft a satisfying encounter with a hateable villain, but draw the line at using "story logic" to impact play. For example - the decision regarding if and when the villain flees, and when any meta currency (such as fate or action points) might be spent on his behalf can be made for any number of criteria. The GM might use strict Morale rules, he might have a fixed number at which the bad guy flees, he might eyeball the situation and base it on which way the wind is blowing, or he might opt for a dramatically appropriate moment. Any or all of these can be used to make the decision, but personal taste might remove some of them from consideration.
And that's fine. We all need to find our personal level of preference. But for all that, it is a very rare game indeed that cannot benefit from the toolbox story provides. Certainly, people may use different tools in different ways, or use more or fewer tools, but they're the tools for drama, for making events personal and compelling. I genuinely can't imagine someone wanting to play RPGs (as opposed to other games) and not wanting those things.
1 - All of the games I'm discussing here are going to assume full buy-in from the table. Every single model of gameplay can break down if the table doesn't buy into it, and if things aren't communicated clearly. So for argument, we're going to assume people communicate and think a bit.
2 - A rendering engine is the part of a program that turns all the bits and bytes into pretty pictures, like the terrain of World of Warcraft.
3 - This is, BTW, a reason you end up with killer GMs. If this is the limit of your expression within the game, and that is frustrating, and adversarial relationship can jazz things ups.
4 - I'd play but solely on the understanding that we're playing a boardgame, not an RPG. And I like boardgames fine.