I have occasionally remarked that for all the incredibly clever and interesting ideas we have managed to develop in the realm of RPG rules, we have let setting be something of the poor cousin in our pursuit of excellence. There are a lot of reasons why this is so, but one in particular keeps bubbling to the surface. We can say with reasonable accuracy whether a set of rules works. It's testable. You consider the goal, run some scenes, and assess it. Maybe you hate it, but it still may do what it's designed to do quite well.
It's much, much harder to assess a setting. Part of that is a problem right on the face of it - setting is going to be even more a function of taste than rules are, and there's no way to get around that. It's easy to like or dislike a particular setting. But more than that, it is much harder to _test_ a setting.
Certainly, there are some rough benchmarks. "Do I get an idea for a game hook on every page of the book?" is a common one, but it's pretty loosey goosey in terms of pinning things down. How well the setting suggests ideas can hinge on a lot of factors, including taste.
Hell, it's hard to even settle on an idea of how setting should be presented. On one end of the spectrum you have entire volumes dedicated to a given setting like the Forgotten Realms or Glorantha, while on the other hand you may have nothing but a roughly sketched out set of ideas and maybe a map. And what's utterly baffling on some level is that both of these models work equally well.
And that, right there, may be the trick of it. For a minute let's set aside the people who buy setting books out of a desire to know more about the setting or a completist urge. What are we left with?
The answer is: play. Play is the eye of the needle which the camel of setting needs to pass through. In any given game the players will see only so much of the world, encounter only so many people, and otherwise explore only so much setting. That amount is often small enough that it's possible to build from scratch, but it's equally easy to shave that portion off a larger block (as in the case of a large setting) for use.
What's interesting is that this doesn't have any relationship to the size of the setting. If the game is world-spanning, players may see many locations, but they will see them very shallowly. The published setting may have an advantage in terms of already having names for those places, and cool maps, but the bulk of the information in that setting is simply not going to ever be seen.
Now, how we arrive at this unit of setting definitely impacts its composition. If it's shaved off a larger whole, there is probably a stronger sense of a "world beyond the horizon". If it's built from the ground up, there's probably a stronger tie between the actions of the PCs and the details of the setting. Both of these are cool, and you can strike differing points of balance between them, but ultimately they're fairly small differences.
With that in mind, I find myself wondering if the trick of setting is not to try to make better or more compelling settings. Neither is it to make more clever rules for generating setting. Rather, I think the trick may be providing more guidelines for how to turn a setting of any size (from a page of notes to the Forgotten Realms) into a game-sized piece.
Obviously, I'm a little late to my own party. In some ways this is one of the purposes of collaborative character creation, especially when it has an explicit setting building element to it (as in the DFRPG). Sitting down and doing this can either build up the elements you want for a game, or pull out the interesting bits from a larger setting based on the interests of the group. This is potent kung fu, but it comes from the rules end of things. What I'm wondering is what we can do with the design of settings to make tools like this more viable.
1 - They're a non-trivial segment of the populace, so I would suggest against ignoring them from a publishing perspective, but that's a whole other discussion.
2 - I feel like there's a soup metaphor here just waiting to be put to use, but I'll resist.
3 - Initially typed that as "Forgotten Reams" and was sorely tempted to leave it as such.