A lot of people will go to great lengths to publish an RPG. This used to be a much bigger problem in the past, when the singular vision for an RPG might require taking out a second mortgage on your house to pay for a giant print run that wouldn't even faintly sell through. Nowadays, various POD and similar options mean the bar is much lower, but the cachet (and, to be frank, the satisfaction) of producing a "real" book is still very strong.
That's cool, but it's also double edged, because there is a difference between a creation of art and a product.
I am firmly in the camp that believes in celebrating creation. If you put in a lot of work into making something and are brave enough to put it out there for the world to see, that effort merits praise, even if the creation itself is flawed. It's a kind of touchy-feely (and somewhat condescending) position, and I acknowledge that, but the hope is that the creation of a "safe harbor" is worth that. Ideally, it opens the door for deeper conversations than simple praise for creation.
However, once you put that product out for sale, and claim the honor of being "published", then you have sailed out beyond that harbor. Once I can exchange money for your product, it's on an even playing field with any other product I can buy. That is to say, if your creation is a giant MS-Word file dumped into a PDF, that might be praise-worthy as an act of creation, but it's not much of a product.
Now, obviously, this isn't an invitation to be unfair. One needs to be cognizant of the realities of creation - to expect that a one-man-shop can produce something with the polish of a WOTC product is unfair and unreasonable (though it makes it all the more praise-worthy when someone like Daniel Solis does). At the same time, however, this does not absolve a creator of responsibility for covering his or her bases.
When I look at one of these games, I find it important to think about it in terms of the three main ingredients that make a product - money, knowledge and work. Most every element in a game is made of some combination of these things, though some elements skew strongly one way or another (for example, unless you're also an artist, art is a function of money).
Now, this is important because if you're publishing your first game, you probably don't have a lot of money. The reality is also that you probably have less knowledge than you think you do. I don't mean this as a knock, it's just something that I think every creator is familiar with. Nothing teaches you more than your first product. That only leaves work, and work is a tricky one. It's admirable, but in the absence of the other factors, it can be like hitting the gas on a car stuck in first that's out of oil - lots of noise and heat, but little speed.
All this comes together when you judge a product. Even if you can set aside the things which cannot be done because of money, you have to wonder if failure are a result of a lack of knowledge or a lack of effort. This is a key difference because the first inspires some sympathy (we all have been in a similar position) while the second inspires disdain (because the one thing we demand is that you do the work). Of course, that it's not always clear where the failing occurs, but whatever the source, there will be failings, and they're fair fodder for discussion.
Anyway, this is on mind because I've been chewing on the failings of a particular product have run the entire course of this line of thinking, and I've found myself torn between two instincts. The first is to cede the ground to the "Don't be mean" line of thinking and just not discuss it at all. The second is to use it as the basis for illustration of how not to make the same mistakes. That chewing has lead to this post, which has really been me thinking it through.
In the end, I think illustration wins.