Greets!

This is actually kind of a difficult topic.  While there are plenty of people who are very quick to (rightly) point out that there's nothing stopping you from selling a FOSS game, it's worth noting that it's going to be a lot harder to make a profit if you're giving away the same thing that you're selling.  The real answer, unfortunately, is to not give away quite everything.

There are a couple of ways you could do this:

• Start out propietary and open source later.  This seems to be the most common.  If you release your game for free too early, though, you might risk angering your customers, unless you warn them ahead of time that it'll be open source fairly soon (that said, if you do that, you could lose out on some sales).
• Keep the art proprietary.  This is effective, but as someone who runs an art site for FOSS games, it's not the option that I would personally prefer.  A lot of games that start out proprietary and eventually go FOSS (like the above) do this as well.
• Keep the engine proprietary.  You can always open up your art and media and keep your game engine proprietary.  The down side here is that it's not really open source or free software. :)
• Keep the 'glue' code proprietary.  There aren't that many projects that do this, but I'd like to see it done.  Sell your whole game, and also put the engine source code and the art files up for free download, but sell the 'script' code.  Here, you have the advantage of creating buzz by encouraing people to make free 'fan' levels for your game, but you're still charging people for the full experience.
• Kickstarter.  If you want to release your game completely and immediately as open source, you're going to take a big sales hit, so you'd pretty much need to make all of your money on the Kikstarter campaign (probably $100k+), which is somewhat unrealistic for an unproven company.  I'd love to see this work, but I doubt it's viable.

With all the thinking I've done on this subject, I'm pretty sure those are the only viable options I've come up with.  The holy grail would be to get people to pay for a game that's compeltely free and open, but I have no idea how to go about doing that. :)

Bart

There are other ways as well:

• Ransom-ware add ons: Release a base game. Provide new features / art / missions if the community paid X amount, release the add on. It doesn't matter if one person pays everything or 10 people 1/10th each (or any other amount, as long the total adds up).
• Sell stuff to pay development costs (art books, key hangers, mugs, t-shirts), as long as your game is successful.

You can also add multiplayer mode to your game.

The fairware model might also be worth a try.

That looks interesting. Only not sure about the popup (nag-ware anyone).

Personally I like the fourth option, well even you could make the script opensource but the level data itself would be sold. As a sales mechanism you may even have more than one "premium" level. 

Re: option four, I'd like to mention an example from outside the gaming world. 37 Signals set out to create an online service (and later more), but the underlying framework proved so generally useful they open sourced it. Nowadays it's known as Ruby On Rails, and doesn't need any more introduction.

How to apply that to a game? I don't know. But the engine isn't everything -- parts of it could be useful in isolation. There are supporting tools that may be essential to the creation process. The art could be released in such a way that parts of it can be easily reused, but reassembling it into a complete build of the game is non-trivial.

All these schemes are quite complicated, though, and that's a bad sign. Maybe the question is wrong in the first place?