Many, many books have been written on this subject, and I am not going to try to cover a significant portion of the topic. My general thinking is that investing in stocks and/or mutual funds that own stocks has been and will be the best and easiest to manage investments over the long-term. That does not mean it should be your only investment, and most likely it won't be. Determining what proportion of your investments should be in equities depends on far too many factors to even try to get into in a brief discussion. So I'll approach part of the question from the other side:
Quite simply, money that you are fairly sure you will need in the not too distant future should not be in stocks or mutual funds that own stocks. Because you will need the money for some purpose (retirement, college tuition, to buy a house, as a cushion if you lose your job, for emergency expenses, ...), you must not take the chance that you may lose a meaningful amount of it and not be able to give the long-term upward tendency of stock prices enough time to recover the losses.
I have seen several interviews with someone who says something like: "In 2000, I had enough in my 401(k) plan to retire in a couple of years, which is what I wanted to do. But the drop in the stock market means that I have to keep working for several years more, and I am afraid I may never be able to retire." Taking this at face value rather than as a gross overstatement that plays well in the media, the best I can say about the person is that he was a greedy fool. Why put your impending retirement at risk? Once he had the assets needed to retire in a few years, he should have moved them out of stocks and into investments that were certain to preserve the needed funds.
That leaves the issue of defining the "not too distant future." There is no one definition that fits all investors. My general guidance is that money in stocks should not be needed for at least five years. The thinking is that will be enough time to capture at least one decent bull market so any near-term losses can be recovered. The most conservative investors could reasonably put or leave money in stocks only if they think it will not be needed for at least ten years. Historically, stocks are almost never down over such a long period. However, "secular" bear markets typically last 15-20 years (and one started in 2000), and in the past it has taken much longer, over 25 years, to recover the purchasing power of stock prices at the start of the long-term period of poor stock returns. Currently (January 2009), stocks as measured by the S&P 500 index are well below the peaks in 2000 and 2007, so it probably won't take that long to recover from more drops over the next few years.
In September 2002 I wrote that given the drop in the stock market over the past two and half years, aggressive investors could reduce their horizon to three years under the assumption that much of the damage of the current bear market had already been done and there will be a strong period of rising stock prices within the next three years. That assumption might have been wrong, which is why the shorter horizon is appropriate only for more aggressive investors, those willing to accept some risks of losses before they will need the money in stocks. As it turned out, the market declines bottomed out about a month later (but I make no claim that I knew that would happen) although a sustained rise in stock prices did not start until March 2003. At current (January 2009) levels, a three-year investment horizon may again be appropriate for the most aggressive investors.