Investors have many choices of portfolios to consider, and they need metrics to guide their decisions. A common metric is the Sharpe ratio, which is intended to be a risk-adjusted return measure. It is defined as return in excess of that of cash, divided by volatility. Comparing Sharpe ratios of different investment options is not as simple as it appears, and not as simple as has been presented in the academic finance literature. A high Sharpe ratio is good, and when one portfolio has a higher Sharpe ratio than another, investors need to know whether that difference is substantial enough to act upon. Small differences relative to variance happen by chance and are unlikely to continue, while large differences allow investors to confidently choose one portfolio over another. The statistical properties of Sharpe ratios, however, are not as well understood as they should be. In this paper I demonstrate that the authors of many academic studies of Sharpe ratios have disregarded limitations of the power of tests of ratio differences, causing them to reach erroneous conclusions. I first review widely cited publications about tests of Sharpe ratio differences that are responsible for the lack of general understanding of the difficulties of performing these tests. I then present a better way of analyzing test findings and I use simulations and other analyses to show that when the power of the test is low, then the very best estimators perform no better than random number generators. Investors should be wary of claims by portfolio managers that their Sharpe ratio exceeds the ratio of other managers.