Here we review major study findings, draw conclusions from them, and make recommendations for various parties concerned with health information on the Internet, including consumers and their advocates, health care providers, government policymakers and regulators, and health information providers, such as search engines and Web sites.

Conclusions

Search Engines Perform Differently: Where You Start Matters

One of our findings is that search engines are not interchangeable. The results of a search will vary markedly depending on which search engine is used. Among English-language search engines, we found that the average amount of overlap in the top 10 Web sites identified by a structured search with different search engines was just 11 percent overall; in other words, searches with two different search engines would produce on average only one common Web site in the top 10 listed. This variation results from differences in the way search engines identify relevant material and differences in the methods they use to rank sites.

With the simple search strategies used in this study, search engines were only moderately efficient in locating relevant information on a particular health topic. About 1 in 5 links identified by English-language search engines and 1 in 8 links identified by Spanish-language search engines led to a Web page with content related to the health conditions. More than half of consumers who use the Internet report that they spend about a half hour looking for health information, so efficiency is an important aspect of performance.

Coverage of Important Health Topics Is Spotty

Coverage of topics varies markedly from Web site to Web site and from topic to topic. Even well designed sites that appear comprehensive often have important gaps. For example, Web sites that provide information on the causes of depression may provide little or no information on the side effects of antidepressant medications.

Information Presented Is Generally Accurate

Most sites provide content that expert reviewers rated as mostly or completely correct. Indeed, there appeared to be a much greater chance that covered topics will be addressed accurately than there is that important topics will be covered at all. However, reviewers did find one or more instances of conflicting information judged to be clinically important in over half of their reviews. Consumers who carefully read most or all of the information on a site might well be confused when they encounter such conflicting information.

Information on the Internet Is Commercialized

A substantial proportion of the information that Internet users are likely to find on Web sites is promotional–i.e., it sells products or services but is not clearly labeled as an advertisement. The commercial nature of this material is not always clear, so it may be difficult for consumers to weigh whether or not commercial interests potentially bias the information they encounter.

Health Information on the Internet Is Written at a High Reading Level

The reading grade level of most Web-based health information is high. Material in English is generally written at a college reading level, and will therefore be largely incomprehensible to the many consumers who read at the 6^th to 9^th grade levels, and will be difficult even for those who read at 10^th to 12^th grade levels. One Spanish-language Web site presented health materials at the elementary school level, all others required at least a ninth grade reading level.

Coverage of Spanish Language Health Information Is Sparse and of Poor Quality

The deficiency of important health information on Spanish-language Web sites was particularly striking. Coverage is much less extensive than it is on English-language sites. Over half of the selected condition-related topics were not addressed.

Recommendations

To Consumers Using the Internet

1. The Internet health space is unimaginably vast, and navigating through that space is more challenging than you might think. Although search engines provide the prospect of quick access to information, they are only moderately efficient at identifying relevant content. Therefore, it is important to set aside adequate time for a search, and plan to visit several sites. Our searchers found that even finding material for a single consumer question took 10-15 minutes per Web site with a high-speed Internet connection. As many as four to six sites must be visited to adequately address many questions.
2. Be aware that sites will not necessarily provide a comprehensive picture of what you need to know about a condition. These sites can only supplement consultation with a health care professional. But they may provide you with information that helps you ask good questions and understand much better what your doctor tells you.
3. Do not be surprised if you find information that is conflicting or difficult to understand. Seek out a health care provider to help you interpret what you find.

1. Advocacy groups could help consumers by "adopting" one or two relevant sites and periodically screen content for coverage and accuracy. One could possibly imagine commercial arrangements between e-Health companies (who would produce the content) and advocacy organizations (who would arrange for impartial review by experts).
2. Advocacy groups could also help consumers by pressing for improvements in site content and presentation to make information more complete, accurate, and accessible, and by referring consumers to better sites.

1. Be aware that patients are exposed to a vast and often confusing array of information, which can sometimes be helpful and sometimes a hindrance to the process of providing good care.
2. Professional societies could work on ways to organize physicians, pharmacists or a whole new group of professionals to provide a more formal interpretive function. In the current health business environment, this may require attention to the mechanisms by which such services could be reimbursed.
3. Specialty societies could provide key clinical content to be placed on Web sites. Much in the way specialty societies have become involved in developing and promulgating guidelines, their involvement in writing and approving clinical content on the Web could significantly improve the coverage, accuracy and presentation of the material. Specialty societies should work with advocacy organizations to ensure that the material they develop addresses important patient concerns and facilitates patient decision making around when to seek medical care.

1. Web site content providers could commission clinical panels of experts to review coverage, accuracy, and factual conflicts before material is put out before the public. They could also work with consumer advocacy organizations to make sure that frequently asked consumer questions are addressed and language used is lay-friendly.
2. Provide information at lower reading grade levels on consumer-oriented health-related Internet sites. Currently, the US Department of Health and Human Services recommends that patient education materials not exceed a 6^th grade reading level. This standard should be recommended to English and Spanish language providers of health-related information on the Internet.
3. Provide for systematic review of clinical content by experts, and incorporate a requirement for such review into standards for quality assessments of health-related Internet sites. There are several major private sector efforts to improve the quality of health sites: Health on the Net (HON), Hi-Ethics, eHealth Ethics Initiative, and the AMA Guidelines. Our results support the need for such review, which should be undertaken under the auspices of an independent party that is unrelated to the Internet health information provider and does not provide such information itself.
4. Incorporate readability standards into overall quality assessments of health-related Internet sites. Readability standards are not currently included in quality assessments such as the HON code. Because readability is an integral part of providing accessible health-related information on the Internet, explicit standards should be articulated. Methods of assessing readability should be disclosed. Readability assessments should be made on a regular basis and disclosed to consumers choosing among health Web sites. Assessments should be made on Spanish- and English-language Web sites and the results should be published and disseminated in English and Spanish.

1. The best English-language Web sites are far better than the best Spanish-language sites. Major gains in quality of Spanish-language information (and very likely other non-English language information) can be achieved by translating and culturally adapting what is now available in English.
2. Continue to fund high-quality sites, especially for consumer audiences who are not well served by free market forces alone.
3. Put greater effort into publicizing and increasing access to high-quality government sites.
4. Fund research on effective communication of health information in Web-based format to readers with a wide range of reading levels.

A Concluding Note

A key challenge across all of these recommendations is the extent to which the market for health information will reward those who provide the highest quality material. This is parallel to the problems faced in the health care delivery system–the focus is frequently on cost rather than quality, or on ease of access rather than likelihood of a good outcome. As the business environment for the Internet evolves, it will serve the public well if mechanisms can be found to sort the higher quality from the lower quality information providers. It seems unlikely that market forces alone will be sufficient. The solutions may have to come from cooperation between Foundations and the not-for-profit sector, including consumer advocacy organizations. Ultimately, we will all benefit from the easy availability of better health information.