• "If my doctor recommends an antidepressant medication for the treatment of my depression, how long should I take it for? What should I expect and when will I feel better?"
  
• "What causes depression?"

Examples of questions related to treatments for obesity:

• "Should I consider weight-loss drugs, and if so, what prescription drugs are currently available?"
  
• "Who should consider weight-loss surgery? What are the risks, and how well does it work?"

Based on reviews of the clinical literature, the panels then developed a series of standardized concepts (clinical elements that should be addressed) for the topics and questions. In effect, these concepts represented the kind of clinical information that the panelists thought consumers should find on a site. For example, the clinical elements for depression included the following:

• Antidepressant medications typically begin to work within several weeks. However, many patients do not experience substantial benefits for 4-8 weeks, and it may take 3-4 months before patients taking antidepressants feel better.
  
• Patients with a single episode of acute depression who experience initial improvement should continue to take the medication, usually for 6-12 months after they feel better to keep feeling well.
  
• With antidepressant medicines, many people have some adverse effects early in treatment (in the first 4-6 weeks); most adverse side effects get better in the first month; for some people, the adverse effects can be bad enough to stop the medicine. Common side effects include anxiety, sexual dysfunction, sleepiness, trouble sleeping, weight gain/loss, restlessness, and nausea.

After the topics and questions were developed, two trained searchers visited each selected Web site looking for information related to the consumer-oriented questions. The results from each search were saved and assembled into separate notebooks. All site identifiers were removed, and the notebooks were sent to physician experts to rate the quality of the material. Four rating forms were developed to help standardize the evaluation.

Readability Assessment of Health Information on the Internet. In the third section of the study, we determined the reading grade level at which information provided by these Web sites is written.

We applied widely accepted readability formulas to randomly selected passages of text from both English- and Spanish-language Web sites. These formulas measure grade levels as a function of the average sentence length and word complexity in text samples.

Key Findings

Search engines are not efficient tools for locating health information on a particular health topic.

• Few searches lead to relevant health information. Consumers using English search engines have only a 1 in 5 chance of finding information that is relevant to their search. Performance is even poorer for Spanish-language search engines, where consumers have only a 1 in 8 chance of finding relevant information.

• Search engines take users to different places. No engine is clearly better than another, but where users start matters. Indeed, different search engines rarely take users to the same site. If the lists of 10 Web sites identified by each of two different search engines were compared, only one of the sites would be on both lists.

• Information on the Internet is commercialized. A substantial proportion of the information that Internet users are likely to find on Web sites is promotional--i.e., it sells products or services but is not clearly labeled as an advertisement. About half of the information located by English-language search engines is of this nature. For Spanish-language search engines, about one-fifth of the information is promotional.

Consumers often find incomplete answers to important questions; however, the information that is provided is generally accurate.

• The average English-language Web site lacked information about one in four of the topics that medical experts and consumer advocates thought were important to consumers. More than minimal coverage was available for only half of the topics.

• Health information on Spanish-language sites was sparse and less consistently accurate. On these sites, half of the four health topics had no coverage at all, and more than minimal coverage was found for only one.

• Nearly two-thirds of the English materials list an author and date. About half of the dated materials were updated within the past year. However, only one-sixth of the Spanish materials have dates and authors, and nearly half the materials have neither.

• It is not uncommon for a Web site to contain conflicting information on a clinical topic.

• Coverage varied by topic. For example, breast cancer topics, especially breast cancer screening, were covered significantly more often than all other conditions on English Web sites. In contrast, topic areas related to childhood asthma and obesity were covered significantly less often than the other two conditions on English Web sites. Topics covered least often included symptoms suggestive of poorly controlled asthma and the safety and effectiveness of dietary supplements for treatment of obesity.

• Coverage also varied across Web sites. For example, among the English-language breast-cancer sites, Oncolink.com provided more than minimal, and completely accurate, coverage significantly more often than the average. Five English sites (oncolink.com, cancernet.nih.gov, cancer.org, onhealth.com, and webmd.com) provided at least minimal coverage for 70% of the breast cancer related topics for which we searched. Among the depression sites, nimh.gov performed significantly above average, and six English-language sites (nimh.nih.gov, intelihealth.com, webmd.com, cbshealthwatch.com, onhealth.com and depression.com) provided at least minimal coverage of 50% of the depression-related topics. No Web site performed statistically better than the average for either childhood asthma or obesity.

Most Web-based health information will be difficult for the average consumer to understand.

According to recent health literacy studies, the majority of many health consumer populations cannot understand material written at a 9th grade reading level. This means that most health Web sites require reading skills beyond the abilities of many consumers, especially those underserved populations who are most in need of this information.

• Half of the English-language materials are written at the college level, and all were at least a tenth-grade reading level.

• Forty percent of the Spanish-language materials are written at the college level, almost all were written at least a ninth-grade reading level.

Conclusions

Search Engine Performance. More than half of the consumers who use the Internet report that they spend about half an hour looking for health information, so efficiency is an important dimension of search engine performance. Search engines do provide users with an enormous amount of information, but most of it is not relevant to the search. Overall, just one in five links identified by 10 English-language search engines and one in eight links from 4 Spanish-language search engines led to a Web page containing content relevant to the search.

Quality of Health Information on the Internet. Although we retrieved more than 20,000 pages of material related to the "need-to-know" topics and "consumer-oriented" questions, we found substantial gaps in the availability of key information. For example, less than half of the Spanish-language materials explained that mastectomy and lumpectomy plus radiation are equivalent treatments for early-stage breast cancer. Few sites indicated that a woman with a persistent breast mass and a negative mammogram usually needs further evaluation. If consumers are relying on the Internet to help make important treatment decisions, such deficiencies in information could have serious consequences.

Readability Health Information on the Internet. Much of the health information available on the Internet is beyond the comprehension of many consumers. All of the English Web site documents assessed had materials that required at least a 9th grade reading level, and more than half presented material at the college level. Four of seven Spanish-language sites presented materials at the 9th grade reading level or higher.

Studies of patients in various clinical settings suggest that a 9th grade reading level is too high for most patients. The reading ability of patients and the readability of health-related information on the Internet must be more clearly matched before the Internet can become an effective medium for patient education or have any potential to reduce disparities in access.

Recommendations

These study findings suggest a variety of steps that consumers, providers, and policymakers could take to help the Internet achieve its promise of making critical health information available to all. Based on our analysis, we make the following preliminary recommendations.

To consumers

1. The Internet health space is unimaginably vast, and navigating through that space is more challenging than you might think. Therefore, it is important to set aside adequate time for a search, and plan to visit several sites. Our searchers found that even finding material for a single consumer question took 10-15 minutes per Web site with a high-speed Internet connection.
   
2. Be aware that sites will not necessarily provide a comprehensive picture of what you need to know about a condition. Web-based information can only supplement consultation with a health care professional. But sites may give you information that helps you ask good questions and understand more clearly what your doctor tells you.
   
3. Many sites are created and maintained for commercial reasons. This does not mean that the information you find there is inaccurate, but it is wise to consider the possibility that someone is trying to sell you something by providing this information.
   
4. Do not be surprised if you find information that is conflicting or difficult to understand. Ask a health care provider to help you interpret what you find.

To consumer advocacy groups

1. Advocacy groups could help consumers by "adopting" one or two relevant sites and continually screening their content for coverage and accuracy. One might imagine commercial arrangements between e-Health companies (who would produce the content) and advocacy organizations (who would arrange for impartial review by experts).
   
2. Advocacy groups could also help consumers by pressing for improvements in site content and presentation to make information more complete, accurate, and accessible, and by referring consumers to good sites.

To health care providers

1. Be aware that the Internet exposes patients to a vast and often confusing array of information, sometimes helping, sometime hindering the process of providing good care.
   
2. Professional societies could play a key role in organizing physicians, pharmacists or other professionals to provide a more formal interpretive function. In the current health business environment, this may require attention to the mechanisms by which such services could be reimbursed.
   
3. Specialty societies could provide key clinical content for Web sites. Their involvement could significantly improve coverage, accuracy, and presentation of the material. Specialty societies should work with advocacy organizations to ensure that the material they develop addresses important patient concerns and facilitates patient decision making about when to seek medical care.

To providers of Web-based health information

1. Web site content providers could commission clinical panels of experts to review coverage, accuracy, and factual conflicts before material is put before the public. They could also work with consumer advocacy organizations to make sure that frequently asked consumer questions are addressed and that language used is appropriate for the intended audiences.
   
2. Information on consumer-oriented health-related sites should be provided at lower reading grade levels. Currently, the US Department of Health and Human Services recommends that patient education materials not exceed a 6th grade reading level. A serious effort should be made reach this standard.
   
3. Clinical content should be periodically reviewed by experts, and such reviews should be required in all quality assessments of health-related sites. Reviews should be conducted under the auspices of an independent party that is unrelated to the Internet health information provider and does not provide such information itself.
   
4. Readability standards should also be part of quality assessments of health-related Internet sites. Because readability is an integral part of providing information, explicit standards should be articulated and the methods used for assessing readability should be made public. Readability assessments of materials on Spanish- and English-language Web sites should be made on a regularly basis, and results should be widely disseminated in both languages to help consumers choose sites that will best meet their needs.

To policymakers and regulators

1. The best English-language Web sites are far better than the best Spanish-language sites. Major gains in quality of Spanish-language information (and very likely other non-English language information) can be achieved by translating and culturally adapting what is now available in English.
   
2. Continue to fund high-quality sites, especially for consumer audiences who are not well served by free market forces alone.
   
3. Put greater effort into publicizing and increasing access to high-quality government sites.
   
4. Fund research on how to effectively communicate health information in Web-based format to readers with a wide range of reading levels.

The Internet offers tremendous potential for consumers to easily access important information about their won health problems and those faced by friends and families. This study takes an important first step in conducting rigorous evaluations of Internet medical content. We hope our findings will be used to guide future improvements in the availability and quality of health information on the Internet.