Guidelines Beyond the Sound Bite and Marketing Hype

Friday, January 21, 2011 | 0

By Joe Paduda
CompPharma and Health Strategy Associates

Is medicine science, art, some combination of the two, or something else?

That's not an idle question.

If you're trying to get more scientific about how you practice medicine or what services/procedures/drugs/treatments you pay for, you are likely relying on clinical guidelines to help provide a little more perspective, hopefully one based on something other than best guess or generally accepted knowledge or tribal wisdom.

A recent study may well give you pause - the key finding is rather alarming - many guidelines are NOT based on solid research, but on work that is kindly described as rather more superficial.

Published in the Archives of Internal Medicine, the research found "More than half of the current recommendations of the IDSA (Infectious Diseases Society of America) are based on level III evidence [expert opinion] only." [emphasis added] Note that the research focused solely on IDSA guidelines, which cover a relatively small fraction of all the guidelines in use today. Largely as a result of that conclusion, the researchers concluded "Until more data from well-designed controlled clinical trials become available, physicians should remain cautious when using current guidelines as the sole source guiding patient care decisions."

This isn't exactly new news. This from research on guidelines published in The Journal of the American Medical Association over a decade ago "Less than 10% of the guidelines used and described formal methods of combining scientific evidence or expert opinion. Many used informal techniques such as narrative summaries prepared by clinical experts, a type of review shown to be of low mean scientific quality and reproducibility.18​ Indeed, it was difficult to determine if some of the guidelines made any attempt to review evidence, as less than 20% specified how evidence was identified, and more than 25% did not even cite any references."

The risk here is our sound bite-long attention span will lead some to use these studies to discount guidelines in their entirety, ignoring entirely the "Until more data from well-designed controlled clinical trials become available" recommendation.

Truth is there are lots of guidelines based on standards of evidence significantly higher than 'expert opinion'. The pre-eminent organization in this area, and the one with the most rigorous standards, is the Cochrane Collaboration. And while not all will meet the randomized double-blind control methodology that most believe is the gold standard, many will indeed provide an ample and durable foundation on which to base medical decisions, treatment recommendations, and reimbursement.

With that said, there are organizations that trumpet their 'guidelines' as providing the basis for coverage and payment decisions, when a more-than-superficial examination indicates the 'guidelines' are built on mighty shaky ground.

The Agency for Healthcare Research and Quality maintains a database of evidence-based clinical guidelines; the listing is not comprehensive as many organizations choose to not submit their guidelines for business reasons. However, while not meeting the 'gold' standard described above, the standard employed by AHRQ is far superior to that of "expert opinion only"; AHRQ requirements include "Corroborating documentation can be produced and verified that a systematic literature search and review of existing scientific evidence published in peer reviewed journals was performed during the guideline development." (while their science is solid, they really need to get some English majors involved in the whole writing thing...)

What does this mean for you?

If an organization or vendor is touting their medical criteria or guidelines, prepare - and ask - pointed questions about the methodology, development process, quality of the evidence, and staffing of the effort. The good ones will be only too happy to share their work, and the others will either not know why you aren't impressed and/or be exposed.

A thoughtful piece on ranking the evidence used in medical guideline development can be found here.

Lots more info on guidelines is available here.

<i>Joe Paduda is co-owner of CompPharma, a consortium of pharmacy benefit managers, and owner of Health Strategy Associates, a Connecticut-based employer consulting firm. This column was reprinted with his permission from his Managed Care Matters blog, which can be found here: http://www.joepaduda.com</i>