Evidence Meter
Sound bytes can be a dangerous thing.
Research conclusions with their many numbers of probabilities have little value without context. Yet so often, these news soundbites sound so specific - specifics lacking context and the limitations documented in the research detail.
The result - misleading hype and noise.
I start with 5 questions below when hearing exploring any claims.
Q 1. What type of research are we looking at? '
Animal? Human? Has it been replicated in Humans? Bio markers or actual health outcomes data? Meta-analysis of multiple studies?
Q2. What was the study time frame?
6 weeks? 6 months? A year? Longitudinal (many years)? Is this something that needs more time to assess mid or long term impact?
Q3. Does the study assess indicators for risk or actual measurable outcomes in humans?
Both have value, but actual measured outcomes is what I always hope to see.
Q4. What else could be influencing the significance between control and intervention groups? What is actually being compared and/or controlled for?
Example 1. Background diets and lifestyles habits can bias results if not controlled for when changing one food item or behavior.
Example 2. Are threshold factors at play? If applicable to the intervention being assessed, one may not see any significant results until the threshold is actually crossed - over or under. While the conclusion may be true, does it paint the full and true picture about the pros and cons for a specific behavior if the discussion about threshold is not addressed (when applicable)? (Here most of us need to rely on analysis by the experts who are aware of these threshold considerations!)
Q5. How meaningful are the "significant differences" quoted? Numbers are easily sensationalized and misrepresented - even when accurate.
Example: Which is greater? A 100% increase of a 1% increased risk of cancer when your current risk is 1% or a 17% probability of acquiring Alzheimer's because you have one APOE 4 allele.
Answer: The Alzheimer's risk at 17% absolute risk is much higher than the 100% increase in the cancer scenario.
-
100% (a doubling) increase of 1% is 2%. Your risk has now gone from 1% to 2%; that doubling is significant but remains low.
-
A 17% probability of acquiring Alzheimer's is 17%. This should be compared to the absolute 2% risk probability identified for cancer.
-
Alzheimer's in this scenarios suggests a serious risk - 8 times the probabiliyt of the 2% cancer risk.
Everyone has their own sensitivity to probability levels. What matters is being clear about the the meaning of the actual risk probabilities so we can make an informed decision.
Q6. What questions were raised by the research?
Example: Why do these findings appear to contradict previous years of research findings? What other emerging research appears to advance or contradict the findings (the hypothesis under investigation)?
Most research triggers more questions to explore to better understand the findings and what else may be at play (other contributing considerations). When discussion of new questions raised is missing, I can't help but wonder what is not being shared.
