Study types explained: from anecdote to meta-analysis

Evidence label: Well studied (as a research method) This label describes the research methods explained here. The studies used as examples keep the labels of the articles they come from, and none of them is a promise of what a practice will do for you.

Author: Raymond Hackland · Last reviewed: 20 October 2026 · Reading time: about 10 minutes

In short

What this is: A walk up the “ladder” of study types, from a personal story to a systematic review. For each step: what it can tell you, and what it can’t. It is part of our series “Reading the Evidence” and follows How to read a health study in 5 minutes.

What the evidence says: Researchers broadly agree on how much weight each design can carry. The Oxford levels of evidence [1] and the GRADE system [2, 3] are the standard tools, and the Cochrane Handbook describes how reviewers judge risk of bias and apply GRADE [5–8]. The ladder is a rule of thumb, not a law. A well-run study low on the ladder can tell you more than a badly run one higher up, and GRADE rates the whole body of evidence for each outcome, not single studies [3]. Our steps are a teaching ladder; the Oxford levels are numbered differently.

What it means for your practice: When you hear “research shows”, the first useful question is “what kind of research?” The answer tells you more than the headline does.

What science says

Label: Well studied (as a research method). Each example keeps the label of the article it comes from.

Every study design is a way of answering one question: did this cause that, or could something else explain it? Designs differ in how well they rule out the “something else”. Here they are, from the bottom of the ladder to the top.

Step 1: Personal experience and anecdote

What you notice in your own practice, and what other people tell you about theirs. On this site that has its own label, Lived experience: it “describes what I notice in my own practice and what participants in my immersions consistently report”, and it “always appears beside an evidence label, never instead of one” (Our labels).

• What it can tell you: what a practice is like from the inside, what people value about it, and which questions might be worth studying.
• What it can’t tell you: whether the practice caused what was noticed. On a good day, many things change at once: rest, mood, expectation, the group, the season. Many complaints also ease on their own over time. A story can’t separate the practice from any of these.

Label of this step: Well studied (as a research method) · 🟣 Lived experience (what the label means).

Step 2: Case reports and case series

A case report describes one person in detail; a case series describes a handful. They are written by clinicians or researchers and published, which makes them more structured than an anecdote. A review of the genre lists their strengths as spotting something new, generating hypotheses, and in-depth description. Its limits are that the results can’t be generalised, cause and effect can’t be established, and there is a real danger of over-interpretation and publication bias [11].

The humming literature has a clear example: a published case report in which one person with severe chronic sinusitis hummed strongly for one hour at bedtime on the first night, and then 60 to 120 times, four times a day, for the next four days [16]. Our article Humming and nitric oxide in the nose cites it only to show why it is not evidence of benefit. One person, no comparison, and an author who also proposed the explanation.

In the Oxford levels for the question “does this intervention help?”, case series sit on level 4 of 5 [1].

Label of the example: Early evidence (humming and nasal nitric oxide).

A side step: laboratory and mechanism studies

Many headlines rest on studies that measure a mechanism rather than an outcome. In the 2002 humming study, ten healthy volunteers had their nasal nitric oxide measured. It was 15 times higher during humming than during quiet exhalation [17]. That is a careful, controlled measurement, and the result is real. But it measures a gas in the nose, not health. If you argue from it to a health benefit, you are using what the Oxford levels call “mechanism-based reasoning”, which sits on level 5, the lowest step for questions about whether a treatment helps [1].

Label of the example: Early evidence (humming and nasal nitric oxide).

Step 3: Cross-sectional studies

A snapshot: researchers measure many people once, for example comparing long-term practitioners with non-practitioners. Such studies are quick and good for describing how common something is, but they “do not permit distinction between cause and effect” [12].

Suppose a survey found that long-term Qigong practitioners had better balance than non-practitioners. That could mean practice helps balance. It could also mean that people with good balance are more likely to keep practising, or that practitioners differ in other ways (age, activity, income). Tai chi researchers have suggested cross-sectional studies of long-term practitioners as one design among several. One challenge they name is that short trials may not reflect what long-term practice does [15].

Step 4: Cohort and case-control studies

These are observational studies that follow time.

• A cohort study follows a group forward and records who develops an outcome. Because events are measured in order, it can help tell cause from effect [12].
• A case-control study starts at the other end. It takes people who have an outcome and similar people who don’t, and looks back at what differed. That makes it useful for rare outcomes and for generating hypotheses [12], but it relies on records or memory.
• What they can tell you: long-term patterns, rare events and harms, things a trial could never ethically assign.
• What they can’t: rule out confounding. People who choose to practise Qigong may differ in many ways from people who don’t, and no statistical adjustment can catch every difference.

GRADE therefore starts a body of observational evidence at “low” certainty, and randomised trials at “high” [3]. It can rate observational evidence up in special cases. The main reason is a large effect: in rigorous studies, at least a two-fold change in risk (a doubling or halving) can raise the rating by one level, and a very large effect, at least a five-fold change, can raise it by two levels. Other reasons are a dose-response gradient, or confounding that would, if anything, hide the effect [4].

Step 5: Non-randomised controlled studies

Here researchers do compare a group that practises with a group that doesn’t, but people are not allocated by chance. They might choose their own group, or the comparison might be last year’s patients. This is a real improvement on a single group measured before and after. But the groups may still differ from the start (“confounding” and “selection bias”). The Cochrane Handbook has a dedicated chapter on these studies and recommends a dedicated tool for judging them, ROBINS-I, developed by Sterne and colleagues [7, 9]. When this tool is used, such evidence starts at “high” but is generally downgraded by two levels for exactly those reasons, so it usually ends up at “low”, just as it would without the tool [5].

Step 6: Randomised controlled trials (RCTs)

In an RCT, chance decides who gets the practice and who gets the comparison. On average this balances everything that might affect the result, including things nobody thought to measure. That is why randomised trials start at “high” in GRADE [3]. They still have to be done well. Cochrane’s risk-of-bias assessment asks, among other things, how the randomisation was done, whether people actually received the intervention they were assigned, whether data are missing, how outcomes were measured, and whether results were selectively reported [6].

Two things are especially hard in mind–body and Qigong research:

1. Blinding. In a drug trial, a placebo pill can hide who gets what. In a Qigong trial, you can’t practise without knowing it. Researchers have described tai chi as a “complex, multicomponent intervention” whose physical, cognitive and ritual parts make it hard to separate specific from non-specific effects and to design a valid sham control [14]. This matters because a large analysis of 146 meta-analyses found that, in trials with subjective outcomes, lack of blinding was linked with exaggerated effect estimates, while there was little evidence of such bias for objective outcomes [13]. Breathlessness and quality of life are subjective; walking distance is measured, and the person measuring can sometimes be blinded.
2. The choice of control. “Usual care” or a waiting list tells you whether practice plus attention beats no extra attention. An “active” control, such as another gentle exercise or a health-education class, gets closer to the question of whether this practice matters. In the 2020 review of Liu Zi Jue trials in chronic lung disease (COPD), 14 randomised trials with 920 patients were included, and the control groups received usual care. The authors name methodological bias and the placebo effect as reasons why further research is needed [18].

Label of the example: Early evidence (Liu Zi Jue).

Step 7: Systematic reviews and meta-analyses

A systematic review sets its question and methods in advance. It searches for all relevant studies, judges each one’s risk of bias and summarises the results. A meta-analysis adds the statistics: it pools the numbers into one estimate with a confidence interval. Cochrane reviews follow especially strict methods. The Cochrane Handbook opens its chapter on meta-analysis with the words “Do not start here!”, a reminder that pooling only makes sense after the question, the search and the quality checks are right [8].

• What they can tell you: the overall direction and size of an effect across many studies, and how consistent it is.
• What they can’t: fix the studies they include. Pooling many small, unblinded trials gives a precise-looking estimate that may be precisely biased. Several reviews of the same topic often share the same trials, so three reviews are not three independent confirmations. Our article Qigong and tai chi: what systematic reviews say discusses this for Liu Zi Jue, where three reviews with overlapping search periods probably share many trials.

Some methodologists therefore suggest seeing systematic reviews not as the top of the pyramid but as a lens through which the other studies are viewed [10].

How GRADE turns all this into a certainty rating

GRADE rates a body of evidence for one outcome, in four levels: high, moderate, low and very low [2, 3].

• Starting point: randomised trials start high, observational studies low [3].
• Rating down, usually one level per problem, or two levels for very serious problems: risk of bias, inconsistency between studies, indirectness (the studies don’t quite match the question), imprecision, and publication bias [3, 5].
• Rating up, mainly for observational evidence: a large effect, a dose-response gradient, or when plausible confounding would, if anything, reduce the apparent effect [4].

The same review can reach different certainty ratings for different outcomes. In the Cochrane review of exercise for preventing falls in older people living in the community, tai chi reduced the number of people who fell by about 20%: risk ratio 0.80, 95% confidence interval 0.70 to 0.91, from 8 trials with 2,677 participants, rated high certainty. Tai chi may also reduce the rate of falls by about 19%: rate ratio 0.81, 0.67 to 0.99, from 7 trials with 2,655 participants. This result is rated only low certainty, which is why the review says “may” [19]. One practice, one review, two outcomes, two levels of certainty.

Label of the example: Well studied (tai chi and falls; see Qigong and tai chi: what systematic reviews say).

The ladder on one card: – Personal experience: tells you what it’s like, not what caused it. – Case report or series: spots something new; can’t show cause. – Lab or mechanism study: shows how something might work; not an outcome. – Cross-sectional: a snapshot; can’t tell which came first. – Cohort or case-control: follows time; can’t rule out confounding. – Non-randomised controlled: has a comparison; the groups may differ. – Randomised trial: chance balances the groups; blinding and controls still matter. – Systematic review or meta-analysis: the overview; only as good as its studies. – GRADE: how sure we can be, one outcome at a time.

One study, wherever it sits, is a starting point. Why that is so, with sample size, replication and p-values in plain words, is the subject of the next article in this series, Why one study isn’t proof (coming soon).

What tradition says

Label for this section: Tradition only.

The classical texts and the teaching passed from teacher to student are a different kind of knowledge, and they don’t sit on this ladder at all. They record what was practised, taught and believed, and they are studied with the tools of history: dating, authorship, comparing editions. Our articles Liu Zi Jue: where the Six Healing Sounds come from and What is Qigong? An introduction treat them that way. A text saying that a sound “governs” an organ is evidence of what its author taught, not a measurement. Equally, the fact that research hasn’t tested a traditional idea doesn’t make it worthless: “Tradition only” is a statement about evidence, not about value.

Where popular claims go beyond the evidence

Label for this section: Speculative.

“Thousands of years of practice is the biggest study ever.” Long use shows that people valued a practice. It is not a comparison: nobody recorded who would have done as well without it, and practices can survive for cultural reasons. Long history belongs to tradition, not to the evidence ladder.

“A meta-analysis proved it.” A meta-analysis is only as good as the trials it pools [8]. Check the certainty rating, if the review gives one, and whether the trials were small or unblinded.

“Only randomised trials count; everything else is worthless.” Also too simple. Case reports spot new problems and harms [11], and observational studies can study things trials can’t [12]. Each design answers some questions well and others badly.

“It makes sense biologically, so it must work.” A plausible mechanism is the lowest step for questions about whether something helps [1]. Many plausible mechanisms have not held up in trials.

“My experience proves it works.” Your experience is real and worth taking seriously. On this site it carries the Lived experience label. But it can’t separate the practice from everything else that changed at the same time.

How this relates to the Six Healing Sounds, Qigong and neigong

Label for this section: Early evidence (research on Liu Zi Jue) · Tradition only (qi, organ pairings, neigong concepts) · 🟣 Lived experience (my practice and participants’ reports).

Here is where the topics on this site sit on the ladder:

• Humming and nasal nitric oxide: small laboratory studies, of which we cite one here [17] (our article Humming and nitric oxide in the nose covers the others), plus one case report [16]. Mechanism level; no health outcomes.
• Liu Zi Jue in COPD: a systematic review of small randomised trials with usual-care controls, in one patient group, studied alongside medical care [18]. Early evidence.
• Tai chi and falls in older people: a Cochrane review with high certainty for one outcome and low certainty for another [19].
• Qi, the organ pairings of the six sounds and the inner work of neigong: traditional concepts. They aren’t on the ladder, and as usually stated they are probably not testable.
• My own music-assisted practice: no study has tested it. What I notice in my own practice and what participants tell me is Lived experience: honest, worth hearing, and a source of good questions, but not a finding. That is why every article here keeps the two apart.

If you have a health condition, please read Who should be careful with breathing practices before you start.

Resonance Heritage Qi is a personal, music-assisted practice inspired by Liu Zi Jue: connected to it, not a replacement for the traditional form or for study with a teacher of that tradition. No study has tested it, and we make no claims about its effects.

Enjoyed this article? Get my free practice guide to the Six Healing Sounds. Just sign up below, confirm your email, and it’s on its way.

Sources

1. OCEBM Levels of Evidence Working Group (Howick J, Chalmers I, Glasziou P, Greenhalgh T, Heneghan C, Liberati A, Moschetti I, Phillips B, Thornton H, Goddard O, Hodgkinson M), 2011. The Oxford 2011 Levels of Evidence. Oxford Centre for Evidence-Based Medicine. https://www.cebm.ox.ac.uk/files/levels-of-evidence/cebm-levels-of-evidence-2-1.pdf
2. Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, Schünemann HJ; GRADE Working Group, 2008. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ 336(7650):924–926. https://doi.org/10.1136/bmj.39489.470347.AD
3. Balshem H, Helfand M, Schünemann HJ, Oxman AD, Kunz R, Brozek J, Vist GE, Falck-Ytter Y, Meerpohl J, Norris S, Guyatt GH, 2011. GRADE guidelines: 3. Rating the quality of evidence. Journal of Clinical Epidemiology 64(4):401–406. https://doi.org/10.1016/j.jclinepi.2010.07.015
4. Guyatt GH, Oxman AD, Sultan S, Glasziou P, Akl EA, Alonso-Coello P, et al.; GRADE Working Group, 2011. GRADE guidelines: 9. Rating up the quality of evidence. Journal of Clinical Epidemiology 64(12):1311–1316. https://doi.org/10.1016/j.jclinepi.2011.06.004
5. Schünemann HJ, Higgins JP, Vist GE, Glasziou P, Akl EA, Skoetz N, Guyatt GH. Chapter 14: Completing ‘Summary of findings’ tables and grading the certainty of the evidence (last updated August 2023). In: Higgins JP, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al. (eds). Cochrane Handbook for Systematic Reviews of Interventions, version 6.5. Cochrane, 2024. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-14
6. Higgins JP, Savović J, Page MJ, Elbers RG, Sterne JAC. Chapter 8: Assessing risk of bias in a randomized trial (last updated October 2019). In: Cochrane Handbook, version 6.5, 2024. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-08
7. Reeves BC, Deeks JJ, Higgins JP, Shea B, Tugwell P, Wells GA. Chapter 24: Including non-randomized studies on intervention effects (last updated October 2019). In: Cochrane Handbook, version 6.5, 2024. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-24
8. Deeks JJ, Higgins JP, Altman DG, McKenzie JE, Veroniki AA (eds). Chapter 10: Analysing data and undertaking meta-analyses (last updated November 2024). In: Cochrane Handbook, version 6.5, 2024. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-10
9. Sterne JA, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, et al., 2016. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ 355:i4919. https://doi.org/10.1136/bmj.i4919
10. Murad MH, Asi N, Alsawas M, Alahdab F, 2016. New evidence pyramid. Evidence-Based Medicine 21(4):125–127. https://doi.org/10.1136/ebmed-2016-110401
11. Nissen T, Wynn R, 2014. The clinical case report: a review of its merits and limitations. BMC Research Notes 7:264. https://doi.org/10.1186/1756-0500-7-264
12. Mann CJ, 2003. Observational research methods. Research design II: cohort, cross sectional, and case-control studies. Emergency Medicine Journal 20(1):54–60. https://doi.org/10.1136/emj.20.1.54
13. Wood L, Egger M, Gluud LL, Schulz KF, Jüni P, Altman DG, Gluud C, Martin RM, Wood AJ, Sterne JA, 2008. Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ 336(7644):601–605. https://doi.org/10.1136/bmj.39465.451748.AD
14. Wayne PM, Kaptchuk TJ, 2008. Challenges inherent to t’ai chi research: part I – t’ai chi as a complex multicomponent intervention. Journal of Alternative and Complementary Medicine 14(1):95–102. https://doi.org/10.1089/acm.2007.7170a
15. Wayne PM, Kaptchuk TJ, 2008. Challenges inherent to t’ai chi research: part II – defining the intervention and optimal study design. Journal of Alternative and Complementary Medicine 14(2):191–197. https://doi.org/10.1089/acm.2007.7170b
16. Eby GA, 2006. Strong humming for one hour daily to terminate chronic rhinosinusitis in four days: a case report and hypothesis for action by stimulation of endogenous nasal nitric oxide production. Medical Hypotheses 66(4):851–854. https://doi.org/10.1016/j.mehy.2005.11.035
17. Weitzberg E, Lundberg JO, 2002. Humming greatly increases nasal nitric oxide. American Journal of Respiratory and Critical Care Medicine 166(2):144–145. https://doi.org/10.1164/rccm.200202-138BC
18. Xiao L, Duan H, Li P, Wu W, Shan C, Liu X, 2020. A systematic review and meta-analysis of Liuzijue in stable patients with chronic obstructive pulmonary disease. BMC Complementary Medicine and Therapies 20:308. https://doi.org/10.1186/s12906-020-03104-1
19. Sherrington C, Fairhall NJ, Wallbank GK, Tiedemann A, Michaleff ZA, Howard K, Clemson L, Hopewell S, Lamb SE, 2019. Exercise for preventing falls in older people living in the community. Cochrane Database of Systematic Reviews 1:CD012424. https://doi.org/10.1002/14651858.CD012424.pub2

Source quality note: – [1]–[9] are standard methods references in evidence-based medicine: the Oxford levels table, three papers by the GRADE Working Group, four chapters of the current Cochrane Handbook and the ROBINS-I tool paper. [10] is a short commentary on the evidence pyramid. – [11] and [12] are narrative reviews of study designs. [13] is a large meta-epidemiological study (146 meta-analyses, 1,346 trials). [14] and [15] are methodological discussion papers on tai chi research. – [16]–[19] are examples already discussed in our other articles: one case report, one small laboratory study, one systematic review of small trials and one Cochrane review. – We checked [2]–[4] and [9]–[19] in their PubMed records and abstracts, and [5]–[8] on the Cochrane Handbook’s own pages. We read [1] in the published PDF table; the CEBM web page itself did not load for us. – The tai chi figures in [19] are taken from the review’s published abstract. – The quotations from “Our labels” are from our own site.

Disclaimer

This article is for general education only. It is not medical advice and does not replace diagnosis or treatment by a doctor or other qualified health professional. Qigong and the Six Healing Sounds are traditional practices; we make no claims that they cure, treat or prevent any illness. If you have a health condition, are pregnant, or are unsure whether a practice suits you, please talk to your doctor first.