Author: Robert Hirst / Codes: SLO10 / Published: 03/09/2020

Hello again. I hope you’ve all settled into your new departments, new jobs, or conversely have managed to give a warm welcome to your new trainees. 

One of TERN’s aims is demystifying clinical research [1]. This month we’re exploring that most mystical of modalities: the Delphi method. The name refers to the Oracle of Delphi, a name its authors disliked, thinking it smacked “of the occult” [2]. Better that, perhaps, than its first recorded usage: estimating how many atomic bombs would be required to disrupt American munition production during the Cold War [3]

We’ll be talking about the technique, its benefits and limitations, and considerations for running these studies.   

The method

The technique is a structured group facilitation technique. A problem is chosen, which can be a closely-defined topic or something broader. This is a particular favourite, which involves estimating the number of spiny lobsters caught by a group of fishermen. 

A group of participants are selected, who are well-informed on a topic and might be considered experts. They are invited to participate in the process. 

A questionnaire looking to explore the problem is written. This was classically an open-ended set of questions that allows participants freedom in their responses, but other approaches use directed questions or present a literature review of the evidence [4]

This questionnaire is distributed to the panel of experts who independently consider the question(s) and return them to the researcher. The researcher collates and analyse the responses to the questionnaire and produces a summarised feedback report of the first-round decisions. 

This feedback report is distributed alongside a second questionnaire to the panellists. Participants review the feedback, and may be asked to rank or rate their earlier responses and complete the second questionnaire, identifying areas of agreement or disagreement. This process of iterative feedback is repeated until consensus is attained. The classic Delphi technique used four rounds [5] although most researchers take a pragmatic approach, with two or three rounds often preferred [6]

Some people like pictures, so here’s the whole process broadly done as a flowchart. 

Common features 

The technique has a near-endless number of modifications on ‘the classic Delphi’*, although they typically feature a number of common elements. 

Participant anonymity

Participant anonymity is a crucial part of the Delphi process, which is intended to reduce unhelpful group dynamics, such as the effects of dominant individuals, reducing ‘noise’ (communication which distorts the data), and pressure to conform [6-7]. It is a method well-suited to ‘The New Normal’ of coronavirusas participants from across the globe can discuss and consider a problem without being in the same room. 

Moving towards consensus

The aim of the Delphi method is to derive consensus from collected opinion. The process of feedback and iterative rounds is designed to drive the panel towards finding consensus. What constitutes this is typically defined from the outset, with some choosing a defined (perhaps somewhat arbitrary) figure, such as 80% [8], whilst others prefer stability of responses across rounds [9]. In some cases, this may prove impossible, but a bimodal distribution is an important finding in itself. 

Controlled feedback

Feedback concerning the group opinions from each round is presented with each subsequent round. It is an organised summary of the previous round’s results, usually with descriptive statistics or qualitative comments used to describe the aggregated group opinion or demonstrate areas of consensus and disagreement. 

Statistical analysis

Statistical analysis is used to quantify consensus. Descriptive statistics, such as the median, mean and standard deviation, are typically used to measure central tendency (a surrogate marker for consensus) and spread (disagreement), although alternative statistical analyses are used. Tracking statistics across rounds allows the researchers to analyse how and when consensus develops throughout the process. 


The Delphi technique is useful as a systematic attempt to develop consensus, and are particularly useful when empirical evidence is lacking, limited, or contradictory [10].  

The Delphi technique can reduce the impact of certain confounding interpersonal processes, such as the influence of dominant individuals, the pressure to conform to group-think or the influence of status [6, 11]. Participants are free to revise their opinion without losing face and input is more likely to be assessed on its own merits [12]

The process can draw input from a much greater breadth of understanding, experience, and specialities than would typically be possible to assemble otherwise, removing the constraints of distance and scheduling [13]. Given this, and the small support structure required, it is also relatively inexpensive [13].


The process can be time-consuming, and the risk of dropout increases as the number of rounds increases [14]. The feedback administered can potentially mould opinion, which was demonstrated when investigators fed distorted feedback to participants, who then changed their answers to more closely match the false consensus[15]

There is a lack of guidance and accepted standards on how to interpret and analyse the results [10], and reporting of methodology is often inconsistent [16]. Many studies using the Delphi technique do not comment on reliability, and questions remain concerning the reproducibility of results between different studies [17].

The technique calls for ‘experts’ (or ‘informed individuals’ [18]) which has been criticised as being unscientific [19] and potentially prone to bias [10]. The utility of this approach was questioned by a study which compared the forecasts of social scientists with lay representatives and found there to be little difference between the forecasts of the two groups [20]. It is also at odds with the move towards greater patient-public involvement [21] that clinical research has embraced.


So, some pros, some cons. Still interested? Here are a few questions to ask yourself. 

  • Is your question suitable for the Delphi method? [22] 
    • Exploring underlying assumptions about a topic. 
    • Seeking out information to generate a consensus on a topic.
    • Correlating informed judgements on a topic that spans a number of disciplines.
  • How will you administrate the study?
    • Most studies are performed using e-mail, survey instruments or through use of a specific Delphi platform.  
    • Consider deadlines, and how much time you’ll need. It can get time-consuming. 
  • Who will be in your panel? 
    • Consider your panel composition – what will be your criteria for inclusion as a participant? How many will you have? 
    • Seriously consider how you will maximise your participant engagement [6]
    • Consider soliciting nominations from people within the target group [4]
  • How will you use statistics? 
    • How will you define consensus? 
    • How will you collect, collate and analyse the data? There are some useful papers on the mathematics involved in the references [23-24]
    • How will you present the results during rounds and at the end – graphically, numerically, or both? 
  • What will your initial round look like? 
    • Qualitative or quantitative?
  • How will you report it? 
    • The CREDES Guidance [25] will help you report (and design) your study with rigor and transparency. 

Well, that’s your lot. See you next month for something less heavy on the references. 

Robert Hirst


[email protected]

Join our mailing list 

* I am dismayed at having to use the words ‘classic Delphi’ so much, as it has a very Alan Partridge vibe.  


  1. About TERN
  2. Adler, M., Erio, Z. (1996). Gazing Into the Oracle: The Delphi Method and Its Application to Social Policy and Public Health. (Jessica Kingsley Publishers). 
  3. Dalkey, N., Helmer, O. (1963). An Experimental Application of the DELPHI Method to the Use of ExpertsManagement Science 9(3), 458-67. 
  4. Hsu, C., Sandford, B. (2007). The Delphi Technique: Making Sense of Consensus. Practical Assessment, Research, and Evaluation 12(12), Article 10.
  5. Erffmeyer, R., Erffmeyer, E., Lane, I. (1986). The Delphi Technique: An Empirical Evaluation of the Optimal Number of RoundsGroup & Organization Management 11(1-2), 120-8. 
  6. Hasson, F., Keeney, S., McKenna, H. (2000). Research guidelines for the Delphi survey techniqueJournal of Advanced Nursing 32(4), 1008-15. 
  7. Dalkey, N. (1972). The Delphi method: An experimental study of group opinion. In Dalkey, N, Rourke, D., Lewis, R., & Snyder, D. (Eds.) Studies in the quality of life: Delphi and decision-making. Lexington, MA: Lexington Books.
  8. Green, B., Jones, M., Hughes, D. & Williams, A. (1999). Applying the Delphi technique in a study of GP’s information requirementsHealth and Social Care in the Community 7(3), 198-205. 
  9. Crisp, J., Pelletier, D., Duffield, C., Adams, A. & Nagy, S. (1997) The Delphi method? Nursing Research46(2), 116-8.
  10. Murphy, M., Black N., Lamping, D., McKee, C., Sanderson,…, Marteau, T. (1998). Consensus development methods, and their use in clinical guideline development. Health Technology Assessment 2(3), 1-88. 
  11. De Meyrick, J. (2003). The Delphi method and health researchHealth Education 103(1), 7-16. 
  12. Scheffer, B., Rubenfeld, M. (2000). A Consensus Statement on Critical Thinking in NursingJournal of Nursing Education 39(8), 352-9. 
  13. Humphrey-Murto, S., Varpio, L., Wood, T., Gonsalves, C., Ufholz, L,…, Foth, T. The Use of the Delphi and Other Consensus Group Methods in Medical Education Research: A ReviewAcademic Medicine 92(10), 1491-8. 
  14. Walker, A., & Selfe, J. (1996). The Delphi method: a useful tool for the allied health researcherBritish Journal of Therapy and Rehabilitation, 3(12), 677–81.
  15. Scheibe, M., Skutsch, M., & Schofer, J. (1975). Experiments in Delphi methodology. In Linstone, H., Turoff, M. (Eds.). The Delphi method: Techniques and applications. Reading: Addison-Wesley Publishing Company.
  16. Sinha, I., Smyth, R., Williamson, P. (2011). Using the Delphi Technique to Determine Which Outcomes to Measure in Clinical Trials: Recommendations for the Future Based on a Systematic Review of Existing StudiesPLoS Medicine 8(1), e1000393.
  17. Woudenberg, F. (1991). An evaluation of Delphi. Technological Forecasting and Social Change 40(2), 131-50.  
  18. McKenna, H. (1994) The Delphi technique: a worthwhile approach for nursing? Journal of Advanced Nursing 19(6), 1221-5.
  19. Strauss, H., Zeigler, H. (1975) The Delphi technique and its uses in social science researchJournal of Creative Behaviour 9(4), 253-9.
  20. Bedford, M. (1972). The Value of Competing Experts and the Impact of ‘Drop-outs’ on Delphi Results. Montreal, Bell Corporation. Referenced in Sackman, H. (1975) A Delphi Critique. Lexington: Lexington Books.
  21. Fleurence, R., Forsythe, L., Lauer, M., Rotter, J., Ioannidis, J.,…, Selby, J. (2014). Engaging patients and stakeholders in research proposal review: the patient-centered outcomes research institute. Annals of Internal Medicine 161(2), 122-30. 
  22. Turoff, M. (1970). The design of a policy Delphi. Technological Forecasting and Social Change. 2(2), 149-71. 
  23. Greatorex, J., Dexter, T. (2000). An accessible analytic approach for investigating what happens between the rounds of a Delphi studyJournal of Advanced Nursing 32(4), 1016-24.
  24. Holey, E., Feeley, J., Dixon, J., Whittaker, V. (2007). An exploration of the use of simple statistics to measure consensus and stability in Delphi studies. BMC Medical Research Methodology 7, Article 52.[25]. Jünger, S., Payne, S., Brine, J., Radbruch, L., Brearley, S. (2017). Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: Recommendations based on a methodological systematic review. Palliative Medicine 31(8), 684-706.