The field of surgery has made significant advances over the last 50 years with minimally invasive surgery, organ transplantation and joint replacement amongst prominent examples. Historically much of this progress has not been robustly evaluated or transparently reported in accordance with best methodological practice, leading the Lancet, in the 1990s, to refer to surgical research as a comic opera .
Solomon et al. assessed 202 RCTs in 1990 and found that the lowest quality ones were those that had a surgeon as the principal author, assessed an actual surgical procedure (as opposed to a drug being used in surgical patients) or were published in a surgical journal .
The IDEAL Framework and Recommendations were developed with the aim of increasing the quality of surgical research . Subsequent papers on IDEAL have focused on its implementation . Recent specific guidance has been developed on how to progress through IDEAL stages 2a and 2b . Table 1 summarises the IDEAL Framework and Recommendations as they currently stand.
|IDEAL Framework (stage of evolution of intervention)||IDEAL Recommendations (stage-specific study design and reporting)|
|Stage 1: Idea. First in human||
Purpose: proof of concept
Number and types of patients: single digit, highly selective
Number and types of surgeons: very few, innovators*
Output: description of intervention or procedure
Status of intervention: evolving, at inception stage
Reporting methods: structured case reports
Outcomes reported: proof of concept, technical achievement, dramatic success, adverse events, surgeon views of the procedure
Stage endpoint: once a decision is made to do a series of cases—ie, to proceed to stage 2a
|Stage 2a: Development. Iterative modification of the intervention until a stable version is achieved. Design: single centre or single intervention, case series or prospective cohort||
Purpose: development of procedure
Number and types of patients: few; selected
Number and types of surgeons: few, innovators, and some early adopters
Output: technical description of procedure and its development, with explanation of reasons for changes
Intervention: evolving, procedure development towards a stable optimised version
Methods: prospective development studies (small prospective cohort studies)
Outcomes: mainly safety, technical, and procedural success
Stage Endpoint: when the procedure is considered optimised, and stable enough to allow replication in stage 2b, there should be no intent at this point to make further major modifications
|Stage 2b: Exploration. Collaborative prospective data collection and analysis aimed at achieving consensus on key issues, to determine if an RCT is feasible, and to define its design features. Intended as a bridge from rational to comparative evaluation||
Purpose: Achieving consensus between surgeons and centres on the parameters for an RCT (if possible)
Number and types of patients: many, broadening indication to include all potential beneficiaries
Number and types of surgeons: many, innovators, early adopters, early majority*
Outputs: effect estimate for the intervention based on a large sample, allowing power calculations, analysis of learning curves, estimate of influence of prespecified technical variants and patient subgroups on outcome, qualitative research to determine operator and patient values, increased mutual confidence amongst operators
Intervention: the procedure is stable in individual hands but variation exists between operators, acceptable variants are subsequently defined by analysis of pooled results
Method: prospective multicentre exploration cohort study or pilot, or feasibility multicentre RCTs
Outcomes: safety, clinical outcomes (specific or graded), short-term outcomes, patient-centred or reported outcomes, feasibility outcomes
Stage endpoints: demonstrate that technique can be more widely adopted, and demonstrate that progression to RCT is desirable and feasible
|Stage 3: Assessment. Definitive comparison of main efficacy and safety aspects of a new technique against current best treatment.||
Purpose: comparative effectiveness testing
Number and types of patients: many, expanded indications (well defined)
Number and types of surgeons: many, early majority*
Output: comparison with current standard therapy
Intervention: stable, with acceptable variations clearly defined
Method: RCT with or without additions or modifications, alternative designs (cluster, preference RCTs, stepped wedge, adaptive designs)
Outcomes: clinical outcomes (specific and graded), potentially patient-reported outcomes, health economic outcomes
Stage endpoints: clear valid evidence on relative effectiveness of innovation, and identification of issues requiring long-term monitoring
|Stage 4: Long term monitoring||
Number and types of patients: all eligible
Number and types of surgeons: all eligible
Output: description, audit, recording of regional and local variations, quality assurance, risk adjustment, detection of indication creep
Method: registry, routine database, rare-case reports
Outcomes: rare events, long-term outcomes, quality assurance
Stage endpoints: dependent on lifecycle of device or procedure, registries for devices —IDEAL-D, registries at earlier stages of IDEAL
Recent evidence shows progress in clinical research in surgery from an IDEAL perspective . There is increasing use of IDEAL and evidence of methodological improvement between the decades before and after the publication of the IDEAL Recommendations, although no causal link can be demonstrated. As more and more studies begin to utilise the IDEAL Framework (552 unique citations of key IDEAL papers, Web of Science searched 19 October 2017), there is a pressing need to develop reporting guidelines to help guide investigators on how to report their research. In a developing field such as this, with very limited numbers of published examples to model from, an expert Delphi consensus process is an appropriate method for doing this.
A Delphi consensus exercise will be undertaken to ascertain the need for specific reporting guidelines for each stage of the IDEAL Framework and to develop such guidelines where required. Recommendations may be to use already existing guidelines for some stages and to develop particular IDEAL stage guidance for those where nothing appropriate currently exists. We anticipate that the greatest need for new guidance will be in IDEAL stages 2a and 2b. We propose to undertake a separate guideline development process for each IDEAL stage requiring guidance, as described below.
We will identify a group of suitable participants to take part in the Delphi rounds to ensure a representative sample to provide a broad and balanced cross-section of those interested in and qualified to shape the development of such reporting guidelines. We will approach members of the IDEAL Collaboration and attendees from the 2016 and 2017 IDEAL conferences who have a working knowledge and interest in surgical research. Our database of potential participants comprises 445 individuals that includes surgeons, research methodologists and triallists, industry representatives and professionals from HTA and commissioning. We will invite all 445 individuals to participate, and those who agree to commit to all rounds will comprise the final list.
We will also establish a working group during the first round of the Delphi. As part of the survey, participants will be asked if they would like to participate in the working group, and approximately 20 volunteers will be selected from this group. The purpose of the working group will be to meet at the end of the Delphi process to compile the survey results and compose the final set of recommendations.
The process will begin with a search of the EQUATOR network (http://www.equator-network.org) to find relevant reporting guidelines already in existence and also previous work undertaken by the IDEAL Collaboration to outline the current available guidance to participants. The participants will be surveyed to reach consensus on which stages require us to develop specific IDEAL guidance. The survey process will take place electronically and will be administered via Google Forms (https://www.google.co.uk/forms/about/).
For each stage required, we will compile a long list of items to put forward into the Delphi consensus exercise. This list will be based on current IDEAL Recommendations and relevant existing reporting guidelines. We will invite participants to recommend adaptations to the items put forward and to suggest new ones, using a simple questionnaire format. We will include a summary of the suggestions for modifications and additions (including the degree of support for each where possible) and scores indicating item importance in the subsequent rounds.
In each round, the participants will rate the importance of reporting each element on a nine-point Likert scale as proposed by the GRADE group . In this scale 1–3 signifies an outcome of limited importance, 4–6 important but not critical and 7–9 critical. If 70% or more of respondents score an item 7–9 and fewer than 15% score it 1–3, then that item should proceed into the reporting guideline. Similarly, consensus that an outcome should not be included would be 70% or more scoring it 1–3 and 15% or less scoring it 7–9. Sequential rounds of questionnaire administration and completion will take place until a final set of recommendations is produced. There is no pre-determined number of Delphi rounds, although the expectation is that at least three will be needed.
Following the final round of the survey will be a consensus meeting of the working group which will be conducted in person or via teleconference. This forum will allow the group to interpret the survey results and to discuss items where the voting process did not yield a definitive result. The product of this meeting will be a refined and final set of reporting guidelines.
No ethical approval is necessary because there is no involvement of patients or experimental subjects. The IDEAL reporting guidelines will be published in a peer-reviewed journal. They will also be presented at relevant national and international conferences. They will be disseminated electronically and in print. Journals publishing IDEAL papers will be encouraged to endorse the statement.
RAA wrote the first draft. AH, AK and PM contributed to revisions and approved the final manuscript for submission. All authors have no conflicts of interest to declare.
No specific funding was obtained for this project. IDEAL is funded by the Oxford NIHR Biomedical Research Centre, educational grants and meetings income. AK is on a visiting fellowship supported by Maimonides Medical Center.
No registration is necessary since there is no involvement of patients or experimental subjects.
Riaz A. Agha firstname.lastname@example.org.
Solomon, M.J. and McLeod, R.S. (1998). Surgery and the randomised controlled trial: past, present and future. Med. J. Aust. 169: 380–383. [PubMed]
McCulloch, P., Altman, D.G., Campbell, W.B., Flum, D.R., Glasziou, P. and Marshall, J.C. (2009). No surgical innovation without evaluation: the IDEAL recommendations. Lancet 374(9695): 1105–1112. [PubMed]
Hirst, A., Agha, R.A., Rosin, D. and McCulloch, P. (2013). How can we improve surgical research and innovation? The IDEAL framework for action. Int. J. Surg. 11(10): 1038–1042. [PubMed]
Pennell, C.P., Hirst, A.D., Campbell, W.B., Sood, A., Agha, R.A., Barkun, J.S.T. and McCulloch, P.G. (2016). Practical guide to the Idea, development and exploration stages of the IDEAL Framework and Recommendations. Br. J. Surg. 103(5): 607–615. [PubMed]
McCulloch, P., Feinberg, J., Philippou, Y., Kolias, A., Kehoe, S., Lancaster, G., Donovan, J., Petrinic, T., Agha, R. and Pennell, C. (2018). Progress in clinical research in surgery and IDEAL. Lancet, pii: S0140-6736(18)30102-8. doi: 10.1016/S0140-6736(18)30102-8. [Epub ahead of print] PubMed PMID: 29361334.
Sinha, I.P., Smyth, R.L. and Williamson, P.R. (2011 Jan). Using the Delphi technique to determine which outcomes to measure in clinical trials: recommendations for the future based on a systematic review of existing studies. PLoS Med 8(1): e1000393. [PubMed]
Supplementary data associated with this article can be found, in the online version, at https://s3-eu-west-1.amazonaws.com/ubiquity-partner-network/ijs/journal/ijsp/IJSP20.zip.
We would like to thank members of the IDEAL Collaboration who took part in helping to shape this protocol.