Accepted on 2 Jun 2017
Submitted on 15 Mar 2017
Evidence based medicine is gaining importance in surgery [1], [2]. Randomised Controlled Trials (RCTs) that are appropriately designed, conducted and reported, alongside systematic reviews, represent the gold standard evidence for assessment of treatment effects. Publication of inadequate, incomplete or unclear information on RCT methodology and results limits RCT validity and generalisability, biases the apparent effectiveness of interventions, precludes subsequent inclusion of RCTs in meta-analyses and systematic reviews and is unethical [3], [4].
The CONSORT (Consolidated Standards of Reporting Trials) statement, developed in 1966 [5] and subsequently revised in 2001 [6] and in 2010 [7], was designed to facilitate RCT design, and improve reporting standards and quality. The guideline describes 25 items that are mandatory features of the article that must be reported in the manuscript. The 25-items were agreed by an international consensus group on the basis that inclusion of each item is essential to judge the reliability or relevance of the findings, or that item exclusion is associated with biased estimates of treatment effect [8].
The CONSORT statement has been successful in improving the reporting transparency and manuscript development of RCTs [5], [9], [10]. Many RCTs, including those published in plastic and craniofacial surgery, still fail to adequately report RCTs according to the CONSORT statement [3], [11]. A systematic review in 2014 conducted on 55 RCTs published within plastic surgery [3] found on average RCTs comply with just half of the CONSORT items (11.5 of 23 items, 50%). A review in 2015 assessing the CONSORT compliance of 65 RCTs conducted on cleft lip and/or palate found RCTs on average adequately reported 63.2% (15.8 out of 25) of the CONSORT items [11]. Both reviews found description of the randomisation methods and blinding to be particularly poorly described across RCTs, two areas which can lead to significant bias in results.
Sub-surgical specialities of plastic and craniofacial surgery are notoriously lacking in evidence-based guidelines and high-quality research [12]. High quality craniofacial surgery RCTs are essential to ensure the best interventions are chosen and that new interventions are adequately assessed. Compliance with the CONSORT reporting criteria across all topics in craniofacial surgery has not previously been formally assessed. An appraisal of RCT compliance is necessary to ensure data published is reported adequately. It can also highlight specific areas of CONSORT compliance where increased vigilance is indicated. Furthermore, poor RCT reporting is associated with poor methodology, and improving compliance with reporting of the CONSORT items may pave the way for improved craniofacial RCT methodologies [13].
Compliance with the CONSORT statement is not only the responsibility of RCT authors. The editorial bodies of journals publishing RCTs play a key role in enforcing adherence to the CONSORT guidelines [14]. Mandatory CONSORT concordance in RCT submission and surveillance of CONSORT reporting through peer-review improves the transparency of reporting and thus the conduct of RCTs in medical research [15]. An appraisal of whether RCTs within relevant craniofacial journals are compliant with CONSORT criteria can also highlight whether the journal reviewers and editorial bodies are enforcing the use of CONSORT, and whether this process can be improved.
This systematic review will assess the compliance of RCTs conducted on craniofacial surgery and published in craniofacial journals with CONSORT reporting criteria. The focus is on RCTs conducted within the last five years to provide an assessment on current reporting adequacy. An objective and reproducible definition of craniofacial RCTs will be used: RCTs within craniofacial surgery journals.
The primary objectives yield the primary outcomes and will be to:
The secondary objectives yield the secondary outcomes and will be to:
It is hypothesised that RCTs in leading craniofacial journals comply fully with CONSORT criteria. Failing this, it is hypothesised that greater CONSORT compliance will be found in multi-centre RCTs, with a larger number of contributing authors, and published in journals with a higher IF, more recently.
This systematic review protocol is compliant with Preferred Reporting Items for Systematic Review and meta-Analysis protocols guidelines (PRISMA-P) [16]. This systematic review will be conducted in line with the Cochrane Handbook for Systematic Reviews and Interventions [17], and will be compliant with PRISMA guidelines [18]. It has been registered a priori with the Registry of Systematic Reviews/meta-analyses (http://www.researchregistry.com/, Unique Identifying Number (UIN): reviewregistry219) and is openly accessible.
To maximise comparability to previous work, a search technique similar to that described in a recent systematic review assessing the methodological quality of craniofacial surgery systematic reviews with PRISMA reporting criteria [19] will be used.
Thomson Reuters InCites Journal Citation Reports (https://jcr.incites.thomsonreuters.com, Thomas Reuters, New York, US) and the 2016 Journal Citation Reports (http://scientific.thomsonreuters.com/imgblast/JCRFullCovlist-2016.pdf) were used to find journals containing the stem ‘cranio’ in their title along with their Journal Impact Factors (IF). This identified 5 journals (IF range 0.7–1.64): The Journal of Craniomaxillofacial Surgery (JCMFS); Orthodontics and Craniofacial Research (OCR); The Cleft Palate-Craniofacial Journal (CPCJ), The Journal of Craniomandibular Practice (JCMP), and The Journal of Craniofacial Surgery (JCS).
The search strategy was developed in collaboration with an experienced information search specialist. A systematic review of all RCTs published in these five journals (JCMFS, OCR, CPCJ, JCMP, JCS) across a five-year period from the search date will be performed. All five craniofacial journals selected for analysis were Pubmed indexed, therefore the electronic database search will be performed through MEDLINE PubMed. The search will include relevant keywords (journal names) in English combined with the Boolean logical operator ‘OR’. The ‘Randomised controlled trial’ filter for Pubmed published by Cochrane (http://work.cochrane.org/pubmed) will be applied with the Boolean logical operator ‘AND’. This limits articles to those published on humans in the last 5 years. A systematic review conducted in 2017 is considered an appropriate time lapse since publication of the revised CONSORT statement in 2010 [7] for it to become an integrated into RCT conduct and publication. The publication language for all the five chosen journals is English, therefore only English language texts will be included. The research question is focused on adherence of published RCTs, therefore the grey literature will not be assessed. An example of the full search strategy conducted on 28.11.2016 is shown in Table 1.
Table 1
Pubmed Search History of a preliminary search conducted on the 3rd March 2016.
| Search | Studies generated |
|---|---|
| “j Craniomandibular Pract”[Journal]) OR “J Craniomaxillofac Surg”[Journal]) OR “J Craniofac Surg”[Journal]) OR “cleft Palate Craniofac J”[Journal]) OR “orthod Craniofac Res”[Journal] | 15362 |
| Randomised controlled trial[pt] OR controlled clinical trial[pt] OR randomised[tiab] OR placebo[tiab] OR “drug therapy”[Subheading] OR randomly[tiab] OR trial[tiab] OR groups[tiab] NOT (“animals”[MeSH Terms] NOT “humans”[MeSH Terms])) AND “j Craniomandibular Pract”[Journal] OR “J Craniomaxillofac Surg”[Journal] OR “J Craniofac Surg”[Journal] OR “cleft Palate Craniofac J”[Journal] OR “orthod Craniofac Res”[Journal] AND ((Clinical Trial[ptyp] OR Clinical Study[ptyp] OR Clinical Trial, Phase I[ptyp] OR Clinical Trial, Phase II[ptyp] OR Clinical Trial, Phase III[ptyp] OR Clinical Trial, Phase IV[ptyp] OR Controlled Clinical Trial[ptyp] OR Multicenter Study[ptyp] OR Pragmatic Clinical Trial[ptyp] OR Randomized Controlled Trial[ptyp]) AND “2012/03/15”[PDat]: ”2017/03/15”[PDat] AND “humans”[MeSH Terms] | 205 |
All RCTs published within these five journals across the five-year period will be examined for inclusion. There will be two levels of screening. Firstly, two independent researchers will screen the title and abstract of each RCT against the agreed eligibility criteria (Table 2). In the second level, full texts will be retrieved for provisional RCTs to be included, and reviewed again for inclusion against the same exclusion/inclusion criteria. Only articles successfully passing both levels of screening will be included. In any situations of unresolved discrepancy the full text will also be retrieved and a senior author will arbitrate. Included studies will be entered into a pre-formatted database spreadsheet using Microsoft Excel Software (Version 15.23, 2016, Microsoft). Reasons for any study exclusion will be noted, and reviewed by the senior author.
Table 2
Study inclusion and exclusion criteria.
The Cochrane definition of a RCT will be used. This states an RCT is “A trial where individuals followed in the trial were definitely or possibly assigned prospectively to one of two (or more) alternative forms of health care using random allocation or some form of quasi-random method (such as alternation, date of birth, or case recorded number” (http://handbook.cochrane.org/chapter_6/box_6_3_a_cochrane_definitions_and_criteria_for_randomised.htm).
Compliance of each RCT with the 25-item CONSORT checklist will be scored by two independent researchers. For each of the 25 items, RCTs will score either: 1 – fully reported; 0 – not reported; NA – not-applicable. RCTs will score ‘1’ for an item only if all the information details for that CONSORT item have been reported, and ‘0’ if the required information is only partially reported or not reported. Reviewers extracting and scoring data are members of a collaborative experienced in systematic review conduct. To ensure familiarity with CONSORT the expanded explanation article on CONSORT will need to be reviewed by each data extractor before extraction begins. They will then extract data from two RCTs conducted on the topic of general surgery prior to formal data collection. The lead author will review the data extracted and will feedback to the data collectors to ensure maximum consistency and reduce any data collection “learning curve”. After the first five craniofacial RCTs have been scored, these will again be checked by the lead author for accuracy. If one extractor makes an error, this error and necessary corrections will be fed back to the entire collaborative team to ensure consistency. Any unexpected findings that present during data extraction will be fed back to the entire collaborative team.
Data extracted from the craniofacial RCTs will be entered directly into a pre-formatted database created using Microsoft Excel Software (Version 15.23, 2016, Microsoft). The data extracted by the two researchers will be compared and discrepancies resolved by consensus or lead author arbitration. A maximum ‘CONSORT score’ for each RCT will be calculated by subtracting the number of ‘NA’ items from the theoretical maximum score of 25. The total points scored out of the individualised maximum for each will give the CONSORT percentage for that RCT. Each item is given equal weighting. This method does not penalise RCTs for not reporting irrelevant items. Compliance with individual items of the statement will be analysed by summating the number of articles fulfilling that item in full divided by the total number of included articles. The editorial body of each of the five journals will be emailed to investigate which of the journals mandate CONSORT declaration at submission.
The two independent researchers will extract the secondary outcomes from each article into another pre-formatted Microsoft Excel database (Version 15.23, 2016, Microsoft). This will include the condition addressed by each review (grouped by field of pathology), the intervention examined and surrogate markers of article quality including: journal IF; publication date; number of contributing authors; whether the RCT was single- or multi-centre. The data extracted will then be cross-checked for validity (two authors will review each other’s’ work). The relationship between CONSORT score and year of publication, single- vs multi-centre, the number of contributing authors, and journal IF will be assessed.
All data will be analysed using Microsoft Excel Software (Version 15.23, 2016, Microsoft). All statistical analyses will be performed using SPSS (Version 23). Continuous variables will be presented as a mean and ranges. Categorical variables will be presented as percentage values. Inter-rater agreement between CONSORT scores and inclusion/exclusion of RCTs will be assessed using the Kappa score. A P value of <0.05 will be deemed significant for all statistical tests. Summary measures will be used to calculate the mean weighted number of CONSORT items adequately reported for all studies and the percentage of studies compliant with each element of the CONSORT guidelines. An initial pilot search has yielded 199 RCTs to be screened for inclusion. This number is similar to that generated in previous systematic reviews assessing CONSORT compliance in other surgical areas [3], [20], [21], [22], [23], and is therefore considered sufficient to provide a true representation of RCT reporting in craniofacial surgery.
The CONSORT score will be compared across RCTs grouped on the basis of their subject topic within craniofacial surgery, intervention studied, as well as by indexes relating to RCT quality: date of publication; IF; number of contributing authors (all to be analysed as a continuous variables) and multi- vs single-centre (analysed as a categorical variable). A one-tailed Student’s t-test will be used to analyse multicentre versus single-centre trials. Regression analyses will be used to analyse relationship between CONSORT score and year of publication, number of contributing authors, and journal IF. No meta-regression or sensitivity analyses will be planned or pre-specified.
The intent will be to publish in a peer-reviewed journal and to present the data at relevant conferences.
This systematic review aims to elucidate areas of RCT reporting can be improved, to help guide authors and ensure high quality methodology RCTs are conducted within craniofacial surgery, which will ensure the best clinical practice is followed, and the best possible patient outcomes are achieved.
The research received no specific grant from any funding agency in the public, commercial or non-for-profit sectors.
There are no conflicts of interest.
MRB, RA and TEP conceived this paper. MRB and TEP drafted the manuscript. All authors critically revised it for important intellectual content and approved the final version for publication. MRB has collected the preliminary data.
Rohrich, R.J. (2010). So you want to be better: the role of evidence-based medicine in plastic surgery. Plast. Reconstr. Surg. 126(4): 1395–1398. [PubMed]
Rhee, J.S. and Daramola, O.O. (2012). No need to fear evidence-based medicine. Arch. Facial Plastic Surg. 14(2): 89–92.
Agha, R.A., Camm, C.F., Doganay, E., Edison, E., Siddiqui, M.R. and Orgill, D.P. (2014). Randomised controlled trials in plastic surgery: a systematic review of reporting quality. Eur. J. Plast. Surg. 37(2): 55–62. [PubMed]
Juni, P., Altman, D.G. and Egger, M. (2001). Assessing the quality of controlled clinical trials. Br. Med. J. 323(7303): 42. [PubMed]
Begg, C., Cho, M., Eastwood, S., Horton, R., Moher, D. and Olkin, I. (1996). Improving the quality of reporting of randomized controlled trials: the CONSORT statement. JAMA 276(8): 637–639. [PubMed]
Moher, D., Schulz, K.F., Altman, D.G. and Group, C. (2001). The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet 357(9263): 1191–1194. [PubMed]
Schulz, K.F., Altman, D.G. and Moher, D. (2010). CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMC Med. 8(1): 1. [PubMed]
Smith, T.A., Kulatilake, P., Brown, L.J., Wigley, J., Hameed, W. and Shantikumar, S. (2015). Do surgery journals insist on reporting by CONSORT and PRISMA? A follow-up survey of ‘instructions to authors’. Ann. Med. Surg. 4(1): 17–21.
Plint, A.C., Moher, D., Morrison, A., Schulz, K., Altman, D.G. and Hill, C. (2006). Does the CONSORT checklist improve the quality of reports of randomised controlled trials? A systematic review. Med. J. Aust. 185(5): 263. [PubMed]
Hopewell, S., Dutton, S., Yu, L.-M., Chan, A.-W. and Altman, D.G. (2010). The quality of reports of randomised trials in 2000 and 2006: comparative study of articles indexed in PubMed. BMJ 340: c723. [PubMed]
Hardwicke, J., Nassimizadeh, M. and Richard, B. (2015). Reporting of randomized controlled trials in cleft lip and palate: a 10-year review. Cleft Palate-Craniofacial J.,
Drolet, B.C. and Lorenzi, N.M. (2012). Registries and evidence-based medicine in craniofacial and plastic surgery. J. Craniofacial Surg. 23(1): 301–303.
Huwiler-Müntener, K., Jüni, P., Junker, C. and Egger, M. (2002). Quality of reporting of randomized trials as a measure of methodologic quality. JAMA 287(21): 2801–2804. [PubMed]
Edison, E., Agha, R. and Camm, C. (2011). No excuse for poor reporting of surgical RCTs. J. Orthodontics 38: 305–307.
Shantikumar, S., Wigley, J., Hameed, W. and Handa, A. (2012). A survey of instructions to authors in surgical journals on reporting by CONSORT and PRISMA. Ann. R. Coll. Surg. Engl. 94(7): 468.
Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A. and Petticrew, M. (2015). Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. System. Rev. 4(1): 1. [PubMed]
Higgins, J.P. and Green, S. (2008). Cochrane Handbook for Systematic Reviews of Interventions In: Wiley Online Library.
Moher, D., Liberati, A., Tetzlaff, J. and Altman, D.G. (2009). Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann. Intern. Med. 151(4): 264–269. [PubMed]
Pidgeon, T.E., Wellstead, G., Sagoo, H., Jafree, D.J., Fowler, A.J. and Agha, R.A. (2016). An assessment of the compliance of systematic review articles published in craniofacial surgery with the PRISMA statement guidelines: a systematic review. J. Cranio-Maxillofacial Surg. 44(10): 1522–1530.
Agha, R.A., Camm, C.F., Edison, E. and Orgill, D.P. (2013). The methodological quality of randomized controlled trials in plastic surgery needs improvement: a systematic review. J. Plast. Reconstructi. Aesthetic Surg. 66(4): 447–452.
Adie, S., Harris, I.A., Naylor, J.M. and Mittal, R. (2013). CONSORT compliance in surgical randomized trials: are we there yet? A systematic review. Ann. Surg. 258(6): 872–878. [PubMed]
Yeow, V.K., Lee, S.-T.T., Lambrecht, T.J., Barnett, J., Gorney, M. and Hardjowasito, W. (2002). International task force on volunteer cleft missions. J. Craniofacial Surg. 13(1): 18–25.
Lee, S.-Y., Teoh, P.J., Camm, C.F. and Agha, R.A. (2013). Compliance of randomized controlled trials in trauma surgery with the CONSORT statement. J. Trauma Acute Care Surg. 75(4): 562–572. [PubMed]
Supplementary data associated with this article can be found, in the online version, at https://s3-eu-west-1.amazonaws.com/ubiquity-partner-network/ijs/journal/ijsp/IJSP8.zip.
The authors are grateful to Mark Keer Clinical Librarian at East Kent Hospital for his help in defining the search strategy.