Abstract: A lot of systematic reviews/meta-analyses compiled by Chinese professionals and published in Chinese medical journals bring a lot of perplexity as they help to a certain extent to make decision in some conflicting clinical results. This study aims to assess the quality of systematic reviews and meta-analyses on antidepressant therapy published in Chinese journals. The reviews/meta-analyses on antidepressant therapy were identified by searching three main Chinese data banks i.e., Chinese National Knowledge Infrastructure (CNKI), Wanfang Data (WF) and Chinese Biomedical Literature Database (CBM). A pre-stated criterion was used for review/meta-analysis selection. All reviews were evaluated by two reviewers separately. Data in qualified reviews were extracted into a Microsoft Excel database for analysis. Two assessment tools were used: (1) the Overview Quality Assessment Questionnaire (OQAQ) and (2) Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA). Sixty eight reviews/meta-analyses were included in this study. More than 50% of them had methodological and reporting defects which could have reduced the reliability of the review/meta-analysis results. The flaws were mainly low quality of search strategy, inappropriate bias control and absence of quality assessment for original study.
INTRODUCTION
Evidence-based medicine is the conscientious, explicit and judicious use of current best evidence in making decisions for the care of individual patient (Sackett et al., 1996). Systematic reviews and meta-analyses have become increasingly important source of evidence for clinical practice. Some authorities have suggested that systematic reviews and meta-analyses of high-quality are evidences of the highest level for clinical decision-making (Dixon et al., 2005; Guyatt et al., 1995).
There are a few differences between systematic reviews and meta-analyses. Systematic review is the application of strategies that limit bias in the assembly, critical appraisal and synthesis of all relevant studies on a specific topic (Porta, 2008). Meta-analysis is a systematic review that uses statistical methods to combine the results of two or more studies (Cook et al., 1997).
Systematic review was introduced in China for social science and behavioral science in the late 1980s, then applied to the medical science gradually (Wei et al., 2007). Currently, if “systematic review” or “meta-analysis” is used as key word in title to search Chinese Biomedical Literature Database (CBM), more than 4600 records will be output. Although, these systematic reviews/meta-analyses could help to a certain extent to make decision in some conflicting clinical results, they bring a lot of perplexity at the same time. If authors were not proficient in the method of systematic reviews, their reviews could not produce accurate and unbiased outcome. These low quality reviews or analyses therefore mislead readers.
As the number of systematic review and meta-analysis published in Chinese journal increased, urgent attention should be paid to the quality of them. In this paper, we evaluated the reporting and methodological quality of systematic reviews/meta-analyses on antidepressant therapy published in Chinese journal. We tried to locate any defects in systematic review and meta-analysis, aiming to enhance the reliability of systematic review and meta-analysis published in Chinese journal.
MATERIALS AND METHODS
Research strategy: A comprehensive search for published systematic review/meta-analysis was performed in Chinese National Knowledge Infrastructure, Wanfang Data and Chinese Biomedical Literature Database. The search key words were used as following: meta-analysis, systematic review, depression and antidepressant (in Chinese). The search deadline was April 30, 2012. In addition, the references in any retrieved review/meta-analysis were traced to identify any studies that might be missed out.
Inclusion and exclusion criteria: Systematic review/meta-analysis was included as long as they reported clinical antidepressant therapy and published in Chinese journals. Review was excluded if (1) it was neither systematic review nor meta-analysis, (2) it did not address antidepressant therapy topics, (3) it was not Chinese literatures, (4) it was duplicate publication and (5) it was not journal article.
Selection process: An initial screening was conducted based on titles or abstracts, following by selection based on full-text review. Review/meta-analysis was considered eligible if it met the inclusion criteria.
Overview quality assessment questionnaire (OQAQ) (Oxman and Guyatt, 1991): The OQAQ was used for quality appraisal to evaluate whether review/meta-analysis authors conducted a comprehensive search, minimized bias in the selection of primary studies, evaluated the primary studies and pooled the data appropriately. The questionnaire consists of 10 questions. The first 9 questions are designed to assess different aspects of methodological quality and have set answers of “yes”, “partially/can’t tell” or “no”. Question 10 is an assessment of the overall scientific quality of the systematic review/meta-analysis on a scale of 1 to 7 which is based on how well the review scored on the first 9 questions.
Preferred reporting items for systematic reviews and meta-analyses (PRISMA): The PRISMA Statement consists of a 27-item checklist and a four-phase flow diagram. It is used for critical appraisal of published systematic reviews. The aim of the PRISMA Statement is to help authors improve the reporting of systematic reviews and meta-analyses (Liberati et al., 2009; Moher et al., 2009).
Data extraction and analysis: A database was established to extract data. The database had two components: (1) general information, including first author, year of publication, journal, citations number, conclusions drawn by the reviewers, (2) quality of included reviews, including OQAQ and PRISMA scale, the approach to assess quality of primary studies, the number of trials with adequate sequence generation and allocation concealment. Data were extracted independently by two reviewers and consensus was obtained on all details prior to data analysis. Data was analyzed with SPSS 13.0. The percentage of “yes”, “partially/can’t tell” or “no” of each item was counted. The overall mean score was calculated and expressed as Mean±SD, as well as a 95% Confidence Interval (CI).
RESULTS
Research results: According to the search strategy, a total of 1041 potentially relevant reviews were output. After selection, 68 reviews were identified. The reasons for exclusion of any review and the selection flow were shown in Fig. 1.
General information of 68 reviews: The numbers of reviews/meta-analyses on antidepressant therapy published in Chinese journals over 12 years were shown in Fig. 2. The overall tread is upward. Because deadline of review search was April 30, 2012, the number of studies in 2012 was only 3. Among the 68 reviews, 20 were published in Chinese Core Journals which represented a higher academic level in China.
| Fig. 1: | Flow chart showing results of search and reasons for exclusion of reviews |
The total citations of all 68 reviews were 127 (the average citation of them was 1.88). The maximum citation was 27 and the minimum was 0.
Quality results from the OQAQ: The results of methodological quality of 68 reviews were presented in Table 1. The mean score of question 10 was 3.82 (95% CI (3.46, 4.17)) with the lowest score 1.5 and the highest 6.5. The mean overall OQAQ score of each year was illustrated in Fig. 3. The scores were improved in recent years. In general, no one in 68 reviews responded “yes” to all first 9 questions. Particularly, positive response rates for question 3, 7, 8, 9 were 74, 100, 85 and 100%, respectively. However, the positive response rates for question 1 and question 2 were only 29, 24%, respectively. The search resources, the years of retrieval and the specific search strategy for database were not mentioned in their search strategies. Only 21% of reviews claimed to avoid the bias by two independent reviewers to select eligible original studies (question 4). The 37% of reviews reported criteria for assessing validity of included studies (question 5) and 31% of reviews used the Cochrane Collaboration’s ‘Risk of Bias’ tool or the Jaded scale (question 6). However, 15% of them did not conduct heterogeneity test or combine the results appropriately.
PRISMA results: Table 2 presented a summary of PRISMA items of the included reviews. The mean score was 15.18 (95% CI (14.23, 16.14)) with the lowest score was 5 and the highest was 23.5.
| Fig. 2: | Numbers of systematic reviews/meta-analyses of antidepressant therapy published in Chinese journals from 2001 through 2012 |
The mean overall PRISMA score of each year was depicted in Fig. 4. Little change occurred in PRISMA score over time. However, no one of 68 reviews was completely in compliance with all 27 checklist items. In general, titles were reasonable (item 1), readers could easily identify whether the review was systematic review or meta-analysis. All abstracts were not well-reported (item 2). Fifty eight reviews narrated the rationale in detail in the introduction part (item 3). Only 20 reviews briefed the objective fully (item 4), others failed to provide an explicit statement of questions being addressed with reference to participants, interventions, comparisons, outcomes and study design. In the methods part, all reviews/meta-analyses did not explain the protocol and registration (item 5). Meanwhile, eligibility criteria (item 6), data items (item 11), summary measures (item 13) and synthesis of results (item 14) were elucidated relatively adequate in all reviews. But information sources (item 7) and study selection (item 9) in most reviews were partially supplied. A part of reviews conducted literature search only in a few specific journals. Besides, search (item 8), data collection process (item 10), risk of bias across studies (item 12) and additional analyses (item 16) were mentioned only in a few reviews. In the results part, study selection (item 17) and risk of bias across studies (item 22) were reported rarely. Study characteristics (item 18) and synthesis of results (item 21) met the standard with a high degree.
| Fig. 3: | Mean overall OQAQ score of systematic reviews/meta-analyses of antidepressant therapy published in Chinese journals over time |
| Table 1: | Summary of OQAQ questions in included reviews |
| Table 2: | Summary of PRISMA statement in included 68 reviews |
| Fig. 4: | Mean overall PRISMA score of systematic reviews/meta-analyses of antidepressant therapy published in Chinese journals over time |
In the discussion section, the report of summary of evidence (item 24) and limitations (item 25) were very little, while the conclusions (item 26) of studies were reported comprehensively. Thirty four reviews reported the source of funds (item 27) but failed to explain the role of funders for the systematic review.
DISCUSSION
The pace of innovation in health care made clinicians difficult to keep up with the current state of knowledge (Manchikanti, 2008). Systematic reviews and meta-analyses are therefore recognized as the easily available evidences. Ideal systematic review and meta-analysis should be conducted comprehensively, maximize the precision, minimize the bias, narrate clearly so that any interested reader could understand and master the review easily (Jadad et al., 1998). Authors who want to do a high quality systematic review should put a lot of time and effort into work. He or she should try to solve the clinical controversy, rather than aim at the publishing paper itself (Idris, 2012). Unfortunately, however, not all systematic reviews and meta-analyses published in Chinese journals are strictly conducted. Although, methodological quality of systematic review is better in recent years, the reporting quality is not increased over time. To enhance the quality of systematic review, one should pay attentions to the following issues:
Methodological quality: The most important issue in systematic reviews is bias control (Oxman, 1994). Methodological quality is mainly focus on how well bias control in systematic review is conducted. In this evaluation, all 68 systematic reviews or meta-analyses had methodological problems:
| • | The goal of a systematic review is to identify relevant studies completely and unbiasedly (Lui et al., 2010). Most reviews failed to conduct an extensive literature search which resulted in selection bias. In order to avoid selection bias, reviewer should retrieve not only the literature published in domestic and international databases but also the unpublished grey literature and the ongoing research. Exclusion of the grey literature from systematic review can lead to exaggerated estimates of intervention effectiveness (Conn et al., 2003; Hopewell et al., 2007; McAuley et al., 2000). No-language limit is necessary to avoid language bias (Gregoire et al., 1995) |
| • | Quality of the original studies is closely related to reliability of the results of systematic reviews. However, 63% of reviews did not assess validity of original studies. It will lead to a clinically significant 30-50% exaggeration of treatment outcome when results of lower-quality trials are pooled (Moher et al., 1998) |
| • | Data merging is particularly challenging. In this research, although some authors complied with the pre-stated inclusion criteria and seemed to control bias, their results were heterogeneous. Some reviews pooled events despite heterogeneity and without analysis of the causes of heterogeneity |
Reporting quality: PRISMA focuses on the transparency and completeness of systematic reviews or meta-analyses, their research processes and findings report. All 68 reviews in this evaluation presented the following deficiencies:
| • | Abstracts should provide key information that enables readers to understand the scope, processes and findings of a review and to decide whether to read the full text or not (Liberati et al., 2009). Unfortunately, all abstracts did not completely tell background, data sources, study eligibility criteria, participants, interventions in the structured summary which make readers difficult to comprehend the review contents |
| • | All 68 reviews missed protocol and registration information. A protocol is important because it pre-specifies the objectives and methods of the systematic review (Liberati et al., 2009) and avoid bias (Sutton et al., 1998). Registration may possibly reduce the risk of duplicate reviews addressing the same question, reduce publication bias and provide greater transparency when updating systematic reviews (Bagshaw et al., 2006; Biondi-Zoccai et al., 2006; Liberati et al., 2009) |
| • | Authors of 68 reviewers were not good at reporting flow diagram of study selection, extracting data and evaluating the validity of the primary studies. Some reviews did not provide the forest plots. Forest plots are important because they supply a simple visual representation of the amount of variation between the results of the studies, as well as an estimate of the overall result of all the studies together (Lewis and Clarke, 2001) |
| • | In the discussion section, reviewers should interpret results, describe potential biases in the review processes and suggest future research agendas (Salari et al., 2011). However, many authors of 68 reviews did not discuss well in their work |
| Sixty eight reviews did not claim if their researches were funded by some institutions and the roles of funders. The claim is necessary for research sponsored by the for-profit organization is more likely to produce results favoring the sponsor than research funded by non-profit institutions (Als-Nielsen et al., 2003; Lexchin et al., 2003; Peppercorn et al., 2007) |
Evaluations on systematic reviews/meta-analyses published in English journals indicated overall quality is high (Salmos et al., 2010; Suebnukarn et al., 2010). Furthermore, the quality of reports published in Cochrane Database of Systematic Reviews is superior to meta-analyses published in regular journals (Delaney et al., 2007; Jadad et al., 1998, 2000; Lundh et al., 2009; Moseley et al., 2009).
CONCLUSIONS
The quality of systematic reviews of antidepressant therapy in China was poor. Further improvement on quality of methodology and reporting of systematic reviews should be highlighted. Chinese reviewers should master the methods of systematic review comprehensively and strict control all kinds of bias according to Cochrane Collaboration procedures.