International Journal of Clinical Medicine, 2013, 4, 108-113
http://dx.doi.org/10.4236/ijcm.2013.42021 Published Online February 2013 (http://www.scirp.org/journal/ijcm)
Reporting of Negative Randomized Trials in Three Major
Medical Journals
Gabriel A. Brooks1, Daniel W. Bowles1,2, Daniel B. Jamieson1, Janice V. Huang1, Alexandra Smart1,
Kathleen A. Heist1, Traci E. Yamashita1, Allan V. Prochazka1,2, Ravi K. Gopal1,2
1Department of Medicine, University of Colorado School of Medicine, Aurora, USA; 2Denver VA Medical Center, Eastern Colorado
Healthcare System, Denver, USA.
Received December 13th, 2012; revised January 15th, 2013; accepted January 22nd, 2013
ABSTRACT
Context: In recent years there has been increasing interest on publication bias and on initiatives to decrease bias, in-
cluding trial registration. Objective: To test whether there has been an increase in reports of randomized control trials
(RCT’s) with negative outcomes in major journals and to identify factors associated with these reports. Design: Retro-
spective review of reports of RCT’s published in the Journal of the American Medical Association, The Lancet and the
New England Journal of Medicine before (2002-’03, pre-registration era) and after (2007-’08, registration era) the in-
stitution of mandatory trial registration. Main Outcome Measure: The primary outcome was the proportion of RCT
reports with negative outcomes compared across the two eras. Secondary outcome includes other factors affecting pub-
lication. Results: We identified 917 reports of RCT’s published in the two study eras. No publications in the pre-regis-
tration era reported trial registration compared with 94.4% in the registration era (p < 0.001). There was a non-signifi-
cant increase in negative trials from the pre-registration to the registration era (29.1% vs. 34.1%, p = 0.10, OR 1.26,
95% CI 0.96 - 1.67). Study characteristics associated with negative outcomes include trials of drugs (OR 1.62, 95% CI
1.08 - 2.43), procedures or devices (OR 2.08, 95% CI 1.29 - 3.35), explicit identification of a single primary endpoint
(OR 1.70, 95% CI 1.40 - 2.47), and increasing sample size (OR 3.08, 95% CI 1.78 - 5.34). Non-inferiority study de-
sign was associated with a decreased likelihood of a negative outcome (OR 0.13, 95% CI 0.05 - 0.31). Conclusions:
The proportion of published RCT reports with negative outcomes in three major medical journals has not significantly
increased since the mandatory clinical trial registration policy. The observed prevalence of negative trials is associated
with increases in sample size and greater specificity in trial endpoints.
Keywords: Clinical Trial Registration; Publication Bias; Randomized Trials; Clinical Trials; Non-Inferiority Trials
1. Introduction
Health care professionals rely on up-to-date, compre-
hensive, and objective data to make clinical, research and
policy decisions. However, a growing body of data sug-
gests that the medical literature used to support these
decisions is subject to demonstrable publication bias,
with publication of study reports depending on character
of the results [1]. Trials with statistically significant out-
comes are nearly twice as likely to be published and are
published more quickly than so-called “negative trials”
[2,3]. Publication bias has been particularly evident
among studies of investigational drugs [4,5], where pro-
prietary interests may lead drug makers to withhold re-
sults of negative trials. These flaws in the medical litera-
ture skew systematic interpretation of clinical trial results,
misrepresent the experiences of patients who altruisti-
cally participate in these trials, and jeopardize the health
and safety of future patients.
In response to concerns about lack of transparency in
clinical trials, the United States Congress passed the
Food and Drug Administration Modernization Act in
1997 calling for the creation of a tool to permit public
access to information regarding ongoing clinical trials.
The National Institutes of Health (NIH) created the
clinicaltrials.gov registry in 2000 in response to this
mandate. Initially this registry was voluntary and limited
to trials involving medications targeted for “serious or
life threatening illnesses”. Even with this limited scope
there was poor compliance with registration of qualifying
studies.
In 2004 the International Committee of Medical Jour-
nal Editors (the ICMJE, made up of the editors of eleven
leading general medical journals) announced that effec-
tive September 2005, registration with clinicaltrials.gov

Reporting of Negative Randomized Trials in Three Major Medical Journals 109
or a similar registry would be required as a precondition
for publication [6]. Before the introduction of the ICM
JE’s policy there were fewer than 14,000 trials registered
at clinicaltrials.gov; within months of the policy’s intro-
duction the number of registered trials increased by 73%.
[7]. To date, over 136,000 trials from 182 countries are
registered at clinicaltrials.gov [8].
The establishment of clinicaltrials.gov by the NIH and
the ICMJE’s policy of mandatory clinical trial registra-
tion were both intended to increase the transparency of
clinical research. Based on anecdotal experience, we hy-
pothesized that the ICMJE’s mandatory clinical trial reg-
istration policy had resulted in a reduction in publication
bias by increasing the proportion of clinical trial reports
with negative outcomes published in major medical
journals. The primary aim of our study was to test whe-
ther there has been an increase in reports of randomized
control trials (RCT’s) with negative outcomes in major
journals and to identify factors associated with these re-
port.
2. Methods
2.1. Study Sample
Our study sample included all reports of RCT’s pub-
lished in the New England Journal of Medicine, The
Lancet and the Journal of the American Medical Asso-
ciation during 2002-’03 (preregistration era) and 2007-’08
(registration era). We selected these journals because
they represent the three general medicine journals with
the highest impact factors [9]. The two eras were cho-
sen to capture publication patterns before and after the
institution of mandatory clinical trial registration policies
(initiated in 2005). Two-year intervals were chosen to
represent each era based on a pilot sample of 50 articles
that suggested an 80% power to detect a 15% absolute
difference between eras in the proportion of published
trials with negative outcomes.
Reports of RCT’s were identified using the Cochrane
Central Register of Controlled Trials (Cochrane Register),
a publicly accessible database maintained by the Coch-
rane Group. We searched using the full journal name in
the source field of the Cochrane Register, limiting sear-
ches to the years of interest and excluding entries with
“comment” in the publication type field. When searching
for entries in the Lancet, we excluded entries with “on-
cology” or “neurology” in the source field to avoid que-
rying the database for RCT reports published in Lancet
Oncology and Lancet Neurology.
After our initial search, we excluded entries in the
Cochrane Register that were not reports of RCT’s (in-
cluding duplicate entries, entries for editorials and letters
to the editor and entries for trials other than RCT’s). We
defined an RCT as a prospective study randomizing hu-
man subjects into two or more groups for comparison of
a defined intervention. We excluded cost-effectiveness
trials because they have incremental outcomes that can-
not be categorized as positive or negative. We excluded
phase I and II trials because these studies frequently do
not have efficacy outcomes (phase I studies were initially
excluded from the ICMJE’s mandatory clinical trial reg-
istration policy).
2.2. Data Abstraction
For each included entry of an RCT report in the Coch-
rane Register we abstracted descriptive information and
assessed the statistical significance of the primary out-
come(s). One reviewer abstracted descriptive information
from the structured article abstracts within the Cochrane
Register. When abstracts provided insufficient informa-
tion on endpoints we accessed full reports at the on-line
sites of the respective journals. Descriptive information
abstracted included year and journal of publication, trial
registration number, intervention type (specified non-
exclusively as drug, procedure/device, surgery, behave-
ioral, diagnostic or treatment strategy), sample size,
number of study groups, use of placebo control, use of a
single primary endpoint and use of a non-inferiority
endpoint. We considered trials to be registered if any
registration number was reported in the abstract.
Two reviewers assessed each RCT report to categorize
the primary outcome(s) as positive or negative with re-
spect to statistical significance of the outcome. We de-
fined a statistically significant (positive) result as having
a p-value of less than 0.05 or reaching a pre-specified
non-inferiority parameter. Studies having more than one
primary outcome or without an explicitly identified pri-
mary outcome were categorized as positive if any main
outcome reported in the abstract was positive. In cases
where there was disagreement between the two reviewers
regarding the significance of a study outcome (positive
vs. negative), study interpretation was labeled as discor-
dant and a committee of three reviewers reached a con-
sensus determination (GB, AP and RG). This study was
deemed exempt from review by the Colorado Multiple
Institutional Review Board and the Department of Vet-
erans Affairs.
2.3. Statistical Analysis
We calculated descriptive statistics for study attributes
and compared these attributes across the two publication
eras using chi-squared or Wilcoxon rank-sum tests. At-
tributes with p ≤ 0.1 were included in a logistic regres-
sion model to identify study attributes that were associ-
ated with an increased likelihood of a negative outcome.
For model interpretability, sample size was categorized
as 200 - 900, 1000 - 4999, and ≥5000. A final regression

Reporting of Negative Randomized Trials in Three Major Medical Journals 110
model was created using a stepwise-selection regression
technique to identify attributes of RCT’s that had an in-
dependent effect on the likelihood of a reported negative
outcome. All analyses were performed using SAS Ver-
sion 9.2 (SAS Institute Inc., Cary, NC, USA).
3. Results
We identified 1091 entries in the Cochrane Central Reg-
ister published in the New England Journal of Medicine,
The Lancet and the Journal of the American Medical
Association in the pre-registration and registration eras.
Of the 1091 entries, 181 (16%) were excluded, leaving
917 reports of RCT’s for our analysis (468 and 449 re-
ports from the pre-registration and registration eras re-
spectively, see Figure 1).
Descriptive statistics for reports of RCT’s from each
era are illustrated in Table 1. We found a high level of
compliance with reporting of trial registration; 94.4% of
trials in the registration era reported registration com-
pared to no trials in the pre-registration era (p < 0.001).
There were more reports of drug trials in the registration
era compared to the pre-registration era (75.4% vs.
69.5%, p = 0.02), and there was a trend toward fewer
reports of trials with procedural or device therapies in the
registration era (13.4% vs. 17.7%, p = 0.07). Compared
to the pre-registration era, more RCT reports in the Reg-
istration era explicitly identified a primary endpoint
(56.0% vs. 69.5%, p ≤ 0.001) and more RCT’s had
non-inferiority primary endpoints (4.7% vs 9.1%, p =
0.01). Median sample sizes were larger for RCT’s re-
ported in the registration era (n = 750 vs. 385, p ≤
1091 entries identified in Cochrane Register
174 of 1091 entries excluded (16%)
- 17 duplicate entries
- 83 editorials/letters to the editor
- 58 studies other than RCT’s
- 13 RCT’s, phase I/II trials
- 3 RCT’s, economic endpoint
917 of 1091 RCT reports included (84%)
- 468 from pre-registration era 2002-’03 (51%)
- 166 from NEJM
- 142 from JAMA
-160 from Lancet
- 449 from registration era 2007-’08 (49%)
- 193 from NEJM
- 101 from JAMA
- 155 from Lanc et
Figure 1. Flow Diagram of RCT reports reviewed for inclu-
sion. Abbreviations: RCT, randomized controlled trial;
NEJM, New England Journal of Medicine; JAMA, Journal
of the American Medical Association.
0.001). With regard to the primary outcome, 29.1% of
RCT reports from the pre-registration era reported nega-
tive outcomes compared to 34.1% in the registration era.
This difference did not meet statistical significance (Ta-
ble 2, p = 0.10 for unadjusted analysis, OR 1.26%, 95%
CI 0.96 - 1.67). When analyzed by individual journal, a
statistically significant difference was noted only for the
New England Journal of Medicine, with a rate of nega-
tive publications of 24.1% in the pre-registration era and
35.2% in the registration era (p = 0.02).
While publication era did not predict a differential
likelihood of negative study outcomes, our logistic re-
gression model did identify a number of factors associ-
ated with an increased or decreased likelihood of nega-
tive outcomes (Table 3). The five factors associated with
an increased likelihood of a negative outcome were trials
of drug therapies, trials of procedure or device therapies,
explicit identification of a single primary endpoint, in-
creasing sample size and discordance (discordant inter-
pretation of the primary outcome by the two reviewers).
The only factor associated with a decreased likelihood of
Table 1. Characteristics of RCT reports.
Characteristic 2002-’03
% (n) 2007-’08 % (n) p-value
Registration reported 0 94.4% (424)<0.0001
Drug trial 69.5% (325) 75.4% (343)0.02
Procedure/device trial 17.7% (83) 13.4% (60)0.07
Surgery trial 4.3% (20) 2.7% (12)0.19
Behavioral trial 10.9% (51) 8.0% (36)0.14
Trial of diagnostic test 2.4% (11) 2.5% (11)0.92
Trial of treatment strategy 13.0% (61) 12.7% (57)0.88
Placebo controlled 40.8% (191) 42.3% (190)0.64
Non-inferiority design 4.7% (22) 9.1% (41)0.01
Single primary endpoint 56.0% (262) 69.5% (312)<0.0001
Discordant 10.3% (48) 6.5% (29)0.04
U.S. correspondence address42.3% (198) 39.0% (175)<0.0001
Median sample size 385 750
Table 2. Published RCT reports with negative outcomes, by
era.
Journal 2002-’03 % (n) 2007-’08 % (n) p-value
All journals29.1% (136) 34.1% (153) 0.10
NEJM 24.1% (40) 35.2% (68) 0.02
JAMA 38.7% (55) 43.6% (44) 0.45
Lancet 25.6% (41) 26.5% (41) 0.87

Reporting of Negative Randomized Trials in Three Major Medical Journals 111
Table 3. Logistic regression, modeling probability of a ne-
gative outcome (adjusted).
OR 95% CI p-value
Publication era
(relative to pre-registration era) 1.26 (0.93, 1.71) 0.14
Drug trial 1.62 (1.08, 2.43) 0.02
Procedure/device trial 2.08 (1.29, 3.35) 0.003
Non-inferiority trial .13 (0.05, 0.31) <0.0001
Single primary endpoint 1.79 (1.30, 2.47) 0.0004
Discordant 3.70 (2.13, 6.25) <0.0001
Sample size <0.0001
200 - 999 relative to <200 1.80 (1.18, 2.74)
1000 - 4999 relative to <200 2.54 (1.61, 4.00)
≥5000 relative to <200 3.08 (1.78, 5.34)
Journal <0.0001
Lancet relative to NEJM .82 (.58, 1.18)
JAMA relative to NEJM 1.87 (1.29, 2.71)
Abbreviations: OR, odds ratio; CI confidence interval; NEJM, New England
Journal of Medicine; JAMA, Journal of the American Medical Association
a negative outcome was a non-inferiority study design
(OR = 0.13, p < 0.001, 95% CI 0.05 - 0.31).
4. Discussion
We proposed that the anecdotal perception of an increase
in negative studies present in our academic journal club
is a real effect and that it would be associated with man-
datory clinical trial registration.
Our analysis did not find a significant difference in the
proportion of published trials with negative outcomes
between the two eras, even though there was a dramatic
increase in reporting of trial registration, suggesting that
trial registration may not have influenced the chances of
negative trials being published in major medical journals.
We also saw changes in the characteristics of reported
trials. Sample sizes of published studies increased, climb-
ing from a median of 385 to 750. The percentage of re-
ports explicitly identifying a primary outcome also in-
creased. Improvement in clinical trial reporting was re-
flected practically in our finding that it became easier for
abstracters to interpret trial outcomes, with discordant
interpretation of trial outcomes (positive vs. negative)
decreasing from the pre-registration era to the registra-
tion era. Though non-inferiority trials represented a small
minority of published trials, the percentage of trials using
a non-inferiority design nearly doubled across the two
study eras. Use of a non-inferiority design was the only
trial characteristic that was independently associated with
a decreased likelihood of a negative outcome. Character-
istics of trials that were associated with an increased
likelihood of a negative reported outcome were drug or
procedure interventions, larger sizes, and reporting of a
single primary endpoint.
To our knowledge this study is the first to examine the
effects of the ICMJE’s mandatory clinical trial registra-
tion policy on publication of negative studies. Previous
studies of publication bias have examined cohorts of tri-
als from within specific institutions or registries, and
have substantiated that negative trials are less likely to be
published [2,3]. We chose to examine the question of
publication bias at the level of the publishing journals,
since the ICMJE’s policy was enacted on this level and
this is the level of exposure for the journal reader. We
looked for evidence of changes in publication bias in the
general medicine journals with the highest impact factors
because research published in these journals is likely to
be influential and because all of these journals are mem-
bers of the ICMJE. In its 2004 policy statement, the
ICMJE cites two primary drivers of publication bias-
inconclusive or negative results that are perceived as
clinically uninformative and negative results that reflect
poorly on a proprietary intervention [6]. The ICMJE’s
policy highlights efforts from participating journals to
give appropriate weight to the importance of clinical trial
results, regardless of whether these results are positive or
negative. However, this policy will do little to affect
publication bias within high-profile journals that is
driven by clinically uninformative negative results; these
trials are likely to be published in journals with lower
impact factors. Our study was most likely to detect pub-
lication bias driven by negative results that reflected
poorly on proprietary interventions. Other recent studies
have demonstrated the connection between publication
bias and selective outcomes reporting, where primary
outcomes for trials of proprietary interventions appear to
have been changed post hoc but prior to publication.
[2,10,11]. Such pre-publication changes in reported out-
comes may partly account for our own negative findings.
We did observe two study characteristics that may
have driven this perceived increase in publication of
negative trials. Surprisingly, the larger the trial the more
likely it was to be negative.
This fact may represent studies in conditions that great
treatment improvements have already been achieved and
further therapies have marginal effect or therapies that
have pushed the envelope too far past benefit to detri-
ment. Also, the fact that more discordant studies were
noted to be negative may be related to selective reporting.
Negative trials may not have been as clear in their end-
points making study abstracters differ on their interpreta-
tion that the study was positive or negative. Only upon
reviewing the full article was one able to determine

Reporting of Negative Randomized Trials in Three Major Medical Journals 112
whether the study’s primary endpoint was truly negative.
Our study has a number of limitations. Our data are
based primarily on reviews of article abstracts, which
may not have contained the same information as full
publications. Our sample size was not adequate to detect
small differences between rates of publication of nega-
tive trials across the two study eras. Because we studied
RCT reports from two eras, it was not possible to control
fully for changes over time other than mandatory clinical
trial registration. The effect of mandatory clinical trial
registration on publication bias may be obscured by other
trends in the medical literature, such as selective out-
comes reporting or changes in the designs of randomized
controlled trials that favor positive outcomes. We note
that the editorial leadership at the three journals remained
constant over the two study eras. Also, because we stud-
ied publication bias in the three general medicine jour-
nals with the highest impact factors, our results do not
address changes in publication bias across the entire
medical literature or within medical journals of lower
impact factors.
Our study findings suggest that any effect of clinical
trial registration on publication bias within top-tier medi-
cine journals is likely to be small, if it exists. Other stud-
ies have shown substantial problems with selective out-
comes reporting even among registered trials. Do these
findings cast doubt on the utility of clinical trial registra-
tion as a tool to increase the transparency of clinical re-
search? It seems probable that standards of adequacy for
trial registration have not been sufficient [12,13], and
stricter enforcement of registration standards as well as
new laws requiring minimal outcomes reporting may
improve the utility of clinical trial registration to reduce
publication bias and selective outcomes reporting.
Our findings also suggest that studies with non-inferi-
ority designs are playing an increasingly important role
in the medical literature. Non-inferiority trials nearly
doubled as a proportion of all trials published in our
study sample, and these trials are much more likely to
have a positive outcome than trials with conventional
superiority designs. While the ascendance of non-inferi-
ority trials is a welcome trend in the arena of compara-
tive effectiveness research, this trend may be less benign
when applied in trials of proprietary therapies. In this
latter arena, a non-inferiority trial design may provide a
lower bar to evaluate the effectiveness of a therapy that is
seeking a wider market share [14].
Future studies examining the rates of published nega-
tive trials and the effect of trial registration on publica-
tion bias should look for this effect in a broader range of
journals, including subspecialty journals and smaller
general medicine journals. Because publication bias may
be driven by proprietary concerns, further inquiries
should shed more light on the interaction between trial
funding, publication bias and adoption of non-inferiority
study designs.
In conclusion, our results do not substantiate an asso-
ciation between clinical trial registration and a reduction
in publication bias in major medicine journals. Clinical
trial registration is meant as a means toward maximizing
the impact of clinical research activities and protecting
the contributions of altruistic patients. As we become
accustomed to the era of clinical trial registration (and
now, mandatory outcomes reporting) it is imperative to
learn what we can and cannot expect from this important
tool.
REFERENCES
[1] R. J. Simes, “Publication Bias: The Case for an Interna-
tional Registry of Clinical Trials,” Journal of Clinical
Oncology, Vol. 4, No. 10, 1986, pp. 1529-1541.
[2] K. Dwan., D. G. Altman, J. A. Arnaiz, et al., “Systematic
Review of the Empirical Evidence of Study Publication
Bias and Outcome Reporting Bias,” PLoS One, Vol. 3,
No. 8, 2008, e3081. doi:10.1371/journal.pone.0003081
[3] S. Hopewell, K. Loudon, M. J. Clarke, A. D. Oxman and
K. Dickersin, “Publication Bias in Clinical Trials Due to
Statistical Significance or Direction of Trial Results,”
Cochrane Database of Systematic Reviews, Vol. 1, 2009,
MR000006.
[4] K. Rising, P. Bachetti and L. Bero, “Reporting Bias in
Drug Trials Submitted to the Food and Drug Administra-
tion: Review of Publication and Presentation,” PLoS
Medicine, Vol. 5, No. 11, 2008, e217.
doi:10.1371/journal.pmed.0050217
[5] K. Lee, P. Bachetti and I. Sim, “Publication of Clinical
Trials Supporting Successful New Drug Applications: A
Literature Analysis,” PLoS Medicine, Vol. 5, No. 9, 2008,
e191. doi:10.1371/journal.pmed.0050191
[6] C. D. Deangelis, J. M. Drazen, F. A. Frizelle, C. Haug, J.
Hoey, R. Horton, S. Kotzin, C. Laine, A. Marusic, A. J.
Overbeke, T. V. Schroeder, H. C. Sox and M. B. Van Der
Weyden, “Clinical Trial Registration: A Statement from
the International Committee of Medical Journal Editors,”
Journal of the American Medical Association, Vol. 292,
No. 11, 2004, pp. 1363-1364.
doi:10.1001/jama.292.11.1363
[7] D. A. Zarin, T. Tse and N. C. Ide, “Trial Registration at
ClinicalTrials.gov between May and October 2005,” New
England Journal of Medicine, Vol. 353, No. 26, 2005, pp.
2779- 2787. doi:10.1056/NEJMsa053234
[8] Clinicaltrials.gov, “A Service of the US National Insti-
tutes of Health,” 2010. http://www.clinicaltrials.gov
[9] ISI Web of Knowledge, “Journal Citation Reports,” 2010.
http://0-admin-apps.isiknowledge.com.impulse.ucdenver.
edu/JCR
[10] S. S. Vedula, L. Bero, R. W. Scherer and K. Dickersin,
“Outcome Reporting in Industry-Sponsored Trials of Ga-
bapentin for Off-Label Use,” New England Journal of
Medicine, Vol. 361, No. 20, 2009, pp. 1963-1971.

Reporting of Negative Randomized Trials in Three Major Medical Journals 113
doi:10.1056/NEJMsa0906126
[11] S. Mathieu, I. Boutron, D. Moher, D. Altman and P.
Ravaud, “Comparison of Registered and Published Pri-
mary Outcomes in Randomized Controlled Trials,” Jour-
nal of the American Medical Association, Vol. 302, No. 9,
2009, pp. 977-984. doi:10.1001/jama.2009.1242
[12] C. D. Deangelis, J. M. Drazen, F. A. Frizelle, C. Haug, J.
Hoey, R. Horton, S. Kotzin, C. Laine, A. Marusic, A. J.
Overbeke, T. V. Schroeder, H. C. Sox and M. B. Van Der
Weyden, “Is This Clinical Trial Fully Registered? A
Statement from the International Committee of Medical
Journal Editors,” New England Journal of Medicine, Vol.
352, No. 23, 2005, pp. 2436-2438.
doi:10.1056/NEJMe058127
[13] J. S. Ross, G. K. Mulvey, E. M. Hines, S. E. Nissen and
H. M. Krumholz, “Trial Publication after Registration in
ClinicalTrials.gov: A Cross-Sectional Analysis,” PLoS
Medicine, Vol. 6, No. 9, 2009, e1000144.
doi:10.1371/journal.pmed.1000144
[14] S. Garatini and V. Bertele, “Non-Inferiority Trials Are
Unethical Because They Disregard Patients’ Interests,”
Lancet, Vol. 370, No. 9602, 2007, pp. 1875-1877.
doi:10.1016/S0140-6736(07)61604-3