Tumor-Associated Autoantibodies: Re-Optimization of EarlyCDT-Lung Diagnostic Performance and Its Application to Indeterminate Pulmonary Nodules

Background: Low-dose computed tomography (CT) screening reduces lung cancer mortality but costs are prohibitive for most healthcare budgets due to high false positive rates. An adjunctive test able to distinguish malignant from benign pulmonary nodules would be hugely beneficial. EarlyCDT-Lung measures serum autoantibodies to tumor-associated antigens and has found clinical acceptance to aid early detection of lung cancer for high risk patients. However performance was optimized for screening. The construction of a receiver-operating characteristic (ROC) curve would enable optimization of performance for alternative settings, including nodule malignancy. Methods: A Monte-Carlo search method was used to construct a ROC curve using a case-control cohort, enabling high and low specificity versions of EarlyCDT-Lung to be determined. These were used for a theoretical evaluation of a nodule cohort, and positive predictive value (PPV) was calculated under the assumption of independence of risk source. Patients or their nodules are typically classified into three risk groups: low (0% 10%), intermediate (10% 65%) and high (>65%) risk of malignancy. The predicted shift in risk group by application of the high and low specificity versions, along with the current commercial EarlyCDT-Lung, was then estimated. Results: The ROC curve, with an area under the curve of 0.743, was constructed. The high specificity (98%), low specificity (49%) and current commercial (91% specificity) versions of EarlyCDT-Lung re-classified 27%, 23% and 26% of intermediate nodules, respectively, to either a higher (10%, 8% and 10%) or lower (17%, 15% and 16%) risk group. Conclusion: A ROC curve was constructed to allow performance prediction of EarlyCDT-Lung at different specificities in the indeterminate nodule setting. This enabled risk re-classification of intermediate risk nodules, and could therefore facilitate alternative more appropriate inHow to cite this paper: Healey, G.F., Macdonald, I.K., Reynolds, C., Allen, J. and Murray, A. (2017) Tumor-Associated Autoantibodies: Re-Optimization of EarlyCDT-Lung Diagnostic Performance and Its Application to Indeterminate Pulmonary Nodules. Journal of Cancer Therapy, 8, 506-517. https://doi.org/10.4236/jct.2017.85043 Received: April 20, 2017 Accepted: May 24, 2017 Published: May 27, 2017 Copyright © 2017 by authors and Scientific Research Publishing Inc. This work is licensed under the Creative Commons Attribution International License (CC BY 4.0). http://creativecommons.org/licenses/by/4.0/ Open Access


Introduction
In the United States, there is a well-documented high rate of lung cancer and pulmonary nodules [1] [2]. In recent years, widespread use of computed tomography (CT) resulting from the finding by the National Lung Screening Trial that annual CT screening reduces lung cancer mortality by 20% compared to chest x-ray [3] has resulted in over 50% of CT-screened patients having at least one non-calcified nodule and over 96% of nodules over 4mm being benign (false positive) [3] [4]. There is also a high false positive rate and morbidity associated with biopsy or resection of these benign nodules [2] [3] [5]. Consequently, a CT screening program will need substantial resources to perform follow up for every pulmonary nodule detected. Potential malignancy has been found to increase with nodule size, which also tends to result in later-stage detection of lung cancer as the nodule grows [4]. Therefore, there is a clear need for a complimentary test to aid evaluation of nodule malignancy potential, ideally a biomarker, to reduce false positive rates of CT screening for lung cancer and enable earlier detection.
Tumor-associated (TA) antigens are currently the most commonly measured biomarkers during management of cancer patients [6]. However they are often of little utility in early disease due to serum levels having a strong correlation with tumor burden. This generally results in levels only being elevated in later stage disease, limiting their clinical use to monitoring of treatment and disease recurrence. TA autoantibodies have been identified for a range of solid tumors and are currently emerging as strong candidates for clinically useful cancer biomarkers [7]. The mechanisms of their secretion have not been clearly determined but their production is thought to occur due to increased immunogenicity of the corresponding antigen. They are produced early in tumorigenesis, being measurable up to 5 years before the development of clinical symptoms [8].
As antibodies they represent biologically amplified markers, increasing the detectable signal for the corresponding antigen. They also persist in the circulation with half-lives of typically up to 30 days [9] and are more stable outside the body than other biomarkers.
EarlyCDT-Lung (Oncimmune Holdings plc.) detects the presence of TA autoantibodies to a panel of seven lung cancer associated antigens using an indirect enzyme-linked immunosorbent assay (ELISA) method [10] [11]. A sample is positive when at least one of the panel of TA autoantibodies is elevated above a pre-determined cut-off [12]. The test has been both technically [10] and clinically validated in seven independent validation cohorts [11] [13] including highrisk control groups matched for age, sex, and smoking history. The performance characteristics have been further validated in the commercial setting by an audit of clinical outcomes for the first 1599 patients who had a valid EarlyCDT-Lung and unknown nodule status [14]. The test consistently identifies lung cancer with 92% accuracy (compared with 50% for CT) with a sensitivity (true positive rate) of ~40% for all stages and types (small cell and non-small cell) of lung tumors and a specificity (true negative rate) of ~93% for all cohorts, showing the robustness and reproducibility of this assay system [12].
The test was launched commercially in 2012 for the early detection of lung cancer in a high-risk screening scenario [14] [15]. However it has also had some clinical acceptance for follow up of patients who had a positive result on CT. It has recently been demonstrated that a positive test result reflects a significant increased risk for malignancy in lung nodules 4 to 20 mm in diameter [16]. This confirms that EarlyCDT-Lung may have clinical value in assessment of malignancy risk for indeterminate pulmonary nodules.
The optimum diagnostic performance of EarlyCDT-Lung in the screening setting may not be appropriate for the indeterminate nodule setting. Performance is defined by the particular set of cut-offs chosen. So by varying the cut-offs, specificity and sensitivity, and hence false-positive and false-negative rates, can be adjusted. So the purpose of this present paper is largely technical, and describes the creation and use of a receiver-operating characteristic (ROC) curve for Ear-lyCDT-Lung. Because of the need for a relatively large number of cancers to achieve this, a case-control cohort (designated the optimization cohort) previously used for validation of EarlyCDT-Lung was used [11] [12]. It was compared with a dataset (designated the nodule cohort) derived from our commercial operations (designated the audit cohort).

Optimization Cohort
This cohort has been described elsewhere and was previously used to set the current commercial cut-offs for EarlyCDT-Lung [12]. Briefly, serum samples from 235 patients with lung cancer obtained at or just after histopathological confirmation of the tumor, were assayed. The lung cancers consisted of 178 nonsmall cell lung cancer (75.7 %), 53 small cell lung cancers (22.6 %), and 4 others (1 sarcoma, 2 × bronchogenic carcinomas, and 1 undefined lung cancer). This cohort was representative of a high risk population with a mean and median age of 65 years and a high proportion of smokers (49%) and ex-smokers (29%). The controls consisted of 266 healthy volunteers, 235 of which were individually matched to the lung cancer patients for age, gender, and smoking status. This group of controls had no evidence of any current or prior cancer including non-melanoma skin cancer.

Audit Patient Cohort
This cohort has been described previously [14]. Briefly, EarlyCDT-Lung was launched commercially in November 2010 with physicians in routine practice across USA ordering the test on behalf of their patients. This cohort was assembled for an audit of clinical practice, reporting the physicians' use of the test and not a prospective study in a population defined by inclusion and exclusion factors. The cohort is comprised of the first 861 patients tested with the seven-antigen test, with some exclusion factors applied. The cohort was followed for clinical outcomes and a total of 35 cases of lung cancer were recorded.

Nodule Patient Cohort
This cohort was derived from the audit cohort by assessing CT imaging reports, from within 6 months of the autoantibody test, for the presence of pulmonary nodules. The size of the largest non-calcified nodule was recorded and exclusion criteria (invalid autoantibody tests, lost-to-follow-up or with previous history of cancer) were applied. The "Exclusive cohort" [14], tested with the seven antigen panel and with cancer being confirmed by pathology reports (n = 166) [16], was used. Nodules were categorized by size (<4 mm, 4 mm -20 mm, >20 mm).

Comparison of Cohorts
The diagnostic performance of the optimization, audit and nodule (by nodule size: <4 mm, 4 mm -20 mm and >20 mm) cohorts was compared statistically using Fisher's Exact test applied to frequency tables for specificity and sensitivity separately. The test is slightly biased in the direction of non-significance due to the nodule cohort being a subset of the audit set.

Construction of ROC Curve
When a diagnostic test is based on a single test result or score, the construction of a ROC curve is straightforward. The cut-off can be varied from minimum to maximum possible values and the specificity and sensitivity are read off at a series of points. EarlyCDT-Lung, however, is effectively a multivariate test, so construction of a ROC curve is not as straightforward. Since the cut-offs for Ear-lyCDT-Lung are based on quantitative results, one method is to re-optimize the test at every point along a range of selected specificity levels. For the optimization cohort individualized antigen cut-offs were re-optimized to maximize sensitivity at each and every specificity level hence allowing the construction of a complete ROC curve. The optimization was a paper exercise not involving reassay of the samples.
A Monte-Carlo direct search was used to explore the large number of possible combinations of cut-off level as follows. For a set of n controls: 1) Choose the specificity.
2) Calculate the number of control subjects that can be specified as positive i.e. s = n (1-specificity).
3) Select an autoantibody at random. To complete the description of diagnostic performance, the PPV, Negative Predictive Value (NPV) and Relative Risk (RR) were calculated based on 20% lung cancer risk for patients with nodules (of any size) [16]. Positive diagnostic ratio (DLRp) and Negative diagnostic ratio (DLRn) were also calculated.

Shift of Risk Category
Patients or their nodules are typically classified into three groups: low (0% - Hence the number shifted up to a higher risk group is the total of true positives and true negatives where the post-test risk is in a higher risk group and the number shifted down to a lower risk group is the total of true negatives and false negatives where the post-test risk is in a lower risk group.

Diagnostic Likelihood Ratio (DLRp)
In a clinical setting pre-test and post-test risk estimation is generally more useful than specificity and sensitivity themselves. With a pre-test risk, r, and making the assumption of statistical independence mentioned above, the process equates to applying the positive diagnostic likelihood ratio (DLRp) [19] for test-positive subjects where: There is an equivalent formula for test-negative subjects which converts the pre-test risk to post-test (NPV) using the negative ratio (DLRn) involving NPV: Expressing the conversion from pre-to post-test risk in this way emphasizes that these estimates of post-test risk rely on the assumption of strict statistical independence of nodule size and marker test under a "both-positive rule".

Comparison of Test Performance for the Patient Cohorts
The diagnostic metrics (specificity, sensitivity and positive diagnostic likelihood ratio) were first tabulated ( The performance was also compared statistically for the cohorts by calculating DLRp (see Table 1) in order to assess whether these cohorts were similar enough to be used for this study. For the audit and nodule cohorts, due to the relatively low number of cancers the DLRp is not estimated with high accuracy, and showed quite wide confidence intervals, but the estimates were consistent across cohorts.
The consistency of diagnostic performance of EarlyCDT-Lung for the nodule and audit cohorts was determined to be similar to that for the optimization cohort. Therefore the ROC curve constructed using the optimization cohort can be used to predict the performance of higher and lower specificity versions of Ear-lyCDT-Lung and the subsequent nodule risk category shift analysis for the nodule and audit cohorts.

ROC Curve
The performance statistics were tabulated for a range of 100 specificity values from 0% to 100% allowing the creation of the whole ROC curve (Figure 1). A selection of the higher specificity points, 49% to 98%, with sensitivity ranging from 80% to 28%, were also tabulated with relevant performance metrics ( Table   2).
The estimated area under the curve (AUC) for the ROC curve was 0.743 (p < 0.001 vs. randomness). The PPV, relative risk, DLRp and DLRn all increase as specificity increases. Thus between 90% and 98% specificity the diagnostic performance increases considerably, PPV from 54% to 78%, relative risk from 3.

Shift of Risk Category
The risk groups and performance statistics for three test versions of interest, high specificity (98% specificity and 28% sensitivity for the optimization cohort), current commercial test (91%/40%) and low specificity (49%/80%) were tabulated ( were re-classified, either to a higher (10.4%) or lower (15.5%) group, and for the low specificity version of the test a total of 22.5% were re-classified, either to a higher (7.6%) or lower (14.9%) group.

Discussion
The current commercial EarlyCDT-Lung performance in a pre-imaging screening scenario is quoted at 90.6% specificity and 41.3% sensitivity, with the speci- ficity unadjusted for occult cancer in the optimization cohort [11] [12]. The performance was determined by fixing the specificity and then optimizing sensitivity. The specificity of around 90% was chosen to restrict the false-positive rate to 10%. This rate is quite high but is justified on the basis that follow-up of positive patients will be by imaging and not anything more invasive. In the situation where a patient has already undergone imaging and a nodule has been observed, then the next stage may be more invasive possibly involving biopsy or surgery. Here the false-positive level needs to be kept low to avoid unnecessary invasive procedures. So the specificity would be set at a higher level, perhaps 95% or higher.
The diagnostic performance of EarlyCDT-Lung in the nodule cohort was found similar to that for the optimization cohort and hence the ROC curve constructed (using the optimization cohort) can be used to predict the performance of higher and lower specificity versions of EarlyCDT-Lung in the nodule context. Note that the ROC curve presented here is effectively a non-parametric curve as opposed to a model-based curve derived, for example, using logistic regression. This allows multivariate optimization without the assumptions required for a model-based method.
Clinicians consider many other factors besides nodule size itself when interpreting indeterminate pulmonary nodules. In some specialist centers nodulebased risk models, incorporating both demographic and nodule characteristics, are used [18]. We combined the EarlyCDT-Lung biomarker test with a nodulebased risk estimator (Swensen/Mayo in this case) to predict the percentage of intermediate risk pulmonary nodules which switched risk group after a positive or negative biomarker test. For the high specificity, commercial and low specificity tests the nodule reclassification was as high as 27%, 26% and 23% respectively, with the majority of reclassification being in the direction of lower risk for all tests.
It is recommended for nodules of moderate size that clinicians estimate the pre-test probability of malignancy either qualitatively by using their clinical judgment or quantitatively by using a validated model. The Fleischner Society guidelines describe the recommended follow-up for patients identified with a pulmonary nodule by CT [20] [21] and balance early cancer detection and treatment against over-use of invasive diagnostic procedures. A biomarker that suggested an increased or decreased risk of malignancy whilst the nodule was still relatively small (e.g. 4 mm -20 mm) would be clinically useful, as it could lead to earlier detection and decreased mortality. For larger nodules there is less need for a biomarker test since the clinical pathway is better established.

Conclusion
We have shown that it is possible to derive a receiver-operator characteristic (ROC) curve for a multivariate assay by re-optimizing the cut-offs for every fixed specificity value. This allows the deployment of different versions of the EarlyCDT-Lung biomarker test in different clinical situations depending on the relative costs of false-positives and false-negatives. In particular, higher and lower specificity versions would seem to be more appropriate for a nodule interpretation scenario where subsequent intervention may well be invasive.