Comment on “ Quantitative Evaluation of Commercially Available Test Kit for Ciguatera in Fish ”

This letter is in regards to the paper, “Quantitative evaluation of commercially available test kit for ciguatera in fish” [1]. We were compelled to respond because the entire premise of this paper is flawed, thus invalidating its stated conclusions. The data presented in the paper is derived from the opinions of four independent readers who evaluated identical Cigua-Check test sticks to screen fish samples for ciguatoxin (CTX), the results of which were then compared with corresponding samples tested in a non-specific bioassay with questionable statistics (see Table 1 [1]). In addition to several factual errors presented in the paper, we have identified several issues with this study, such as insufficient detail and questionable data analyses, that make its interpretations unreliable.

First, while the sodium channel-specific N2a neuroblastoma cell bioassay used for comparative analysis in this study is a useful tool for detecting and measuring general cytotoxicity as a result of exposure to CTXs and related toxins, it is not specific for CTXs.Like the Cigua-Check ® test, the N2a bioassay is a screening method, and the N2a bioassay does not provide "actual ciguateric status" as claimed by the authors in the Discussion.Instead, it is a non-specific sodium channel assay that requires an analytical method in order to definitively identify specific neurotoxins in a given sample.This bioassay is not capable of discriminating between different neurotoxins that act in a similar manner [2].Indeed, the N2a bioassay has been used to detect and measure brevetoxins, saxitoxins, neosaxitoxin, gonyautoxin II plus III, decarbamoylsaxitoxin, and palytoxin, in addition to ciguatoxin [3][4][5][6][7][8][9], and requires confirmation of the identity of specific toxic compounds using analytical methods whenever possible [2].Even the US Food and Drug Administration (FDA) employs a two-tiered protocol to test fish for CTXs, including an in vitro assay such as the N2a bioassay and an analytical chemistry technique (liquid chromatographymass spectrometry, LC-MS) because the N2a screening procedure does not specifically identify the sodium channel active agent present in fish samples [10].Given that all comparative analyses in this paper were based solely on positive or negative determinations made using the N2a neuroblastoma cell bioassay, any comparisons must be approached with caution.
Second, the N2a bioassay dose-response curve for Pacific Ciguatoxin-1 (P-CTX-1) in Figure 1 presented in the Bienfang, et al. [1] paper has many shortcomings.Importantly, this figure does not show any error bars for the individual data points.Instead, averaged "relative standard deviations about the means" for the standards and controls are described in the figure legend.However, if the data is evaluated using this questionable method of applying the average relative standard deviations around the means for the entire control and standard groups as shown in Figure 2, it is clear that the overlap between the standard and control curves eliminates any statistically significant differences.The only point on the standard curve that could be considered different from the control curve had a well concentration of approximately 200 pM P-CTX-1 (Figure 2).In addition, the dose-response curve depicted does not show a standard S-shaped curve, suggesting further inconsistencies with the data.An example of the correct presentation of the curve can be found in [2].Another issue regarding the N2a bioassay data is the 39% cell death rate listed for the control (i.e., cell death with the addition of the reagents ouabain and veratridine without addition of toxin or samples).In previous studies documenting the utility of the N2a assay for detection of marine toxins that activate sodium channels, the cell death rate of the control is generally suggested to be about 20% [11,12].The high cytotoxicity of ouabain and veratridine alone against the N2a cells shown in Figure 1 suggests that the baseline concentrations of  these reagents are too high and might in fact mask the true cytotoxicity of the toxin, thereby significantly influencing the results.
Third, the authors do not define the threshold values used in the N2a assay to determine the definitive P-CTX-1 content of the fish samples tested, nor do they provide information on their source or the purity of the P-CTX-1 standard used.They state that results were statistically analyzed using Students t-test to identify significant differences between various controls and fish sample means, but do not present any of these statistical analyses.Together with the dubious dose-response curve shown in Figure 2, using the given data it is impossible to compare the sensitivity of the N2a bioassay the authors used against the accuracy of the Cigua-Check ® method.Moreover, because their standard curve does not deviate from the control group until the last data point (see Figure 2), it cannot be used to accurately determine P-CTX-1 concentrations, nor can it be used as the basis for any of their comparisons or conclusions.Since commercial CTX standards do not exist, clear information about the reference CTX used is relevant.
Fourth, the maximum concentration of tissue equivalent (TE) recommended for use in the N2a bioassay ranges from 2 mg TE/ml [13] to no more than 20 mg TE/ml because an excessive matrix quantity of fish samples may in fact be cytotoxic (see [2]).In the Bienfang, et al. [1] paper, the authors briefly described a sample extraction process using 10 g of each fish, but they did not specify how much of this fish extract was used per sample in the N2a assay.If they used 10 g of fish per sample, then the TE would be in the range of 50 g/ml, assuming the maximum volume of 200 µl per well in a 96-well plate, which may explain the extremely high rate of positive samples (70% for roi and 57% for kole).The overall lack of detail in the Materials and Methods section of the Bienfang, et al. [1] paper prevents accurate replication of their experiments for verification.
Fifth, comparing the results of a test that uses the means of triplicate samples (n = 3) against the results of a different test that uses single samples (n = 1) is not reasonable.In this paper, the authors used n = 1 for Cigua-Check ® samples and n = 3 for the N2a samples.Furthermore, the fish samples tested in the N2a assay were extracted using a more rigorous method, including lyophilization, extraction with methylene chloride, sonication, evaporation, and resuspension in methanol.Cigua-Check ® uses only a simple methanol extraction of the fish tissues, which is not explained in the Materials and Methods section of the paper.This difference in fish tissue extraction procedures alone would definitely result in varying toxin extraction efficiencies and subsequent quantification.
Sixth, Cigua-Check ® test results will fade slightly over time, even within 15 minutes.The paper does not make it clear whether the Cigua-Check ® sticks were read by different people at one time, it would have taken a minimum of 6 -8 minutes per person even if they only took 2 seconds to score each stick.This would result in a minimum of 24 -32 minutes for 4 different people to score the results, which would likely affect the interpretation between readers.
Given that the flawed standard curve in Figure 1 could not have been used to assess the accuracy of the Cigua-Check ® test results, the data in Table 1 of the Bienfang, et al. publication [1] are also flawed and inconclusive.This table depicts data distribution, but without being subjected to statistical analysis such as Fleiss' kappa to assess the reliability of agreement between readers, and is not an analysis by itself.If the raw Cigua-Check ® test results are evaluated simply by calculating the percentage of agreement among the four independent readers, the majority of readers (3/4 and 4/4) agreed on the same result for each stick in 78.2% (147/188) of the samples, while a minority of readers (2/2 and 1/2) disagreed on the same result for each stick in 21.8% (41/188) of the samples.Home pregnancy tests (HPTs), which are commercially available antibody-based assays, have been shown to be misread by 1 in 4 women; this translates into a 25% disagreement/75% agreement rate among independent readers [14].While most of these HPTs claim "over 99% accuracy" or a similar statement on their packaging or insert [15], their actual accuracy rates can range from 8.3% to 97% [14,16].The high false negative rates for these tests may be caused by many factors, including the length of time after a missed menstrual period at the point the test is conducted, late implantation, an ectopic or heterotopic pregnancy, heterophilic antibodies, spontaneous abortion, or non-compliance with the HPT instructions [17,18].It would be expected that Cigua-Check ® kits have a similar rate of interpretation.
Additional factual errors in this paper include the statement that, "Polyclonal antibodies in an assay for detecting haptens via a luminescent probe was pursued over years [19][20][21], and subjected to considerable analytical controversy [22][23][24][25][26]." While polyclonal antibody to CTX was used in the first two references [19,20], a monoclonal antibody to CTX (MAb-CTX) was used in [21].None of the assays utilized luminescent probes: the first assay used 125 I in a radioimmunoassay [20]; the second assay used horseradish peroxidase in an enzyme immunoassay [19]; and the third assay employed dyed latex particles in a membrane immunobead assay [21].The "analytical controversy" referred to in the five references cited was primarily due to mistaken identity and differences or flaws in protocol.A similar immunobead test (Ciguatect) developed by a separate research group [22], which is often confused with the Cigua-Check ® method, showed high false positive and negative detection rates [23].The evaluation of the Ciguatect test by Dickey, et al. [23] has often been incorrectly identified with the Cigua-Check ® method, and this mistaken perception has been continued by other researchers citing this study when referring to Cigua-Check ® (including [24,25].The Ciguatect test employed a different antibody that was not specific for CTXs because the antigen (hapten) used to generate this antibody was crude fish extract and not purified CTX, which was used to develop the MAb-CTX.A study of fish samples from Hong Kong utilizing the Cigua-Check ® method also resulted in high false negative and positive rates [26], but this was probably due to slight yet important differences in the proper protocol for the Cigua-Check ® method, and for comparison against the mouse bioassay, which is not specific nor as sensitive as the Cigua-Check ® method for CTX [27].
The authors also mention that "anecdotal information from researchers and fishermen who used these kits portrayed shortcomings (i.e., false positives and false negatives) in their application," without citing any factual support.Additionally, in the last paragraph of the Bienfang et al. [1] paper, the authors mention that they used the Table 1.Summary data from four independent readers evaluating identical Cigua-Check ® test sticks.Ciguatera status was determined by N2a neuroblastoma bioassays.N gives the number of fish specimens that were evaluated.4/4 (+) indicates that all four readers agreed that the samples were positive, and 4/4 (−) indicates that all four readers agreed that the samples were negative, etc. [1].
Cigua-Check ® kit to test orange "roughly" [sic], chicken, and two types of cheeses, which all yielded positive test results.Due to the nature of antibodies and the lipid polyether structure of CTX, samples containing high amounts of lipid and/or similar structures can cross-react with the MAb-CTX.Thus, the Cigua-Check ® kits should be used only to test tropical reef fish, for which the method has been designed and has been extensively tested.Past research has proven that CTX screening with various immunological methods utilizing the MAb-CTX, employed in the Cigua-Check ® method, is effective in preventing ciguatera poisoning and in detecting CTX in fish samples from clinically documented ciguatera cases [19,20,[28][29][30][31].Using a modified enzyme-linked immunosorbent assay (ELISA) technique, the MAb-CTX has been demonstrated to detect Pacific (P-CTX-1, P-CTX-2, P-CTX-3) and Caribbean (C-CTX-1) CTX congeners at concentrations ranging from 0 to 5.0 ppb, with statistically insignificant cross-reactivity at similar concentrations to okadaic acid, palytoxin, and domoic acid [32].A study by Dierking and Campora [33] of 30 Cephalopholis argus fish samples showed strong and significant correlation (r = 0.64, P < 0.001) between results obtained using the MIA and N2a assays.Furthermore, over 50,000 Cigua-Check ® tests for CTX have been commercially sold or distributed since 1997, with no verified reported incidences of false negatives.
Due to missing information and incomplete analyses in the Bienfang et al. [1] paper, its conclusions are not reliable.Because the N2a bioassay and the Cigua-Check ® immunobead assay are both considered screening methods, a more reliable, robust analysis should include the ciguatoxic status of individual fish samples obtained using an analytical method.

Figure 1 .
Figure 1.Dose-response curve for the N2a neuroblastoma bioassay to pacific ciguatoxin.Signal strengths in response to a range of PCTX-1 concentrations are given as % of control wells (n = 20) following addition of ouabain and veratridine.Relative standard deviations about the means for standards and controls averaged 11% and 14%, respectively [1].

Figure 2 .
Figure 2. Dose-response curve for the N2a neuroblastoma bioassay to pacific ciguatoxin.Signal strengths in response to a range of PCTX-1 concentrations are given as % of control wells (n = 20) following addition of ouabain and veratridine.Relative standard deviations about the means for standards and controls averaged 11% and 14%, respectively [1].