Retrospective Evaluation for ORCA Surveillance Comparison with Prescription Surveillance

Background: “Online Receipt Computer Advantage” (ORCA) surveillance, an online medical claim service produced and provided freely to medical facilities by the Japan Medical Association, has been available for public use since 2010. However, its preciseness has not been evaluated. The object of the present study is comparison with ORCA surveillance and prescription surveillance (PS), which is the most reliable daily surveillance for influenza nationwide in Japan, and evaluating preciseness of ORCA surveillance. Method: We specifically examined influenza as a targeted disease. We regarded PS as the gold standard daily real-time monitoring system for influenza nationwide and for each prefecture. This study assesses correlation and discrepancies between ORCA and PS results for influenza nationwide and by prefecture. Results: Nationwide, the correlation coefficient of PS and ORCA was 0.9653; the discrepancy rate was 27%. Among prefectures, results show that Mie and Fukui prefectures had two quite similar epidemic curves. Conclusion: Results demonstrate that ORCA surveillance is comparable to PS nationwide. However, some prefectures exhibited comparable results whereas others did not. ORCA surveillance cannot break down to the municipal level.

presented results for public use. The results include the estimated daily number of patients by prefecture or municipalities in Japan for influenza, hand-foot-and-mouth disease, Erythema infectiosum, Pharyngo conjunctival fever, RS virus infection, measles, rubella, and insolation from ORCA, which is an electronic medical claim system (http://infect.orca.med.or.jp/). The ORCA online medical claim service is produced and provided to medical facilities freely by the Japan Medical Association. As of the end of September, 2019, ORCA data were used at 17,174 medical facilities.
This information was collected from about 4218 medical facilities as of the end of September, 2019. Given Japan's total number of medical facilities as about 100 thousand, its participation rate was then about 4%. Nevertheless, its precision has not been examined to date. This report describes examination of its precision compared to that of PS: the gold standard of daily nationwide surveillance.

Method
We specifically examined influenza as a targeted disease. We regarded PS as the gold standard of a daily real-time monitoring system for influenza nationwide and for each prefecture. The study period was September 1, 2010 to December 30, 2018. The study areas were regarded as nationwide and in 47 prefectures. We We examine correlation between ORCA and PS results, and discrepancies between ORCA and PS results for influenza nationwide and by prefecture. The discrepancy was calculated as the weighted average of the absolute value of difference between ORCA surveillance results and PS results weighted by the number of influenza patients in PS.

Ethics
Data from ORCA surveillance and PS were aggregated and de-linked from personal information related to patients, medical facilities, and pharmacies: these are anonymous data. Moreover, these were published to the general public.
Therefore, no ethical issues are posed by the use of these data for this study.

Results
About 2000 facilities, at maximum, reported influenza patients. The numbers of influenza patients estimated by ORCA surveillance and PS nationwide by season are shown in Figure 1. Figure      were Ehime, Mie, Fukui, Yamagata, and Akita. Discrepancies in the top two prefectures were especially low: less than 20%. Nagasaki, Hiroshima, Okayama, Gifu, and Kagoshima were the worst five prefectures in terms of correlation: around 0.55. Discrepancy rates in these five prefectures were higher than 100%. curves showed high correlation and low discrepancy. We conclude that ORCA surveillance has sufficient precision. At the prefecture level, results show that Mie and Fukui had similar epidemic curves. Nevertheless, correlation and discrepancy even in these best two prefectures were worse than the results obtained nationwide. In Saga prefecture, PS covers about 60% of all pharmacies, constituting the highest participation rate in all of Japan. Therefore, PS in Saga was used as the most reliable monitor of influenza by the Saga prefectural office [13]. Nevertheless, the correlation and discrepancy figures there were, respectively, about 0.6% and 100%. Therefore, ORCA surveillance showed no high similarity with PS, at least in Saga. This result suggests that ORCA surveillance provides insufficient precision at the prefecture level.

Discussion
Moreover, even though ORCA surveillance provides information about municipalities, the ORCA surveillance precision for each municipality was not confirmed because PS does not provide information for municipalities. For the more than 1700 municipalities in Japan, medical facilities reporting to ORCA surveillance were about one per municipality, on average. Therefore, the information might represent information at only one medical facility, not community data. For PS, the smallest areas were defined as those where more than10 pharmacies had joined PS. Such areas are sometimes smaller than a municipality, sometimes defined as joint neighborhood municipalities. For example, one area in Saga prefecture was defined as a junior high school area. Therefore, utilization of information from ORCA surveillance at a municipality level, especially for such areas that include fewer than 10 facilities, demands some caution.
In this sense, because there were about 40 facilities in a prefecture in average, we might consider virtual districts as joint municipalities. To contain more than 10 facilities, one prefecture will have four such vertical districts, in average. That is a limitation of fineness when monitoring community outbreaks using ORCA surveillance. In decades later, because participating facilities has been increasing by about 1% annually, the preciseness of ORCA surveillance at prefecture level might improve and breaking down to municipality level might be possible.
The study described herein has some limitations. We first considered ORCA surveillance retrospectively. However, results might be different in a prospective setting. Because ORCA itself is the system for medical claims and because medical facilities claim public health insurance at the end of month, entry of diagnosis to ORCA is sometimes delayed until the end of the month. In other words, the number of patients in ORCA surveillance is probably under-reported in real-time. Totals might increase up to the end of month. In other words, the precision of real-time ORCA surveillance is expected to be less than that of retrospective evaluation, which we considered in the present study. When we trace the number of patients from visiting day to the end of month, the number of pa-tients increased by 15%. It may be substantial uner-reporting. Prospective evaluation for ORCA surveillance is anticipated as our next challenge.
Secondly, we regarded PS as the gold standard. That might be true if one considers daily real-time surveillance for influenza. However, calculation of the total number of influenza patients during a season requires no real-time capability. In Japan, the National Official Surveillance for Infectious Diseases (NOSSID) is operated based on a related law. However, it is the sentinel surveillance. It cannot report the estimated number of infectious diseases [2]. Moreover, information from all medical claims is disclosed in the "National Database of Health Insurance Claims and Specific Health Checkups of Japan" [14] by the Ministry of Health, Labour and Welfare (MHLW) of Japan [15] [16] [17] [18]. We designate this as the National Database of Electronic Medical Claims (NDBEMC). These data were associated with 98.4% of all medical claims made in 2015 [19]. They areal most complete data, which include data from PS and ORCA as components. However, MHLW requires issuance of a permit to use NDBEMC. Although many researchers have sought permission to use it, its approval process alone requires more than one year. Therefore, evaluation of precision during the season using NDBEMC has been postponed.
Thirdly, we evaluated ORCA surveillance only for influenza patients. However, ORCA surveillance include other infection diseases and insolation as described above. Although PS includes the numbers of influenza and varicella or herpes zoster patients, it does not include some information associated with other infectious diseases. Therefore, we limited analyses in this study to the number of influenza patients. Future studies must evaluate the precision of ORCA surveillance for other infectious diseases or insolation.
Fourthly, we showed that ORCA surveillance did not provide reliable information at municipalities level due to limitation of participated medical facilities per municipalities. In average, there was only one participated medical facility to ORCA surveillance per municipalities. Instead of municipalities, we suppose virtual districts as joint municipalities which contain more than 10 facilities. Examination for these vertical districts in ORC surveillance should be the next challenge.

Conclusion
We confirmed results of ORCA surveillance as comparable to PS results nationwide. Regarded at the prefecture level, these were comparable in some prefectures, but not in other prefectures. The precision of ORCA surveillance was not confirmed at the municipal level. More evaluation must be done of ORCA surveillance prospectively, and using NDBEMC, and for diseases other than influenza.