Retrospective Evaluation for ORCA Surveillance Comparison with Prescription Surveillance ()
1. Introduction
Prescription Surveillance (PS), as operated in Japan by the Japan Medical Association, Japan Pharmaceutical Association, School of Pharmacy, and EM Systems Co. Ltd., has been functioning as a nationwide real-time syndromic surveillance system [1] - [7] as along with (N) SASSy which has been serving 65% of Japan [6] - [12]. Since 2009, PS has been reporting the estimated number of influenza and chicken pox patients, in addition to the numbers of patients who are prescribed drugs of certain types, based on prescriptions at pharmacies. In fact, PS estimates the numbers of patients every day from the number of prescriptions by prefecture by age group. As of the end of April 2015, approximately ten thousand pharmacies were participating, collectively accounting for about 20% of all pharmacies nationwide. The estimated numbers of patients are presented on the PS web page the following morning (http://prescription.orca.med.or.jp/kanjyasuikei/).
That estimate by PS represents multiplication of the reciprocal of the pharmacy PS participation rate and the reciprocal of the proportion of prescriptions at external pharmacies in the prefecture by the total number of prescriptions issued in the prefecture. The results are estimates of the numbers of patients prescribed neuraminidase inhibitors (NI), anti-herpes virus drugs (AHV), antibiotic drugs (AB), antipyretic analgesics (AP), multi-ingredient cold medications (MIC), and antidiarrheal and intestinal drugs (AD). Antibiotics are classified into five types: penicillin (ABP), cephem (ABC), macrolide (ABM), new quinolone (ABQ), and others (ABO) [1] [2].
Since 2010, Online Receipt Computer Advantage (ORCA) surveillance has presented results for public use. The results include the estimated daily number of patients by prefecture or municipalities in Japan for influenza, hand-foot-and-mouth disease, Erythema infectiosum, Pharyngo conjunctival fever, RS virus infection, measles, rubella, and insolation from ORCA, which is an electronic medical claim system (http://infect.orca.med.or.jp/). The ORCA online medical claim service is produced and provided to medical facilities freely by the Japan Medical Association. As of the end of September, 2019, ORCA data were used at 17,174 medical facilities.
This information was collected from about 4218 medical facilities as of the end of September, 2019. Given Japan’s total number of medical facilities as about 100 thousand, its participation rate was then about 4%. Nevertheless, its precision has not been examined to date. This report describes examination of its precision compared to that of PS: the gold standard of daily nationwide surveillance.
2. Method
We specifically examined influenza as a targeted disease. We regarded PS as the gold standard of a daily real-time monitoring system for influenza nationwide and for each prefecture. The study period was September 1, 2010 to December 30, 2018. The study areas were regarded as nationwide and in 47 prefectures. We identified the number of influenza patients in ORCA surveillance as the number of patients diagnosed and recorded in ORCA multiplied by the ORCA surveillance participation rate for all medical facilities in the considered area. Influenza season was defined as September 1 to the end of August of the subsequent year.
We examine correlation between ORCA and PS results, and discrepancies between ORCA and PS results for influenza nationwide and by prefecture. The discrepancy was calculated as the weighted average of the absolute value of difference between ORCA surveillance results and PS results weighted by the number of influenza patients in PS.
Ethics
Data from ORCA surveillance and PS were aggregated and de-linked from personal information related to patients, medical facilities, and pharmacies: these are anonymous data. Moreover, these were published to the general public. Therefore, no ethical issues are posed by the use of these data for this study.
3. Results
About 2000 facilities, at maximum, reported influenza patients. The numbers of influenza patients estimated by ORCA surveillance and PS nationwide by season are shown in Figure 1. Figure 2 depicts the epidemic curves of ORCA surveillance and PS nationwide. The correlation coefficient of PS and ORCA nationwide results was 0.96. The discrepancy rate was 27.45%.
Figure 3 presents correlation coefficients and discrepancy rates between ORCA surveillance and PS results by prefecture. Large correlation coefficients represent higher similarity. A lower discrepancy rate implies higher similarity. Figure 3 depicts the best five prefectures for high correlation as Wakayama, Tottori, Mie, Fukui, and Nagano. Especially, correlation in the top four prefectures was higher than 0.8. The best five prefectures in terms of low discrepancy
![]()
Figure 1. Estimated number of influenza patients from ORCA surveillance and PS by Season. Note: Black bars represent the estimated number of influenza patients from ORCA surveillance. Gray bars represent data from PS.
![]()
Figure 2. Epidemic curves for Influenza from ORCA and PS. Notes: Black line is the epidemic curve in PS. Gray line is the epidemic curve in ORCA.
![]()
Figure 3. Correlation coefficient and discrepancy rate of the number of influenza patients from ORCA and PS by prefecture. Note: Gray bars show correlation coefficients for PS and ORCA. Lines show discrepancy rates among PS and ORCA. Therefore, the higher bar and lower line means that PS and ORCA are similar in the prefecture. Discrepancy rates was defined sum of absolute value of difference among the number of patients in PS and ORCA divided by total number of patients in PS in the study period.
were Ehime, Mie, Fukui, Yamagata, and Akita. Discrepancies in the top two prefectures were especially low: less than 20%. Nagasaki, Hiroshima, Okayama, Gifu, and Kagoshima were the worst five prefectures in terms of correlation: around 0.55. Discrepancy rates in these five prefectures were higher than 100%.
4. Discussion
Figure 1 depicts similarity in earlier seasons. However, discrepancies among the two surveillance modes tend to be larger more recently. For instance, discrepancies in 2011/2012 and 2012/2013 seasons were less than 0.5 million, but grew to about 2 million in the 2016/2017 season. At least nationwide, two epidemic curves showed high correlation and low discrepancy. We conclude that ORCA surveillance has sufficient precision.
At the prefecture level, results show that Mie and Fukui had similar epidemic curves. Nevertheless, correlation and discrepancy even in these best two prefectures were worse than the results obtained nationwide. In Saga prefecture, PS covers about 60% of all pharmacies, constituting the highest participation rate in all of Japan. Therefore, PS in Saga was used as the most reliable monitor of influenza by the Saga prefectural office [13]. Nevertheless, the correlation and discrepancy figures there were, respectively, about 0.6% and 100%. Therefore, ORCA surveillance showed no high similarity with PS, at least in Saga. This result suggests that ORCA surveillance provides insufficient precision at the prefecture level.
Moreover, even though ORCA surveillance provides information about municipalities, the ORCA surveillance precision for each municipality was not confirmed because PS does not provide information for municipalities. For the more than 1700 municipalities in Japan, medical facilities reporting to ORCA surveillance were about one per municipality, on average. Therefore, the information might represent information at only one medical facility, not community data. For PS, the smallest areas were defined as those where more than10 pharmacies had joined PS. Such areas are sometimes smaller than a municipality, sometimes defined as joint neighborhood municipalities. For example, one area in Saga prefecture was defined as a junior high school area. Therefore, utilization of information from ORCA surveillance at a municipality level, especially for such areas that include fewer than 10 facilities, demands some caution.
In this sense, because there were about 40 facilities in a prefecture in average, we might consider virtual districts as joint municipalities. To contain more than 10 facilities, one prefecture will have four such vertical districts, in average. That is a limitation of fineness when monitoring community outbreaks using ORCA surveillance. In decades later, because participating facilities has been increasing by about 1% annually, the preciseness of ORCA surveillance at prefecture level might improve and breaking down to municipality level might be possible.
The study described herein has some limitations. We first considered ORCA surveillance retrospectively. However, results might be different in a prospective setting. Because ORCA itself is the system for medical claims and because medical facilities claim public health insurance at the end of month, entry of diagnosis to ORCA is sometimes delayed until the end of the month. In other words, the number of patients in ORCA surveillance is probably under-reported in real-time. Totals might increase up to the end of month. In other words, the precision of real-time ORCA surveillance is expected to be less than that of retrospective evaluation, which we considered in the present study. When we trace the number of patients from visiting day to the end of month, the number of patients increased by 15%. It may be substantial uner-reporting. Prospective evaluation for ORCA surveillance is anticipated as our next challenge.
Secondly, we regarded PS as the gold standard. That might be true if one considers daily real-time surveillance for influenza. However, calculation of the total number of influenza patients during a season requires no real-time capability. In Japan, the National Official Surveillance for Infectious Diseases (NOSSID) is operated based on a related law. However, it is the sentinel surveillance. It cannot report the estimated number of infectious diseases [2]. Moreover, information from all medical claims is disclosed in the “National Database of Health Insurance Claims and Specific Health Checkups of Japan” [14] by the Ministry of Health, Labour and Welfare (MHLW) of Japan [15] [16] [17] [18]. We designate this as the National Database of Electronic Medical Claims (NDBEMC). These data were associated with 98.4% of all medical claims made in 2015 [19]. They areal most complete data, which include data from PS and ORCA as components. However, MHLW requires issuance of a permit to use NDBEMC. Although many researchers have sought permission to use it, its approval process alone requires more than one year. Therefore, evaluation of precision during the season using NDBEMC has been postponed.
Thirdly, we evaluated ORCA surveillance only for influenza patients. However, ORCA surveillance include other infection diseases and insolation as described above. Although PS includes the numbers of influenza and varicella or herpes zoster patients, it does not include some information associated with other infectious diseases. Therefore, we limited analyses in this study to the number of influenza patients. Future studies must evaluate the precision of ORCA surveillance for other infectious diseases or insolation.
Fourthly, we showed that ORCA surveillance did not provide reliable information at municipalities level due to limitation of participated medical facilities per municipalities. In average, there was only one participated medical facility to ORCA surveillance per municipalities. Instead of municipalities, we suppose virtual districts as joint municipalities which contain more than 10 facilities. Examination for these vertical districts in ORC surveillance should be the next challenge.
5. Conclusion
We confirmed results of ORCA surveillance as comparable to PS results nationwide. Regarded at the prefecture level, these were comparable in some prefectures, but not in other prefectures. The precision of ORCA surveillance was not confirmed at the municipal level. More evaluation must be done of ORCA surveillance prospectively, and using NDBEMC, and for diseases other than influenza.