Accuracy of Physicians in Diagnosing HIV and AIDS-Related Death in the Adult Population of Addis Ababa , Ethiopia *

Background: The lack of cause of death information is the main challenge in monitoring the effectiveness of interventions aimed at reducing HIV and AIDS-related deaths in countries where the majority of deaths occur at home. Objective: To evaluate the accuracy of physician reviewers of verbal autopsies in diagnosing HIV and AIDS-related deaths in the adult population of Addis Ababa, the capital of Ethiopia. Methods: This study was done within the context of a burial surveillance system in Addis Ababa. Trained interviewers completed a standard verbal autopsy questionnaire and an independent panel of physicians reviewed the completed form to assign cause of death. Physicians’ review was compared to a reference standard constructed based on prospectively collected HIV-serostatus and patients’ hospital record. Sensitivity and specificity were calculated to validate the physicians’ verbal autopsy diagnoses against reference standards. Results: Physicians accurately identified AIDS-related deaths with sensitivity and specificity of 0.88 (95% CI: 0.80 0.93) and 0.77 (95% CI: 0.64 0.87), respectively. Generally, there was high level of agreement (Cohen’s Kappa Statistic (K > 0.6) between the first two physicians with some yearly variations. In 2008 and 2009 there was an almost perfect agreement (K > 0.80). Conclusion: This study demonstrated the agreement level between two independent physicians in diagnosing AIDS-related death is very high and thus using a single verbal autopsy coder is practical for programmatic purposes in countries where there is critical shortage of doctors.


Introduction
In many developing countries causes of death are not systematically recorded in a recognized database [1][2][3].Although knowing cause of death is a critical input for planning health interventions and evaluating the impact, this information is not readily available in these settings since most deaths occur at home for cultural and health service related reasons [4][5][6][7][8].
The verbal autopsy (VA), which involves interviewing the next of kin or other caregivers of a deceased, has been widely recognized as an alternative method for de-termining the cause of death at population level where death certification is not regularly done [9][10][11][12].
There has been growing interest in using verbal autopsy procedures to identify causes of death in the context of surveillance systems among adult populations [13].There are different methods for interpreting verbal autopsy interviews to derive probable cause of death.These include physician review, algorithms, and use of neural networks [10].The interpretation of verbal autopsies largely relies on physician review and has an acceptable sensitivity and specificity for cause of death [14].It involves the interpretation of verbal autopsy data by one or more trained physicians [5,12,15].Coding of verbal autopsy by physicians has several drawbacks.For instance, physician coders' prior knowledge of disease patterns in the community can result in high agreement.The other concern is that in countries where there is shortage of doctors, engaging them for verbal autopsy coding may be a waste of such scarce resources.Thus, some argue a single trained physician coder is as effective as multiple coders [16].In this paper, we evaluated the ability of physicians to diagnose HIV and AIDS-related deaths by reviewing verbal autopsy information in the adult population of Addis Ababa, and the level of agreement over these diagnoses.

Population
This study was conducted as part of the Addis Ababa Mortality Surveillance Program (AAMSP) in Addis Ababa, Ethiopia.Addis Ababa City Administration is the capital and largest urban area of Ethiopia; and the largest urban center in African countries.Detailed description of study area is elsewhere [17][18][19].The surveillance comprises four different but interrelated projects: death registration at burial sites, sample verbal autopsy interviews among bereaved families or the deceaseds' caregivers, hospital surveillance, and community-based surveillance of deaths (called "Iddir" surveillance)."Iddirs" are indigenous voluntary associations established to provideamong others-mutual aid and support in burial matters [20,21].Data for this study were obtained from the sample verbal autopsies conducted on random sample of the burial surveillance records that matched with the hospital records.The overall process of the study is described in the study protocol in Figure 1.[22].The instrument was translated into the local language and pretested to ensure understandability.The questionnaire includes a section where the respondent's brief account of the illness prior to death is recorded; and a section containing closed-ended questions where respondents are prompted to respond about particular signs and symptoms.This study used adult deaths (age ≥ 12 years old) with the inclusion and exclusion criteria based on the ability to identify the address of the deceased.The detail is described elsewhere [17][18][19].Pairs of interviewers (male and female) conducted the verbal autopsy interviews after respecting a mourning period ranging from 1 -7 months (Mean = 3.7) from the time of death.The verbal autopsy interviewers were trained for one week.About 15% of the burial surveillance records did not have a complete residential address and hence were not eligible for the VA interview.In this study, verbal autopsy interview was successfully completed for 5599 (88%) adult deaths for the aforementioned study years.

Hospital Surveillance
The second data set was produced through prospective surveillance of hospital-based patients conducted in Zewditu Memorial Hospital, a large governmental hospital in the center of Addis Ababa, from May 2003 to January 2004.A ward nurse was trained to collect admission and discharge diagnosis along with basic socio-demographic characteristics of the patients admitted to the hospital.HIV-serostatus testing was done using Determine rapid HIV1-2 test kit after pre-test and post-test counseling following the guideline issued by the Federal Ministry of Health of Ethiopia.A brief history, physical examination, laboratory investigations, final diagnosis, course of treatment, condition of discharge and immediate cause of death were retrieved from the patient card and summarized in a data abstraction form prepared for this purpose.

Physician Review
An independent panel of physicians reviewed the completed verbal autopsy questionnaires to assign cause of death.Every completed verbal autopsy questionnaire was assessed independently, and immediate, contributory and underlying causes of death were assigned whenever possible and were recorded.In the event of disagreement between the two physicians, a third physician reviewed the questionnaire independently.If the cause of death assigned by the third physician concurred with that of either of the first two physicians, the final cause of death was determined by a verbal autopsy research assistant.In the event where the diagnoses of all three physicians did not concur, final cause of death was assigned after a panel discussion.If agreement still could not be reached, cause of death was recorded as "undetermined".We coded the final diagnosis based on the International Classification of Causes of Death, version 10 [23].

Reference Standard: A Gold Standard
As shown in Table 1 below, a reference standard was constructed by a clinically experienced nurse using information from the admission diagnosis (or discharge diagnosis when available) and HIV-serostatus.All cases with a negative HIV-test result were classified as non-AIDS-related deaths.Among those with an HIV-test positive result, deaths were classified as AIDS-related deaths if the admission diagnosis was indicative of an opportunistic infection or the patient had AIDS complex syndrome (Table 1).

Analysis
First, analysis was done to compare cause of death classified as HIV and AIDS-related death by physicians against the reference standard.For this purpose, we dichotomized causes of death as AIDS and Non-AIDS to calculate sensitivity and specificity of the physicians' classification of AIDS-related death against the reference standard (the HIV-serostatus and patients' medical records).The first analysis was conducted on hospital patient records matched with completed VA interviews (N a In cases where HIV-serostatus was unknown, the causes of death were obtained from the information on the patients' cards when available; b Because of missing HIV-serostatus information in the reference standard, only 167 out of the 193 matched cases could be classified as either AIDS or non-AIDS-related deaths. = 193) and that after they reviewed by physicians, the 167 were classified as AIDS (n = 108) and non-AIDS (n = 59) (Table 1 and Figure 1).Second, analysis focused on the concordance of physicians in identifying AIDS-related death using the Cohen's Kappa Statistic (K) [24].The Kappa-statistic measures agreement between the two physicians on a scale of 0 to 1: a score closer to zero indicates the observed agreement is no better than chance and when one it indicates perfect agreement.We classified the intermediate Kappa based on Landis and Koch classification.Values above 0.8 were considered to reflect almost perfect agreement; those between 0.6 and 0.81, substantial agreement; those between 0.4 and 0.61, moderate agreement; and those below 0.41 as fair or slight/poor agreement [24].Analyses were carried out using STATA version 11.

Ethical Considerations
Ethical clearance for the on-going surveillance was obtained from the Institutional Review Boards (IRBs) of Addis Ababa University and the Ethiopian Science and Technology Agency.Official permissions were obtained from municipal and religious authorities to undertake the burial surveillance; and from the Addis Ababa City Administration Health Bureau and the Zewditu Memorial Hospital management board.All informants provided information having given consent.

Second Analysis Results: Physician Reviewers Agreement
Table 3 below shows the Cohen's Kappa statistic (K) of HIV and AIDS-related cause of death assigned by the first two physician verbal autopsy reviewers.AIDS-related cause of death assigned by the first two independent physician verbal autopsy reviewers showed substantial agreement accounted for an overall 91.0% (K = 0.72; 95% CI: 0.70 -0.74) through the study years (Table 3).Figure 2 below clearly demonstrates that the pattern of HIV and AIDS-related death as assigned by independent physicians was similar over the years.AIDSrelated deaths ranged from 43% in 2003 to 9.3% in 2010.Apart from the proportion of HIV and AIDS-related death the computed Kappa statistic values reflect substantial agreement among the first two coders (K > 0.6) except 2007.There is almost perfect agreement for the years 2008 and 2009 (Table 2).

Discussion
In this study, we used a unique reference standard as a "gold standard" to compare HIV-serostatus of hospital  patients with physician verbal autopsy diagnosis.This study demonstrated that the verbal autopsy diagnosis for HIV and AIDS-related mortality has high sensitivity and specificity against the reference standard.The agreement level between the first two physicians on the verbal autopsy diagnosis of HIV and AIDS-related death was generally high indicating using a single physician might be sufficient to get fairly reliable information on the causes of death in resource-constrained countries.
To detect HIV and AIDS-related mortality patterns and other causes of deaths, a mortality surveillance program and over six Demographic Surveillance Sites (DSS) have been established in the last decade.In all these DSS sites, verbal autopsy data is regularly performed to identify the current status of HIV prevalence and pattern of AIDS [25].Although the verbal autopsy process has several stages that may influence the level of physician accuracy [10]; almost none of the DSS sites in Ethiopia have done physician accuracy studies to show its limitations or to maximize the utility of verbal autopsy as a source of mortality data.It is critical to do validation studies so that the degree of uncertainty, which will vary by cause of death, can be factored into mortality burden estimations [15].In this sense, our physician accuracy study helps in building confidence in using the verbal autopsy procedure for identifying AIDS-related causes of death in the Ethiopian context [15].
Any study of the accuracy of verbal autopsy faces the question of how to obtain a suitable reference diagnosis.Several validation studies have used causes of death based on medical records as the "gold standard" [25][26][27][28][29][30].However, validation studies should also take into consideration that the "gold standard" of medical records of diagnosis is often an imperfect one, at best [15].A carefully conducted verbal autopsy may be superior to poorly maintained or scanty medical records, as seen, for example, in stillbirth [31].Physician diagnosis of medical records may or may not be supported by different diagnostic tests which affect the accuracy of the "gold standard" itself.To avoid this limitation of medical records, we used HIV-serostatus result combined with clinical history of HIV and AIDS cases as "gold standard".Physician accuracy of verbal autopsy is considered to have an acceptable level of diagnostic accuracy at the population level, if sensitivity is at least 50% and specificity at least 90% [12].Our finding of physician accuracy in detecting HIV and AIDS-related causes of death is comfortably greater than the acceptable level of sensitivity (88%), but the specificity is lower (78%).However, these criteria of diagnostic accuracy are not uniformly regarded as acceptable [9], because low sensitivity and specificity does not necessary imply low level of accuracy; and relatively high sensitivity and specificity may result in serious misclassification errors.In the case of low sensitivity and specificity, the false positives and false negatives may cancel each other and hence not affect the physician accuracy [27,32].Specific to HIV and AIDS-related causes of death, an earlier multi-center adult verbal autopsy validation study undertaken in rural Ethiopia (Jimma), Tanzania and Ghana indicates a sensitivity ranging from 56% in Ghana to 82% in Ethiopia [27].A similar study from two countries on causes of adult mortality indicates a sensitivity of >50% in Tanzania [15].
Some studies have reported one physician may be adequate enough to verbal autopsy coding, (e.g., by Joshi et al. 2010) [16] that is supported by our study.Despite the number of physician coders, our study is unique based on several reasons.Firstly, our study is conducted in the larger urban center of African countries-the Ababa City Administration with a population of nearly 3 million as reported by the Central Statistical Agency of Ethiopia [33].Our study represents the whole population within the city limit and hence the data we obtained was representative with no doubt of generalizability.Moreover, Addis Ababa is the capital and largest urban center of Ethiopia.Ethiopia is a sub-Saharan Africa with the second rank most populous country in Africa with about 74 million populations as reported by the CSA of Ethiopia [34].Secondly, the random samples for verbal autopsy interviews were selected from burial records that were collected by means of a novel research design-the prospective surveillance of burials.The surveillance of burials was conducted at all cemeteries within the city limit of Addis Ababa.This assures representativeness and generalizability of our data.While the other studies were conducted in smaller rural villages with lesser number of populations and the verbal autopsy interviews were conducted on deaths identified from household that may not be reported accurately and completely that may lead to affecting its representativeness and selection bias [16].
In conclusion, in this study we have demonstrated that physicians can assign cause of death with high validity, by reviewing verbal autopsy information.We have also shown that within a well established verbal autopsy system with well trained and supervised verbal autopsy interviewers, trained physician reviewers can consistently identify HIV and AIDS-related deaths.The study also indicates substantial agreement between the verbal autopsy physician reviewers suggesting that for programmatic purposes, a cause of death determined by a single physician may be sufficient in resource-poor settings.