Evaluating Displayed Depression Symptoms on Social Media Sites

Social networking sites (SNSs) are immensely popular and allow for display of personal information, including references to depression. Evaluating displayed content on a SNS for research purposes requires a systematic approach and a precise data collection instrument. The purpose of this paper is to describe one approach to the development of a research codebook for depression so that others may develop and test their own codebooks for use in SNS research. The depression SNS research codebook was grounded in ethics, theory and clinical criteria. The key elements in the codebook developmental process included an iterative team approach to develop variables of interest and data collection sheet layout. Training protocols involve coding practice and reliability assessments. Interrater reliability remains a critical assessment tool. Codebook successes include consistently high interrater reliability. Challenges include time investment in coder training, SNS server changes, and social or cultural norms regarding public displays of mental health. We provide detailed information about a systematic approach to codebook development so that other researchers may use this structure to develop and test their own codebooks for use in SNS research. Future directions for the codebook include expanding areas of interest such as anxiety or other depression evaluation criteria, and expansion to other SNSs such as Twitter.


Introduction
Social networking web sites (SNSs) may present new opportunities to investigate depression, particularly among adolescents and young adults.The vast majority of adolescents and young adults use a social networking site such as Facebook [1][2][3].SNSs allow members to build a personal web profile, to communicate with others and to build a social network.Profile users may incorporate text, photos, graphics, audio and video to build a personal representation of their identity.
Through content displayed on a SNS profile, profile owners may give researchers and health care providers insight into aspects of their behavior that are not always apparent in offline life.These disclosures may include references to depression symptom in the form of "status updates," personally written text describing the profile owner's current experiences.Previous work found that approximately a quarter of publicly available Facebook profiles of undergraduates displayed depression symptom references, and a small proportion of profiles displayed depression symptoms in patterns consistent with the Diagnostic and Statistical Manual (DSM-IV) criteria for a major depressive episode (MDE) [4,5].Additionally, display of depression symptoms was positively associated with reporting depression symptoms using a clinical screen [6].
Why might a profile owner display feelings of depression in such a public online venue?Previous work illustrates that computers elicit higher levels of self-disclosure and uninhibited personal expression compared to offline interactions [7].Adolescents report that they often disclose more about themselves on SNSs than they do in person [8].Further, the social risk hypothesis suggests that humans try to minimize social risks in order to maintain ties to a group [9].It has been posited that people with depression will change their behaviors to make "safer" social decisions in order to avoid social failures.Thus, social networking websites may present a venue to those with depression that feels safer for expressing emotion during periods of isolation.
These online displays of health risk behaviors are important as they give health care providers and researchers new insight into adolescent health concerns that are clearly linked to morbidity and mortality among this population.The most common form of depression within the adolescent and young adult age group is major depressive disorder, which has a yearly incidence of approximately 8% [10,11].Additional 22% of adolescents and young adults suffer from "sub-diagnostic" levels of depressive symptoms [12].Both those with a major depresssion diagnosis and those with depression symptoms experience impaired functioning and morbidity [13].Adverse outcomes of depression include increased rates of substance use, co-morbid psychiatric conditions and suicide [14][15][16][17].
Despite the prevalence of depression in this population, college students struggling with depression symptoms are frequently undiagnosed as many students do not perceive a need for help or do not seek clinical services [10,11,18].While 30% of college students report feeling so depressed in the past year that it was difficult to function, only 10% of college students repot seeking any type of mental health care in the past year [19,20].Concerns about the stigma related to mental illness are also associated with less perceived need for help and decreased treatment seeking behavior [21].Recent media stories highlight tragic suicides of young adults that were referenced on Facebook prior to being carried out; illustrating the urgent need to better understand the validity of displayed references to depression and other mental health concerns [22].Thus, social media could provide an innovative and complementary approach towards early identification of depression.
Barriers to evaluation of social media for displayed depression disclosures include concerns about subjectivity in coding or bias in evaluation of social media sites.Thus, in order to establish SNSs as a credible approach for understanding depression displays, a framework for consistency and reliability in evaluation and measurement of SNS profiles is essential.The purpose of this paper is to describe one approach to the development and use of a research codebook to evaluate adolescents' displayed references to depression symptoms on Facebook.Our intention is to provide a framework on which researchers or clinicians may further develop and adapt their own codebooks for use with mental health SNS research.Given that SNSs may rise and fall in popularity, our goal is to provide a framework that is usable and adaptable to researchers as new technologies and new mental health topics of interest emerge over time.

Foundation of the Codebook
In developing the codebook, we began with a review of salient literature to contribute to the foundations of the codebook's theoretical, methodological and clinical foundations.We first investigated theoretical models to guide its structure so that we could understand the role of SNSs in adolescents' lives.The Media Practice Model describes that adolescents choose and interact with media based on who they are, or who they want to be, at the moment [23].This model influenced our early interest in viewing SNSs as representations of adolescents' identities and is supported by literature that suggests that adolescents may disclose more about themselves in online venues than they do face-to-face [24].
Codebook development was further influenced by a review of the literature for qualitative methods.As qualitative data can provide rich data, sometimes called "thick description," of experiences or events we wanted to maximize the reliability in collecting and interpreting such data [25].We followed other investigators' strategies for interpreting qualitative interview data, such as holding frequent team meetings to discuss and refine the coding structure as it evolved [25,26].
Finally, in developing codes for references of depression we consulted established clinical criteria: the Diagnostic and Statistical Manual of Mental Disorders (DSM-IV) [4].By grounding the coding scheme on clinical criteria, our goals were to maximize the clinical application of data obtained via SNSs and to promote translation of findings into further clinical research.Further, we wanted to provide robust and detailed criteria that coders could apply during each profile evaluation, so that coders did not need to apply personal or subjective judgment to displayed content.Thus, references to depression symptoms were defined using the DSM-IV symptom criteria for a MDE [4].

Confidentiality
In creating the codebook, a first objective was to address ethical concerns that may arise in evaluating data from SNS profiles.Consistent with recommendations by the Institutional Review Board, a priority area was to develop protocols towards maintaining confidentiality of the data obtained from profiles.This focus on confidentiality was applied to the data collection stage.The coding protocol specified that data recorded from profiles would never be identified by name but by an assigned code number.Confidentiality protocols have also been applied to presentation of data through papers or at conferences.For example, when presenting direct text quotes from participants' profiles, quotes are always slightly altered so that an online text search will not match the quote to the participant's profile.Further, no other identifiable information is documented with text examples.If names are displayed with in the text they are taken out and replaced with the label of "profile owner" or "friend".

Profile Security
SNS profiles may be "publicly available" such that profile content is accessible by the entire online social network, or "privately available" in which profile content is only available with explicit permission of the profile owner.Many early studies focused on SNS profiles that were publicly available [27,28].In the past few years, both privacy settings and the culture of privacy on Facebook have changed, and many users favor settings that limit public access.However, findings from one study illustrated that many participants were open to allowing their profile content to be accessed by researchers [29].Thus, researchers may consider asking participants to Facebook "friend" a research assistant to allow content to be mutually viewable between the research team and the participant.

Defining Variables of Interest
To develop a coding scheme for the complex variable of depression, we used an iterative process.The first step was establishing clinical criteria to operationally define depression symptoms.References to depression symptoms were defined using the Diagnostic and Statistical Manual of Mental Disorders (DSM-IV) symptom criteria for a Major Depressive Episode (MDE) [4].The criteria for MDE included depressed mood, loss of interest/ pleasure in activities, appetite changes, sleep problems, psychomotor agitation or retardation, energy loss, feeling worthless or guilty, decreased concentration or suicidal ideation [4].
Thus, text references were considered a depression symptom reference if they fit one of the depression crite-ria by keyword or a synonym.For example, one symptom keyword of major depression is "hopeless," therefore a status update stating "I feel hopeless" would be coded as a reference to depression.The term "giving up" is a synonym of "hopeless," therefore, a status update disclosing "I feel like giving up" would be coded as a reference to depression as well.Status updates that clearly referenced a person other than the profile owner (i.e."Molly is sitting next to me in class and she looks sad"), or references to the common situational experience of having a bad day (i.e."I'm having a depressing day") were not considered depression references.
The next step involved applying these coding criteria to the SNS of choice; initial codebooks were developed to be applied to Facebook.Applying these clinical criteria to a pilot sample of profiles resulted in identification of further keywords and synonyms linked to the variable of interest.Team discussions of both findings from profile content, as well as notes on the evaluation process itself followed.If modifications were suggested, the process began again with the coding of a practice set of profiles and then presenting findings and notes on the process to our research group.This process helped further develop the coding schemes.

Preparing the Codebook
Following this iterative process, the codebook protocol was created.This included documentation prompts for each proposed depression symptom including: a description of the variable including any relevant clinical criteria, a list of example terms that represented the variable and a list of example terms that might be considered but that do not represent the variable.Please see Table 1.We then established details of the protocol.These included in what time period the profile should be evaluated.For example, many profile owners have had a Facebook profile for several years; investigators using the codebook would need to know how far back in time to conduct a profile evaluation.The codebook protocol also included instructions regarding which comments on the SNS profile should be evaluated (comments from the profile owner only versus comments from other users that have posted on that profile owner's wall).
The codebook data collection sheets were then outlined.The data collection cells were designed so that sections of the profiles (i.e.information section, profile wall) would be evaluated in the same order each time a participant's profile was coded.Variables in the codebook were presented in the codebook in the same prescribed order that the coder follows during the profile evaluation process.

Codebook Layout
The codebook was initially designed in Microsoft Excel ® .Excel data sheets allowed separation of layers of the codebook onto several separate excel sheets within one excel file.The layers included the following: 1) the codebook protocol and coding criteria, 2) the data collection sheet with variables on the horizontal axis, allowing for participant data to be entered across each individual row, 3) the form for coders to track questionable references to discuss with the team, as well as document the outcome of the team's discussion regarding those references.Table 2 includes an example section of a data collection sheet including demographic information and references to depressive symptoms.

Coding Procedure
Coders follow several steps with each profile that is evaluated.First, the "information section" of the profile is viewed to obtain demographic information.Second, to evaluate depression symptom disclosures, investigators review each profile's status updates from the predetermined start date up through the current date of evaluation.Each status update is evaluated.For each status update that includes a depression symptom reference, coders assess whether the displayed symptom meets one of the DSM-IV criteria for a MDE [4].If the displayed symptom meets criteria, the coder recorded the date of the disclosure, verbatim text and the depressive symptom in which the verbatim text was correlated.The next status update is then evaluated.

Adaptation of the Codebook
The initial depression codebook was designed to be applied to Facebook profiles.Over time, our research interests have included evaluation of Twitter pages given the rise in popularity of this SNS [30].For each adaptation of the codebook, the same iterative process is applied including application of relevant clinical criteria, development of an initial set of keywords to apply to profiles, evaluation of a practice dataset, and discussion with the team to further refine the codebook.We continue to routinely communicate with our IRB and research ethics colleagues to maintain the highest standards of integrity in our data collection and reporting approaches.Further, we have examined applying our codebook to other data storage systems such as Filemaker © .We have found that using software like this is particularly helpful when recording longitudinal data or larger sample sizes.

Coder Training
The training of coders who will use the codebook follows a similar process that was used in codebook development.Coders are first trained in the ethics of human subjects research, and provided additional training in the importance of confidential procedures in SNS and mental health research.Second, coders learn the theoretical foundation and diagnostic criteria that underlie the variables of interest.Coders are then trained to use the codebook through several hands-on sessions in which coders practice evaluation of profiles and active participation in discussion is encouraged.After these sessions, coders begin applying the codebook to practice sets of profiles.Coding is first done with a mentor to provide real-time feedback, and then practiced alone with a review of coding afterwards with that same mentor.Coders are encouraged to take their time to fully understand the process of profile evaluation and the theoretical basis underlying the codebook.Coding training can last up to 3 months before coders participate in a formal research project.
Interrater reliability is a critical parameter for the credibility of the codebook.Our procedure includes selec- Cohen's Kappa statistic is used to evaluate the extent to which there was overall agreement in the coding of the presence or absence of depression symptom references on a profile.In our previous work, Cohen's kappa was 0.79 for depression symptom references [5].

Special Concerns
In the coding of health behaviors on SNSs, there may be instances when concerning displays are identified which suggest that the profile owner's safety is at risk.It is critical to determine your research team's approach to how such concerning disclosures will be handled, and to train coders so they are prepared to deal with worrisome situations that may become apparent when coding.In coding depression, suicidal ideation or threat was a major concern for our group.In response to this concern, we developed a suicide protocol to manage any suicidal references that may arise.In developing a response protocol for any specific area of mental health, it is advisable to first identify what major areas of concern may be encountered while coding this behavior.Examples may include suicidal ideation, or disclosures of abuse or neglect.Second, the times in which it may be crucial to intervene if such disclosures are noted must be designated.This is important because a worrisome disclosure may be time stamped anywhere within the last week to one year ago.Third, a plan re-garding how a coder will respond if these issues are present must be developed.This intervention may be in the form of a phone call, email, or text message sent by the coder, the principal investigator, or another designated person.The coder may be asked to provide mental health resources, or alert appropriate authorities.Last, this plan must be readily available to all research team members.A brief outline of an example protocol is available as Figure 1.

Codebook Successes and Challenges
Our codebooks have undergone considerable adaptation over the past five years.The iterative process has contributed to adapting codebooks to account for new variables of interest as well as new trends and structural changes among SNSs.However, this ongoing process of codebook development has required time, patience and team commitment.Development and maintenance of a codebook leads to challenges inherent in team research: the need for coordination of schedules and tasks to accommodate frequent meetings, the time required to reach group consensus.The time required in training a new team member as a coder is substantial, when a team member needs to periodically or permanently relinquish their involvement in projects this can impact the overall productivity of the team.
Another challenge has been adapting to changes within SNSs.One example was changes in Facebook layout in 2011.During an ongoing study, we found that the layout of Facebook changed to a "Timeline" format and additional types of information were added, such as a "cover photograph."This led to modifications to our codebook determine how best to consider newly available data in our assessments.Further, upgrades to SNSs can lead to server outages and changes in how search engines function within the website, which may impact access to Facebook as well as challenges in locating particular profiles.
A final challenge has been adaptation to the social and cultural norms that are found in a particular study population.These norms contribute to behavioral expectations defined with a group and are often present on SNS.For example, a challenge for coders may be to interpret references that may be meant as sarcasm in a certain social group, or may have an unclear context from an outsider's perspective.Providing a systematic way to address these questions by discussions and group consensus is a key strategy, but requires time and scheduling.Additionally cultural norms about mental health, for example, may affect a particular culture's display of depression symptom references on SNS.Certain cultures may be impacted by social stigma in that culture against displaying any signs of mental illness.This may be easier to account for if the investigators conducting the coding are from the same social or cultural demographic of the owners of the profiles, however this is not always be possible.It is important to be able to identify these social and cultural norms and consider how they may affect the displays being evaluated.Our team has found it particularly beneficial to have input from people in the demographic being studied as contributors to codebooks and partners in interpretation of data.

Future Codebook Developments
Despite these challenges, our codebook development process has been both rewarding and beneficial.We will continue to revise and adapt our codebook as new research questions arise and new trends in social media use emerge.
This paper has focused on development of depression coding criteria based on clinical standards.This standardized approach can be applied to the development of codebooks for other areas of mental health towards achieving a consistent and reliable assessment tool.An example of the application of this codebook development method includes developing a codebook to evaluate cognitive vulnerabilities, previously applied to diary entries [31].The cognitive vulnerability coding approach or content analysis of verbatim explanations (CAVE) was developed in 1984 by Seligman and colleagues [32].The coding approach assesses cognitive vulnerabilities which may be a precursor for depression symptoms [33].In this coding approach, each text disclosure is examined for both an event and attribution, which are then categorized as internal versus external, global versus specific, and stable versus unstable.The text disclosures describing a more internal, global, and stable state are considered more indicative of depression.After applying the steps outlined in this paper, our team recently pilot-tested this coding approach on status updates on Facebook and tweets on Twitter.We encountered challenges in identifying status updates and tweets that included both an event and an attribution due to the text references being short in length and lacking detail, but we were able to apply this coding scheme to a small percentage of status updates and tweets on both sites.This example illustrates how other content analyses strategies can be adapted to SNS codebooks following the steps outlined in this paper.

Future Research Approaches
We believe that SNSs may provide new opportunities to increase student help-seeking behavior.It is possible that such screening could be triggered by the content of a SNS profile.When Facebook users view their profile, advertisements triggered by keywords present on the profile are displayed at the side of the profile.It is possible that university counseling centers could link messages about counseling services or links to online screening to keywords such as "depressed" or "hopeless".There are reasons to believe that online counseling may be an acceptable approach in this digitally savvy generation, a recent study evaluated an interactive web-based program designed to screen students for depression and suicide risk.After the initial online screening, 24% of students entered into an online dialogue with a counselor, 19% later attended an in-person session with the counselor and 14% entered a treatment program [34].SNSs may provide an innovative venue to provide access to online screening and follow-up resources.
Given the difficulty in identifying students at risk for depression, and the potential negative consequences of untreated depression, is it imperative to embrace innovative public health opportunities to reduce the burden of mental illness in this population.However, any SNS intervention ideas hinge on designing programs that are acceptable to students and respect their privacy and confidentiality [35,36].Mental health disclosures are personal and potentially stigmatizing, thus, proper attention to privacy will be essential.However, there are reasons to be optimistic that today's college students may welcome online mental health programs.A study assessing a web-based intervention program found that among students with an un-met need for mental health care, over 90% reported interest in or intention to use the program [37].
In conclusion, it is our hope that this paper will provide detailed information about one systematic approach to mental health codebook development so that other researchers may use this structure to develop and test their own codebooks for use in mental health SNS research.It is not our intent to present this paper as the only way to develop or use a codebook; rather, we hope to assist other researchers in adapting this process to fit their own areas of interest.

Table 1 . Examples of mental health references. DSM-IV Criteria for Depression Examples of Key Words/Phrases Don't Include: Depressed mood Sad
, empty, crying, tearful, alone, lonely, sad face emoticon "I had a bad day", "FML"IndecisivenessCan't decide on something, don't feel like deciding, can't make up mindRecurrent thoughts of death or suicidal ideationThinking of ways to commit suicide, references to jumpingDifficulty concentratingCan't study, can't finish work, can't concentrate Don't want to concentrate, can't concentrate because of activity (TV, friends, Facebook)