Improving STEM Education in Research: Preliminary Report on the Development of a Computer-Assisted Student-Mentor Research Community

Recommended Citation Rios, David A.; Chetbotko, Artem; Reilly, Christine; Carlson, Ralph; Tomai, Emmett; Weimer, Amy; Weimer, Nicholas; Pearson, Thomas; Andoh-Baidoo, Francis; Winkle, Robert; Ammons, David; and Rampersad, Joanne, "Improving STEM Education in Research: Preliminary Report on the Development of a ComputerAssisted Student-Mentor Research Community" (2012). Computer Science Faculty Publications and Presentations. 19. https://scholarworks.utrgv.edu/cs_fac/19


Introduction
Providing students with research experiences in the disciplines of Science, Technology, Engineering and Mathematics (STEM) is a nationally recognized objective (STEM Education Coalition; US Department of Education; National Science Board). However, despite considerable expenditures of private and public funds to provide research experiences (The National Conferences on Undergraduate Research; National listing of REU Programs), it is surprising that success is still being measured by the number of students participating, money spent, and largely anecdotal evidence of whether the research experience was enjoyable (see e.g., Crowe et al., 2008). To address this problem, some programs offering research experiences have turned to the sci-entific assessment of students' experiences/activities (Russell, 2006; Surveys of undergraduate experiences). Although laudable, such assessments are of limited utility because they cannot presently be linked to specific research activities, making it difficult to correlate outcome with students' actual activities. Equally undesirable is that accountability is diminished when a student's research activities are not known, since it cannot be demonstrated whether students participated in true research (formulated hypothesis, designed experiments, etc.), nor whether they were actually "mentored" by their mentor. To ensure both accountability and the quality of research experiences, a detailed understanding of student activities and interactions with their mentors must be both known and associated with student assessment. However, it simply is not feasible to expect students and mentors to make note of every meeting, updated research goal, time spent in the laboratory, fellow students that they mentored, presentations given, etc.-it is simply too cumbersome, and history has shown that this approach does not work. Unfortunately, many of these problems are not adequately addressed by different, existing mentoring software (see for example, Chronus, Icouldbe and iMentor).
A possible solution to the problem is found in a community structure. The benefit of a structured community is that it provides tools and services that make activities easier, while automatically providing detailed information about a member's activities and relationships. For example, the act of using a credit card (a tool), intended or not, documents our personal preferences, interests, physical location, buying habits, psychological parameters (e.g., willingness to incur debt, and the level and type of debt we are willing to incur), etc. This raises the question of whether a student-mentor community, that provides tools facilitating student-mentor research activities, can automatically document the details of a student's research activities. If so, then large amounts of detailed information on student/mentor activities could be collected. The challenge is to design a community with computer-assisted tools that members find useful and want to use, but that are also capable of automatically collecting the desired information. Herein we provide a preliminary report on the development, testing and students' perception of prototype software that supports a community of student-mentor researchers.

Developing Prototype Community-Building Software
Prototype community-building software called Student Research Organizer (SRO) was developed to create a local student research community on the campus of the University of Texas-Pan American (UTPA). UTPA Institutional Review Board approval was obtained to develop and implement the prototype software. Informed consent from participants was obtained. SRO used a server-mounted MySQL database with a user interface built with Microsoft Access. SRO was designed from the bottom up (i.e., functionality was primarily added and perfected by satisfying needs and addressing comments from students and their Mentor, as opposed to building the community based on a preconceived design). SRO thus went through continuous growth, evaluation, and improvement cycles. To promote its widespread use, SRO development was guided by a user-oriented philosophy based on two key functional objectives: 1) To provide tools to facilitate activities that students and mentors are normally engaged in (i.e., to make their existing activities easier/more productive for them to perform); and 2) Not to burden users by requesting information that does not pertain to their activities (i.e., minimize the collection of information solely for use by education researchers or administrators). SRO was initially accessible to students only from computers in a single laboratory, which eventually was upgraded to any computer on the UTPA campus 24/7, including students' own personal computers.

Research Environment
All activities described in this report occurred at the UTPA, a predominantly Hispanic-serving institution located along the Texas border with Mexico that is recognized as a Predominately Undergraduate Institution (PUI). A large portion of the university's students come from low income families with almost 80% of the students receiving need-based grants or scholarships. UTPA receives substantial funding from both public and private organizations to facilitate the participation of students in STEMrelated research. SRO was used in both a guided/open inquiry research teaching laboratory (defined according to Buck et al., 2008; Advanced Biochemistry Laboratory 3 rd year course, Spring 2009Spring , 2010Spring , 2011, and continually from 2008-present with students participating in research in a Faculty laboratory. In both cases the mentor was study participant Dr. Joanne Rampersad.

Student Perceptions of SRO Subjects
Forty-four subjects responded to 15 items on the Student Interest in Using-SRO scale (SIU-SRO) after using SRO in a science laboratory. Forty one subjects, consisting of two groups, twenty-one males (n = 21), and twenty females (n = 20) completed the (SIU-SRO) scale.

Measures
A scale for measuring SIU-SRO was administered to subjects after utilizing SRO. The 15 items on the SIU-SRO used an equal appearing eight point Likert Scale.
Exploratory factor analysis was used to derive underlying dimensions that the SIU-SRO was measuring. Varimax rotations with an eigenvalue set at 1.0 was used to derive the underlying dimensions. Two underlying domains were obtained with the first factor explaining 54.36% of the SIU-SRO variance, and the second factor explaining 23.60% of the SIU-SRO variance. A total of 77.97% of the total SIU-SRO variance was explained by these two factors. The first factor was measuring a positive attitude toward using SRO, and the second factor was measuring a negative attitude toward using SRO.
Raw scores from the two factors were transformed into two scales, Positive Interest in Using-SRO (PIU-SRO) and Negative Interest in Using-SRO (NIU-SRO) through a linear transformation and thus deriving a common metric for the two scales.
Ten items loaded on the first factor, PIU-SRO. These loadings ranged from .68 to .94. Four items loaded on the second factor, NIU-SRO. These loadings ranged from .67 to .97. Item number 14 was deleted because of its cross loading of .60 and .64 on factors I and II, respectively. The Cronbach's alpha reliability coefficients for the PIU-SRO and the NIU-SRO were .97 and .89, respectively. Given that only 41 subjects were used in deriving the psychometric properties of the two scales, caution should be maintained in interpretation of factor structure and pattern. A simple structure was obtained, however, and the Cronbach's alpha reliability coefficients of .97 and .89 for the two scales indicate that these underlying dimensions can be assumed to represent the phenomena of a PIU-SRO and NIU-SRO. In addition, the discrimination indices for items on the PIU-SRO ranged between .74 and .96, and items on the NIU-SRO ranged between .77 and .93.

Creating a Database Schema for a National Web-Based Community
Efficient and scalable data management is an important requirement for the SRO system. Once the prototype SRO software had been built, developed and tested, a better understanding of what data should be collected and stored in the SRO database emerged. The design of the database for the web-based community was then carried out using a three-step methodology commonly employed for designing relational databases. First, a conceptual data model was designed using the entity-relationship modeling methodology (Chen, 1976), as shown in Online Resource 1. Second, the entity-relationship model was translated to a relational data model (Codd, 1970). Finally, the relational data model was used to obtain a database schema with physical tables, data integrity constraints, indices and triggers. The database schema was expressed using Structured Query Language and instantiated in MySQL (MySQL). dents nowadays gravitate toward social community environments (e.g., the success of community-oriented resources such as Facebook, and Twitter attest to this); 2) A community structure allows students to see, communicate and interact more freely with other students and their projects, thus enriching the student's research experience; 3) An organized, structured community of student researchers and their mentors facilitates educators in conducting educational research, quickly/broadly implementing new educational methods, and allows assessment results to be linked with actual activities; 4) A student-mentor research community can provide needed accountability of the

Results and Discussion
The Advantages of Choosing a Community Structure educational process by documenting activities in context to specific relationships; and 5) A student-mentor community can be used to help teach students ethics and behavioral norms that Intuitively there are many reasons to look to a communitybased structure to improve student research education: 1) Stu-Online Resource 1. Entity-relationship diagram representing the conceptual data model of the SRO system. will facilitate the development of their professional identities. Technology can play an important role in managing educational activities, such as a community of researchers (Sharaf & Musawi, 2011).

Structural Design of the SRO Student-Mentor Community and Its Approach to Training Students in the Responsible Conduct of Research
SRO establishes a community structure based on the concept of a population of student researchers that engage in activities with mentors. The SRO model allows an individual to become a community member only once, but a member can have a limitless number of "roles", where each role can establish multiple relationships with one or more other roles in the community. Student roles are based on the university or institution they belong to, while mentor roles are defined by the university department, or non-university institution (e.g., name of high school), they belong to. Once a relationship is formed between a student and mentor, all activities that arise from the relationship are automatically documented and attributed to the two roles that define the relationship. Thus a member of the SRO community will always have just 1 username/password to enter the community, at which point they will choose which specific role and associated relationship they want to assume. Figure 1 presents a screen shot of a supervisor's account on the prototype SRO software, highlighting a student's "Projects and Goals" page, and the user interface design.
SRO's structural design also addresses the common privacy issues that arise in a community setting, as well as providing a framework to help train students in the Responsible Conduct of Research (RCR). SRO was premised on character-building models, as opposed to rule-governed models of ethics education, which are inadequate in several respects. Rules are invariably formulated in the most generic terms to be applicable to a variety of cases; but that very generality renders their application to specific situations uncertain. The complexity of many ethically congested cases in research makes it difficult to determine reliably when a situation properly fits under a particular rule. In addition, "the rules run out," meaning that new circumstances give rise to novel cases that have not previously been addressed by any rule. This is notably common in scientific research, where new techniques and discoveries often provoke unexpected situations that demand a moral judgment from the community of researchers, but for which no articulated rule has yet emerged. Finally, rule-governed models in ethics tend to treat moral decision-making as an exercise in problem-solving, as if moral issues were a kind of puzzle in search of a satisfying resolution. This portrays ethical concerns in an atomistic fashion, as discrete dilemmas, and emphasizes the perfection of abstract techniques of calculation as the proper method for seeking a solution to these dilemmas. Not only does this produce a failure to recognize the inherent connections between many different kinds of ethical situations, but it removes the human element from moral deliberation: on a rule-governed model, a computer could as easily do the calculations leading to a good moral decision as Figure 1. A screenshot, from a supervisor's account, of the prototype SRO's main user interface window opened to the "Projects and Goals" tool. could any human being-and perhaps with even greater efficiency and accuracy.
By contrast, character-building models do not focus on rulegoverned strategies for moral decision-making, but rather on crafting the conditions under which individuals can develop a self-understanding informed by virtue that frames their conscious identity as researchers. Instead of asking the question, "What decision should I make?" as rule-governed systems do, character-building models ask the question, "What kind of person do I want to become as a researcher, such that I can make better decisions?" These latter models emphasize the intentional maturing of moral expertise through immersion in the practices of scientific research, learning to appreciate and absorb the standards of ethical excellence embodied in the activities of research, and responding to the example and counsel of experienced mentors within the community of scientific researchers. While affirming the usefulness of rule-governed models for exercises in moral decision-making, we argue that the nurturing of moral character is a necessary preparation for such decision-making, and thus is an endeavor that properly comes before making moral decisions. In short, a person already possessed of a good moral character is more likely to make appropriate ethical decisions. We believe the character-building approach promises to enhance the moral acuity of student researchers as they engage the issues related to RCR.

The Advantages of Joining the SRO Research Community
The success of any community depends on what the community offers its members. If joining the SRO community did not represent a clear benefit to the mentor or student, they would either not join or not fully participate. It is for these reasons that SRO was built on the philosophy of providing tools and activities that members want, and benefit from (and thus be most likely to use), and where information important to students, mentors, educators and administrators can be collected indirectly and automatically as individuals use the community's tools to facilitate their daily research-related work and activities.
To ensure the inclusion of tools and functions that the user would find useful, the community prototype (SRO) was built from the bottom-up, starting out as nothing more than a platform to assign a student a research project, and set/update the student's research goals over time. From this very fundamental student-mentor interaction, functions were added in response to suggestions made by students and the mentor. For example, the mentor in this study had approximately 15 undergraduate research students working in her laboratory who would randomly pop into her office to discuss their work. The resulting chaos created severe problems for the Mentor. In response, a community tool was added that allowed students to request a meeting, which, if approved by the mentor, was added to the Mentor's meeting schedule, and made visible to all students. Similarly, students found that they needed to contact other students in their research group or laboratory, but keeping updated lists of student emails was a problem. From this need, a community mail system was added that allowed students to send mail based on community criteria, such as students participating on a particular research project. Developed in response to a need expressed by either a student or their mentor, some of the other principle user functions included in the prototype SRO community were: defining/assigning research projects to students, updating research goals, organizing student-to-student skill training, managing and reserving research equipment, managing drafts and deadlines for posters and presentations, organizing/documenting students' laboratory safety training, organizing collaborators and student scholarships, an advisor window to automatically alert students and mentors to any community activity that requires their attention, criteria-driven reports for obtaining information such as a list of students possessing a particular laboratory skill, and deploying student assessments and collecting responses.

Analysis of Results
The Positive Interest in Using-SRO (PIU-SRO) scale and the Negative Interest in Using-SRO (NIU-SRO) scale were administered to 41 students who used SRO in the research and teaching labs. There were 21 male and 20 female students included in the study. Obtained descriptive statistics are shown in Table  1 below.
A two-way factorial ANOVA (2 × 2) with one between subjects factor, gender, and one within subjects factor, scales/trials, was used to analyze obtained data (see Table 2).

Summary and Interpretation of Results
There was a difference between PIU-SRO, mean of 4.38 and NIU-SRO, and mean of 2.02, F = 186.9 (1, 39), p < .05. The effect size for this obtained difference is assessed through a partial eta squared of .83 (see Table 2) and Cohen's d value of 1.99 or approximately two unit size difference between PIU-SRO and NIU-SRO in favor of a PIU-SRO. There is no difference between means for males and females (see Table 2). There is a difference between the average PIU-SRO compared to the average NIU-SRO. From this we can conclude that after using SRO, students have significantly greater interest/motivation in using SRO than a disinterest. Perhaps the biggest surprise encountered while working with the community prototype was the student's unexpected interest in documenting their effort. Students participating in research experiences will commonly work long hours and, at times, engage in activities that do not directly pertain to their research, such as instructing other students on performing laboratory techniques. Normally, all this work and effort is not reflected in the student's research output, nor recognized by their mentor and others. Unexpectedly, we found that students appreciated that by conducting their research activities within the SRO community, these efforts were both documented and made available to their mentor. For example, a "Time Card" was added for students to clock in and out of the laboratory. Although useful to the mentor in managing the laboratory and monitoring student effort, it was initially feared that students would find this tool a burden, intrusive or even accusatory-surprisingly, the opposite was true. Students were actually very upset when, due to a downed server, they could not record their time spent working in the laboratory. Students would even contact the Community's Administrator and demand that they be credited unrecorded time. The student's high level of interest in documenting their effort has influenced greatly the design of student-oriented tools to document student effort. Based on this finding, functions that give students a community rating are also being developed. For example, a student that is in the top 5% for helping other students with skill learning will be recognized on the community (i.e., a special community title, etc.)

Creating a Data Management Model for a National Web-Based Community
From the information technology and computer science perspective, SRO is a complex computer-based information system that involves human and computational resources to gather, process, analyze, and preserve data. The importance of a welldefined data model for SRO cannot be underestimated. Through the course of several years, as the prototype software was developed and tested in the production environment, an ad-hoc approach to data modeling and database design "on-demand" was in use. The experience and insights gained from this process enabled us to take SRO's data model to the next level using a three-step methodology commonly employed for designing relational databases.
The conceptual data model serves as the first and most complex step in the database design process. Data collection and management in SRO relies on the extensible conceptual data model designed using the entity-relationship methodology (Chen 1976), the diagram of which is presented in Online Resource 1. Entity types (rectangles) and relationship types (diamonds) in the diagram are organized into 17 modules (shaded boxes) that support different functional requirements of the SRO system, including the recording of information about community members, their roles, education, employment, research activities, projects, certifications, scholarships, grants, assessments, presentations, and so forth. As the system evolves, new modules can be added or existing ones can be extended to address new requirements.
The design of the relational model (Codd, 1970) is the next step in the design process. Using standard procedures, the SRO entity-relationship model is translated into a relational model (also referred to as a logical data model) resulting in approximately 100 relations with various integrity constraints (not shown in this work). These integrity constraints ensure that the data stored in the database will always be in a consistent state that reflects the real community.
The third step in the design process is creation of a physical data model based on the relational data model obtained in the previous step. The physical data model is represented by a set of statements written in Structured Query Language, which can be executed to create a database schema in a Relational Database Management System (RDBMS), such as MySQL (MySQL). In addition to the tables that store the data, the resulting database schema includes a number of indices that can support efficient querying of the database and multiple triggers, which are automatic procedures that maintain data integrity.
We expect that as the SRO community grows, both mentors and students will need SRO to provide the functionality necessary to track collaboration data not contemplated in our current design. Perhaps the greatest benefit achieved through our database design process was the creation of an open-ended model that supports both the growth of the community and the future addition of new community roles and student activities.
We are currently in the process of building a web-interface to interact with the database that replicates the main functions, tools and architecture developed in the SRO prototype, along with new capabilities. This web-interface will allow community members to easily interact with SRO without requiring them to have knowledge of the underlying data model.

New SRO Capabilities
While developing the SRO prototype, due to the high level of effort/resources required, it was not feasible to develop the temporary prototype software to support and test three community needs. However, these needs are being addressed in the national web-based community currently under development, and are discussed below.

"Kids" Are Also Members of the Community
The SRO prototype community was primarily designed for undergraduate and graduate students and their mentors. However, there is a very large body of younger students that participate in research via this nation's Science Fair program. Although the student-mentor structure of the SRO research community would work well for student researchers participating in the Science Fair, the community could be improved by adding tools specifically for their unique needs. Unlike university students, science fair students are much less experienced and rarely have a dedicated full-time mentor to assist them. We are therefore interested in taking advantage of the SRO community structure by facilitating student-student mentorship, where college students can assume the community role of a mentor for Science Fair students, thereby promoting the participation of universities in the national Science Fair system. We would also like to provide Science Fair students with special tools that will help them develop their projects in compliance with the Scientific Method. Unquestionably, helping to motivate and educate these younger researchers is an exciting challenge in which the SRO community can play a significant role.

Communities Benefit from Experienced Elders
As a student begins her/his research, perhaps through participation in the Science Fair, followed by undergraduate and post graduate studies, the student gains a wealth of experience, not only in performing research but in knowing what it is like as a student to face the challenges of research. Unfortunately, unless the student decides to stay in both teaching and research, all this experience and knowledge will be lost from the community. Individuals who have gone on to positions that do not include mentoring should be enticed to stay active within the community and act as mentors to science fair students, or perhaps, through a forum, provide advice to students facing the same challenges they did. Retaining the experience of community members is important to the success of the research community, and is being actively pursued.

Good Researchers Do Not Always Make Good Mentors
It is difficult at times for a mentor to realize that a student is having trouble, and even more difficult to know what to do to address the problem. This is especially true for mentors with relatively large numbers of students. The SRO student-mentor community would therefore benefit from tools that automatically identify problems with students and provide suggestions on how to deal with the problem. The SRO community structure, and the large amount of detailed information it collects, is ideally suited for the task. The large and diverse amount of data on time spent in the laboratory, assessment results, student and mentor evaluations of the student's performance during each goal cycle, etc. can all be used to identify potential problems with a student. For example, SRO might, based on certain indicators, notice that a student is experiencing a possible loss in self-confidence. SRO could then automatically alert the mentor to the potential problem, display a summary of the indicators that reflect the problem, and then offer the mentor suggestions, such as reducing the number and complexity of the research goals/shortening the goal cycle to help boost the student's laboratory success and confidence.

Conclusion
Prototype software was developed that successfully support undergraduate student-mentor activities through a community structure. By using community tools to facilitate their work, details of the students' activities were automatically gathered in a relational database without negatively burdening the students. In fact, after using SRO, students showed a significantly greater interest/motivation in using SRO than a disinterest. Student assessments were easily deployed via the SRO software and the results associated with the student's research activities, thus allowing a direct correlation between assessment responses and the activities being assessed. An Entity Relationship model based on the prototype software was created, from which a database schema was obtained for the establishment of a larger nationwide student-mentor research community. Three community needs not incorporated in this study's prototype software were identified for inclusion in the nation-wide community.