The Relationship between Presence and Performance in Virtual Simulation Training

Abstract

The use of virtual simulation-based training continues to expand, as organizations explore alternative methods to reduce the cost of training. While virtual simulation has been empirically validated to be effective in the transfer of skills to the live environment, what is still unknown is what effect, if any, that an individual’s sense of presence in the simulation has on their performance. In this paper, we examine the relationship between presence and performance while performing a psychomotor task in a virtual environment. The independent variables were visual display and expertise level. The dependent variables were presence and performance. We found evidence of a moderate relationship between the degree of presence experienced in the simulation and an individual’s performance over three training trials. Results of this study may inspire future research to investigate whether higher presence in virtual simulation results in higher performance.

Share and Cite:

Stevens, J. and Kincaid, J. (2015) The Relationship between Presence and Performance in Virtual Simulation Training. Open Journal of Modelling and Simulation, 3, 41-48. doi: 10.4236/ojmsi.2015.32005.

1. Introduction

The goal of this paper was to examine whether higher presence in virtual simulation training was associated with higher performance by the trainee. The goal of virtual simulation-based training is to optimize the degree of transfer to the trainee, extrapolating that superior performance in the simulated environment translates to superior performance in the live environment. The determination that a relationship exists between presence and performance would be significant, even more so if one’s sense of presence could be manipulated by the simulation designer. In this manner, simulation practitioners could potentially maximize the trainee’s sense of presence in order to maximize their performance.

As human performance optimization is the goal of simulation-based training, this research is an important undertaking as organizations continue to augment, and potentially replace their live training with virtual training due to cost, time and effort constraints. In order to optimize the degree of transfer to the trainee, simulation designers must identify what variables in the simulation can be manipulated in support of that goal. Examining the literature, we identified presence as a potential variable that could directly influence the performance outcome. In this empirical study, we examined whether or not presence was correlated with performance in a virtual training simulation. If so, a subsequent study will examine best practices to maximize presence in virtual simulation in order to optimize human performance. This article extends our previous work on this topic [1] [2] .

2. Background

2.1. Fidelity, Immersion & Presence

A training simulator’s fidelity, or degree of realism, is commonly thought of as being inherently associated with that simulator’s training effectiveness. Multiple studies [3] - [7] have now partially refuted this notion, with the literature indicating that only a sufficient level of fidelity is required for effective training. Conversely, what is known is that simulator cost is positively associated with the simulator’s degree of fidelity. Thus, the more realistic the simulator is, the higher the cost to conduct training will be. As organizations’ training budgets decline, justification of simulation-based training procurement and design decisions will need to be made based on empirical evidence and not anecdotal belief.

Physical immersion is a function of the simulator’s technology [8] - [10] . For virtual simulation, immersive systems are generally characterized by the use of head-tracking technology coupled with the minimization of real world stimuli [11] . Similar to fidelity, the degree of physical immersion is an active literature topic as simulation developers are increasingly forced to justify their design decisions. The highest immersive virtual simulation may not be required for effective training.

Presence is defined as “the subjective experience of being in one place or environment, even when one is physically situated in another” [12] and is sometimes referred to as mental immersion [13] . Presence can also be thought of as a user’s sense of “being there” in the simulation [14] . The literature is replete with studies that not only attempt to categorize factors of presence but also highlight simulator attributes that influence an individual’s sense of presence [12] [15] [16] . The Witmer and Singer Presence Questionnaire (PQ) is the most common instrument to quantitatively measure an individual’s degree of presence.

2.2. Training Transfer

The objective of virtual simulation-based training is to optimize the degree of transfer to the trainee, where transfer is defined as the application of knowledge and skills learned in training and subsequently applied to the real environment [17] . Ideally, transfer will be positive, where the trainee performs his real world task better after simulator exposure. However, transfer can also be negative, which occurs when performance degrades in the real environment, usually as a result of poor simulation design or a mismatch between training curriculum and the correct simulator. Successful training simulations are ones in which the degree of transfer experienced by trainees is positive and high [18] .

The measurement of transfer in simulation-based training is rare because it is difficult. This assertion applies to medical simulation [19] , military simulation [20] and other simulation-centric domains [21] [22] . However, the literature does offer numerous examples that empirically prove the training effectiveness of virtual simulation based training [23] - [25] . One of the primary purposes of simulation-based training is to conduct training on tasks that are either too costly or too dangerous to train in the live environment. In these instantiations, the measurement of transfer is impractical and therefore the transfer condition is the simulator itself. This is true in the case of this study.

2.3. The Relationship between Presence and Performance

The literature is inconclusive whether or not a clear relationship exists between presence and training transfer. Anecdotally, higher presence is associated with higher transfer and some studies have empirically demonstrated a positive association between presence and transfer [26] [27] . However, other empirical studies have shown that this may not actually be the case [28] - [31] .

Conversely, while less intuitive, the literature clearly establishes the independence between both fidelity and immersion and presence. Since fidelity and immersion are determined in the simulation design phase, they are generally treated as independent variables in the aforementioned studies. The literature is rich with empirical studies that achieve a high level of presence with corresponding lower levels of fidelity and immersion [32] - [34] and vice versa.

Since presence has been shown to be independent of both a simulator’s fidelity and physical immersion, the purpose of this study was to examine if an individual’s sense of presence, treated as an independent variable, was associated with their individual performance. If we treat presence as an independent variable (as a function of the individual’s response to the simulator experience) then we can begin to examine if a relationship exists between the two. If such a relationship can be found, then the manipulation of presence in virtual simulation- based training could potentially yield higher performance results. Higher performance, in turn, translates to positive training transfer, the goal of simulation-based training. While the relationship between presence and performance is still not well understood, we attempt to contribute to the subject’s body of knowledge with the intent to improve human performance in training simulation.

3. Method

3.1. Experiment Objective

The objective of this study was to determine whether or not a relationship exists between presence and performance in virtual simulation-based training. If such a relationship could be established, then simulation designers could potentially architect higher presence into their simulation design, with the eventual goal of improving human performance. This would represent a significant finding, as many tasks trained in simulation are too dangerous or costly to train in the real environment and thus simulation itself may serve as the transfer condition. In essence, we could potentially improve human performance in critical tasks by manipulating the degree of presence experienced in the simulation.

3.2. Participants

96 United States Army soldiers participated in this experiment (n = 96, M = 28.9 years of age, SD = 8.5 years). All soldiers were qualified on the M240 Machine Gun and were drawn from multiple, disparate units located throughout the United States. This sample was 100% male.

3.3. Experimental Procedure

This experiment utilized the U.S. Army’s Non-rated Crew Member Manned Module (NCM3) simulator as its apparatus. The NCM3 is a mobile virtual simulation training device designed to train soldiers on aerial door gunnery, a highly dangerous and expensive training task. Thus it represented the transfer condition, since the acquisition of live aircraft and ammunition to serve as the transfer condition were outside the scope of this study.

This was a between-subjects design of experiment, with participants performing aerial door gunnery in one of two treatments: helmet mounted display or flat screen display. All soldiers utilized a real M240 machine gun with simulated recoil, donned a simulated flight helmet and engaged synthetic enemy forces through an emulated aperture. The simulation was operated by an instructor/operator who centrally issued commands to four simultaneous participants.

Participants were screened for prior machine gun experience (a prerequisite) and were further classified into one of two expertise groups: expert or novice via a demographic survey. All participants executed one familiarization scenario and three formal scenarios with the goal of destroying as many synthetic enemy forces as possible. The task was performed while in simulated flight using the simulator’s machine gun. All scenarios contained fifty synthetic enemy personnel that participants could potentially destroy. Subsequent scenarios increased in difficulty so as to match the predicted gain in performance (simulated with a subject matter expert) so as to normalize performance results. Performance results were automated and captured at the Instructor/Operator station, with the individual’s average score over the three trials representing their numerical performance outcome. Presence was measured immediately after simulator exposure using the Witmer and Singer [12] Presence Questionnaire (PQ). The PQ is a post-hoc, 24 question (7 point scale) self-reporting checklist that measures participants’ degree of presence experienced in the simulator.

4. Results

4.1. Performance and Presence Results

Performance was averaged over three trials by individual. Mean performance results are depicted in Table 1. A two-way ANOVA found that visual treatment did have a significant main effect on performance, F (1, 95) = 4.20, p = 0.04. ANOVA found no significant main effect of expertise on performance, F (1, 95) = 1.62, p = 0.21. The interaction between visual treatment and expertise was not significant F (1, 95) = 0.02, p = 0.90. ANOVA was conducted at α = 0.05.

Presence was measured post-hoc using the Witmer and Singer Presence Questionnaire. Presence scores are depicted in Table 2. A two-way ANOVA found that visual treatment did have a significant main effect on presence, F (1, 95) = 8.06, p = 0.01. ANOVA found no significant main effect of expertise on presence, F (1, 95) = 0.06, p = 0.82. The interaction between visual treatment and expertise was not significant F (1, 95) = 1.18, p = 0.28. ANOVA was conducted at α = 0.05.

Synthesizing the performance and presence results, we determine that the HMD treatment resulted in both higher performance and higher presence experienced in the simulation. Post-hoc Student t-Tests employing a Bonferroni correction (α = 0.01) confirmed these results for both performance [t (91) = 2.62, p < 0.01] as well as presence [t (94) = 2.73, p < 0.01]. Expertise level did not affect either performance or presence. For both performance and presence, no significant interaction between visual treatment and expertise was found. Therefore, we conclude that visual display did have an effect on both performance and presence.

4.2. Performance and Presence Correlation

The primary purpose of this study was to investigate whether a relationship existed between an individual’s presence experienced in the simulator and their subsequent performance. Both performance and presence samples were confirmed to be normally distributed by Shapiro-Wilk test (total sample, p = 0.86 and p = 0.95 respectively) as well as through visual examination of both variables’ QQ plots. Normality was also confirmed for the additional four correlation tests. Performance and presence were found to be moderately correlated for the entire group, r (96) = 0.22, p = 0.03, the Novice population r (76) = 0.36, p = 0.002 and the HMD treatment, r (48) = 0.41, p = 0.004. The correlation magnitude descriptor is based upon guidelines found in [35] . No significant correlation was found for the Expert population nor the flat screen treatment. Correlation results are depicted in Table 3. Significant correlations are displayed in Figures 1-3.

5. Discussion

The primary purpose of this study was to examine whether a relationship existed between an individual’s presence

Table 1. Mean performance (treatment × expertise).

Table 2. Mean presence (treatment × expertise).

Table 3. Correlations between performance and presence.

Figure 1. Regression analysis (all participants: presence and performance).

experienced in virtual simulation and their performance in the simulation. This is a worthwhile objective as many tasks trained in simulation are too hazardous, costly and/or resource-intensive to train in the live environment; thus, the simulator also represents the transfer condition. Inferring, based upon our literature review, that performance improvement in the simulator improves the trainee’s live performance, the resultant outcome from simulator exposure is positive training transfer. Positive transfer is the goal of simulation-based training. Thus, presence may serve as a potential variable to be manipulated by the simulation expert so as to improve the performance of his trainees.

In this study, we found that visual treatment had a significant effect on soldiers’ performance and sense of presence, while performing a psychomotor task in virtual simulation. Specifically, higher immersive visual display resulted in both higher performance and presence. Expertise level had no significant effect on each of these variables and there was no significant interaction found between visual treatment and expertise.

We discovered a moderate, significant correlation between all trainees’ sense of presence and their performance, with the strongest correlation between the two variables found in the higher immersive visual display treatment. Significant correlation was also found for the novice population.

Figure 2. Regression analysis (novice participants: presence and performance).

Figure 3. Regression analysis (HMD condition participants: presence and performance).

The results of this study are noteworthy. First, higher visual immersion was found to induce both higher presence and higher performance on the part of the trainee. Since the goal of virtual simulation-based training is to optimize human performance, this finding is significant, especially so when simulator exposure also serves as the transfer mechanism. Second, we identified a simulator attribute (visual display) that clearly affected both performance and presence. This will undoubtedly assist with the identification of additional attributes we will calibrate in order to determine a subsequent effect on both performance and presence. Third, we were able to establish that a moderate relationship existed between trainees’ presence and performance, with the strongest relationship occurring in the treatment that most affected trainees’ performance and presence. This is notable and will influence future experimental design.

This study found an association between presence and performance, with the strongest association attributed to higher immersion in the simulator. Future research should be conducted with this apparatus in which the researchers attempt to manipulate additional simulator variables in order to increase or decrease the trainees’ sense of presence and examine the resultant effect on performance.

Conflicts of Interest

The authors declare no conflicts of interest.

References

[1] Stevens, J. and Kincaid, P. (2014) Measuring Visual Displays’ Effect on Novice Performance in Door Gunnery. Proceeding of the Interservice/Industry Training, Simulation, and Education Conference, Orlando, December 2014, 1-10.
[2] Stevens, J. and Kincaid, P. (2014) Measuring Visual Modalities Effect on Expert Performance on Mixed Reality Aerial Door Gunnery. MODSIM World Conference, Hampton, April 2014.
[3] Salas, E., Bowers, C. and Rhodenizer, L. (1998) It Is Not How Much You Have But How You Use It: Toward a Rational Use of Simulation to Support Aviation Training. The International Journal of Aviation Psychology, 8, 197-208.
http://dx.doi.org/10.1207/s15327108ijap0803_2
[4] Hays, R. and Singer, M. (1998) Simulation Fidelity in Training System Design: Bridging the Gap between Reality and Training. Springer-Verlag, New York.
[5] Feinstein, A. and Cannon, H. (2002) Constructs of Simulation Evaluation. Simulation & Gaming, 33, 425-440.
http://dx.doi.org/10.1177/1046878102238606
[6] Stewart, J., Johnson, D. and Howse, W. (2008) Fidelity Requirements for Army Aviation Training Devices: Issues and Answers. U.S. Army Research Institute, Arlington.
[7] Summers, J. (2012) Simulation-Based Military Training: An Engineering Approach to Better Addressing Competing Environmental, Fiscal, and Security Concerns. Journal of the Washington Academy of Sciences, 9-12.
[8] Slater, M. and Wilber, S. (1997) A Framework for Immersive Virtual Environments (FIVE): Speculations on the Role of Presence in Virtual Environments. Presence: Teleoperators and Virtual Environments, 6, 603-616.
[9] Ragan, E., Sowndararajan, A., Kopper, R. and Bowman, D. (2010) The Effects of Higher Levels of Immersion on Procedure Memorization Performance and Implications for Educational Virtual Environments. Presence, 19, 527-543.
http://dx.doi.org/10.1162/pres_a_00016
[10] Dalgarno, B. and Lee, M. (2010) What Are the Learning Affordances of 3-D Virtual Environments? British Journal of Educational Technology, 41, 10-32.
http://dx.doi.org/10.1111/j.1467-8535.2009.01038.x
[11] Bailenson, J., Yee, N., Blascovich, J., Beall, A., Lundblad, N. and Jin, M. (2008) The Use of Immersive Virtual Reality in the Learning Sciences: Digital Transformations of Teachers, Students and Social Context. Journal of the Learning Sciences, 17, 102-141.
http://dx.doi.org/10.1080/10508400701793141
[12] Witmer, B. and Singer, M. (1998) Measuring Presence in Virtual Environments: A Presence Questionnaire. Presence, 7, 225-240.
http://dx.doi.org/10.1162/105474698565686
[13] Nalbant, G. and Boston, B. (2006) Interaction in Virtual Reality. 4th International Symposium of Interactive Medical Design (ISIMD).
[14] Slater, M., Usoh, M. and Steed, A. (1994) Depth of Presence in Virtual Environments. Presence, 3, 130-144.
[15] Lackey, S. and Reinerman, L. (2013) Immersion: Concepts and Definitions. University of Central Florida, Orlando.
[16] Lombard, M. and Ditton, T. (1997) At the Heart of It All: The Concept of Presence. Journal of Computer-Mediated Communication, 3.
http://dx.doi.org/10.1111/j.1083-6101.1997.tb00072.x
[17] Farr, M. (1986) The Long-Term Retention of Knowledge and Skills: A Cognitive and Instructional Perspective. Institute for Defense Analyses, Alexandria.
[18] Alexander, A., Brunye, T., Sidman, J. and Weil, S. (2005) From Gaming to Training: A Review of Studies on Fidelity, Immersion, Presence, and Buy-In and Their Effects on Transfer in PC-Based Simulations and Games. DARWARS Training Impact Group, Woburn.
[19] Schout, B., Hendrikx, A., Scheele, F., Bemelmans, B.L.H. and Scherpbier, A. (2010) Validation and Implementation of Surgical Simulators: A Critical Review of Present, Past, and Future. Surgical Endoscopy, 24, 536-546.
http://dx.doi.org/10.1007/s00464-009-0634-9
[20] Insinna, V. (2013) Army, Marine Corps Look for Better Data on Simulator Effectiveness. National Defense, December 2013, 26-28.
[21] Sotomayor, T. and Proctor, M. (2009) Assessing Combat Medic Knowledge and Transfer Effects Resulting from Alternative Training Treatments. The Journal of Defense Modeling and Simulation: Applications, Methodology, Technology, 6, 121-134.
http://dx.doi.org/10.1177/1548512909350170
[22] Whitney, S., Tempby, P. and Stephens, A. (2014) A Review of the Effectiveness of Game-Based Training for Dismounted Soldiers. Journal of Defense Modeling and Simulation, 11, 319-328.
http://dx.doi.org/10.1177/1548512912472773
[23] Blow, C. (2012) Flight School in the Virtual Environment. United States Army Command and General Staff College, Fort Leavenworth, Kansas.
[24] Seymour, N., Gallagher, A., Roman, S., O’Brien, M., Bansal, V., Anderson, D. and Satava, R. (2002) Virtual Reality Training Improves Operating Room Performance: Results of a Randomized, Double-Blinded Study. Annals of Surgery, 236, 458-464.
http://dx.doi.org/10.1097/00000658-200210000-00008
[25] Hays, R., Jacobs, J., Prince, C. and Salas, E. (1992) Flight Simulator Training Effectiveness: A Meta-Analysis. Military Psychology, 4, 63-74.
http://dx.doi.org/10.1207/s15327876mp0402_1
[26] Winn, W., Windschitl, M., Fruland, R. and Lee, Y. (2002) When Does Immersion in a Virtual Environment Help Students Construct Understanding? Proceedings of the International Conference of the Learning Sciences, Mahwah, 2002, 497-503.
[27] Mikropoulos, T. (2006) Presence: A Unique Characteristic in Educational Virtual Environments. Virtual Reality, 10, 197-206.
http://dx.doi.org/10.1007/s10055-006-0039-1
[28] Mikropoulos, T. and Natsis, A. (2011) Educational Virtual Environments: A Ten-Year Review of Empirical Research (1999-2009). Computers & Education, 56, 769-780.
http://dx.doi.org/10.1016/j.compedu.2010.10.020
[29] Persky, S., Kaphinqst, K., McCall, C., Lachance, C., Beall, A. and Blascovich, J. (2009) Presence Relates to Distinct Outcomes in Two Virtual Environments Employing Different Learning Modalities. CyberPsychology & Behavior, 12, 263-268.
http://dx.doi.org/10.1089/cpb.2008.0262
[30] Slater, M. (1999) Measuring Presence: A Response to the Witmer and Singer Presence Questionnaire. Presence, 8, 560-565.
http://dx.doi.org/10.1162/105474699566477
[31] Selverian, M. and Sung, H.H. (2003) In Search of Presence: A Systematic Evaluation of Evolving VLEs. Presence: Teleoperators and Virtual Environments, 12, 512-522.
http://dx.doi.org/10.1162/105474603322761306
[32] Mikropoulos, T. and Strouboulis, V. (2004) Factors that Influence Presence in Educational Virtual Environments. CyberPsychology & Behavior, 7, 582-591.
http://dx.doi.org/10.1089/cpb.2004.7.582
[33] Dede, C. (2009) Immersive Interfaces for Engagement and Learning. Science, 323, 66-69.
http://dx.doi.org/10.1126/science.1167311
[34] Villani, D., Repetto, C. and Cipresso, P.R.G. (2012) May I Experience More Presence in Doing the Same Thing in Virtual Reality than in Reality? An Answer from a Simulated Job Interview. Interacting with Computers, 24, 265-272.
http://dx.doi.org/10.1016/j.intcom.2012.04.008
[35] Hemphill, J. (2003) Interpreting the Magnitudes of Correlation Coefficients. American Psychologist, 58, 78-79.
http://dx.doi.org/10.1037/0003-066X.58.1.78

Copyright © 2024 by authors and Scientific Research Publishing Inc.

Creative Commons License

This work and the related PDF file are licensed under a Creative Commons Attribution 4.0 International License.