1. Introduction
Technology has evolved the classroom into a powerful and effective learning environment. Technology and machine assisted learning have been around for over half a century since the beginning in the early 1950’s with B. F. Skinner’s teaching machines. One proposed advantage of this type of learning is that the pace of the learning is user driven such that faster learners may learn material more quickly and slower learners may learn material at a pace that is useful for them. Regardless of how fast each student has learned information; in theory, the student will have learned the material sufficiently. Another advantage of machine assisted learning is that each student follows a programmed method which is expected to foster effective learning. By applying effective learning methods over ineffective learning methods efficiency of learning can be maximized.
Technologies for learning apparatuses today are still rapidly advancing. A wide variety of these language learning tools can be seen for modern day computers, tablets, and even mobile phones (Barrs, 2012; Godwin- Jones, 2011; Sandberg, Maris, & de Geus, 2011) . Such learning tools allow the learner to continue learning and provide effective instruction methods even outside of the classroom. However, cognitive psychologists and other researchers are continuously finding new ways to improve learning procedures in order to make them both more effective as well as more efficient. As cognitive psychologists, we must continue to provide new learning methods that will be effective alongside technology as it continuously improves. In other words, as new effective and efficient learning methods are found, they also should be applied to the software that runs new technological devices. This project aims to further develop learning software that is to be used in conjunction with new technological devices through use of an electronic learning environment.
1.1. Effective Language Learning
Cognitive psychology and its memory enhancement/effective learning literature have much to offer language learning applications. The simplest way of foreign language learning may be via rote learning―merely exposing a learner to a native and foreign language word pair. However, rote learning is not the most effective method of learning (Anderson, 2000) . Research consistently finds that retrieval practice often outperforms rote learning ( Fritz, Morris, Action, Voelkel, & Etkind, 2007; Kang, 2010; Pyc & Rawson, 2009 ). Retrieval practice, also known as the testing effect, can be utilized by testing learners immediately after acquisition of new information. In other words, if participants learn new information once and then are tested, they can retain the information more than learning the material twice. It is proposed that testing can strengthen retrieval pathways towards to be remembered information so that it can be more easily recalled later (Tulving & Craik, 2000) . In addition, Karpicke and Roediger (2007) provide evidence that repeated retrieval practice is beneficial to long term memory retention over a single retrieval practice. Pyc and Rawson (2009) found that the first retrieval practice made the greatest impact on memory retention, more so than the second and third tests. The first retrieval must be the most effortful instance, thus the effortful retrieval practice may further strengthen the retrieval pathways. This principle is also closely related with the learning principle known as desirable difficulty.
Although it may sound paradoxical, sometimes it is easier to learn difficult materials. For example, Sungkhasettee, Friedman, and Castel (2011) demonstrated that inverted words were recalled more than upright words. Encoding of inverted words must be more difficult than encoding upright words, thus the increased difficulty during encoding process seems to result in a strong memory trace (leading to a greater likelihood of future recall). Interestingly, a metacognitive questionnaire revealed that learners were not aware of the benefit of the difficulty in learning. Researchers refer to this area of research as desirable difficulties. More specifically, desirable difficulties are learning aids that are introduced during an initial training phase that will typically increase long term memory retention for that respective to-be remembered information (Schmidt & Bjork, 1992; Yue, Bjork, & Bjork, 2013) . The advantage of retrieval practice can also be understood in the framework of desirable difficulty as retrieval practice, which would be the most difficult, yields the largest impact in learning.
1.2. Keyword Method
Another learning method that has been known to be effective in foreign language learning is the keyword method (Fritz et al., 2007; Sagarra & Alba, 2006) . The keyword method requires learners to use strong, vivid imagery to link a foreign word to the equivalent English word. For example, the German for map is karte. Learners can try to make a strong image of former president Jimmy Carter reading a map. In this instance Carter would be the keyword that links the German word karte to its English equivalent map. Prior research suggests that this method of learning is as beneficial for learning as one instance of retrieval practice (Fritz et al., 2007) . The benefit of the keyword method can be explained with the depth of processing phenomenon.
Craik and Tulving (1975) proposed that the depth of processing can explain the strength of memory. In their classic experiment, participants answered various questions about a list of words presented to them. Questions such as “is there a word present?” or “is the word in all capital letters?” were considered to be shallow level of processing questions because the questions activate is shallow or minimal. On the other hand, questions such as “does the word belong to this category?” or “does the word fit in this sentence?” belong to the deeper level of processing questions. Compared to the shallow levels of processing, deeper levels of processing resulted in better memory. The idea that effortful, deeper levels of processing can lead to better memory retention, is now a well-accepted element in cognitive psychology.
1.3. Experiments
The overall goal of this project is to explore the effectiveness of different foreign language learning methods using a computerized learning environment. The learning conditions of the present study are designed such that they can be implemented for language learning applications for computers, tablets, and smart phones. First, we hypothesized that conditions utilizing retrieval practice will outperform the rote learning condition. Second, it is expected that repeated retrieval practice will yield better memory recall than a single instance of retrieval practice. Although retrieval practice has been known to be beneficial for the long-term memory retention, keyword method has shown unique value in foreign language learning. Thus, it is hypothesized that both the repeated retrieval practice, and keyword conditions will be beneficial in foreign language vocabulary learning when administered within an m-learning environment.
2. Method (Experiment 1)
2.1. Participants
Two hundred and two students from the University of Oklahoma participated in Experiment 1. Twenty-two participants were excluded from this study for failing to return to the second part of the experiment. Eight participants were also excluded from this study for indicating intermediate or better previous experience with the German language. For the remaining 172 participants, 30 were male and 142 were female with ages ranging from 18 - 23 (M = 18.66, SE = 0.07).
2.2. Materials and Procedure
Materials used in this experiment were four randomly selected wordlists from Roediger and McDermott (1995) . Each wordlist contained 15 words for a total of 60. These wordlists were translated using an online translator (dict.cc) and were then verified for accuracy by a German instructor. Articles were dropped from all German words (e.g. der, die, das, etc.) and all words were presented in lower case. German words that contained an umlaut were switched to the vowel-e notation. For example the German word kätzchen (kitten in English) would be presented as kaetzchen. The experiment used in this project was programmed using E-prime 2.0. All participants in this experiment were approximately 24'' away from a 17'' square monitor.
This project implemented a 4 (learning schedule) × 2 (retention interval) between factorial design. The learning schedules used in this experiment were similar to Zaromb and Roediger (2010) in which participants were exposed to various combinations of study (S) and test (T). Each participant was exposed to only one combination of study and test. The combinations used in this experiment were as follows: combination one = SS SS SS SS [S], combination two = SS SS SS SS [T], combination three = ST SS ST SS [T], combination four = ST ST ST ST [T]. The [T] represents that at the end of the experiment participants were ran through all the wordlists one final time using the test phase (for a total of 60 trials). All combinations consisted of 540 trials. Participants were asked to return for a retention test after a 1-day delay or 7-day delay.
Before starting the experiment, participants were asked to rate their previous knowledge of the German language (none, beginner, intermediate, advanced, or fluent). Participants were randomly assigned to one of four combinations of study and test conditions and one of the retention intervals. During the first part of the experiment, participants viewed an English word with its corresponding German word translation, and then typed the German word into a text box. For the test phase participants viewed an English word and were asked to type the corresponding German word. All wordlists were counter balanced in order to account for any potential order effects. In between each trial, a fixation point was presented in the middle of the screen for 500 ms. Participants were allowed to take a break between each wordlist. The first part of the experiment took approximately 45 mi- nutes to complete. For the second part of the experiment, participants returned to the same location after 1-day or 7-day delay and were tested over the German words they had learned. The second part of the experiment took approximately 10 minutes to complete.
2.3. Results (Experiment 1)
The number of correctly learned German words was analyzed using a two-way ANOVA. An interaction was found between learning schedule and retention interval such that 1-day delay condition outperformed the 7-day delay condition; likewise as the number of tests increased during the learning phase so did the learners performance F (3, 164) = 2.66, p = 0.05, η2 = 0.05 (see Figure 1). A main effect was found for the learning schedules F (3, 164) = 15.46, p < 0.001, η2 = 0.22. A multiple comparison Bonferroni suggests that there is a significant difference between combination 1 (M = 10.08, SE = 1.50) and all other combinations, combination 2 (M = 18.06, SE = 1.42) and 3 (M = 19.46, SE = 1.41) were significantly different from 1 and 4, and combination 4 (M = 24.21, SE = 1.48) significantly outperformed all other combinations in terms of number of German words learned. These findings suggest that repeated tests are beneficial for long term memory retention. A main effect was also found between the retention intervals such that the 1-day delay (M = 26.28, SE = 1.06) significantly outperformed the 7-day delay (M = 9.62, SE = 1.00) in terms of number of German words learned F (1, 164) = 130.88, p < 0.001, η2 = 0.44. Due to a large difference between the retention intervals each retention interval will be analyzed individually to further investigate the impact of repeated testing and delay length between study and test on foreign language acquisition.
The number of correctly learned German words after 1-day delay was analyzed using a one-way ANOVA with the study-test combination condition as a between subject factor. A significant main effect was found, F (3, 76) = 8.06, p < 0.001, η2 = 0.24. A multiple comparison Bonferroni revealed that there was a significant difference between combination one (M = 15.16, SE = 2.81) and all other combinations. There were no significant differences between combination two (M = 28.80, SE = 2.73), combination three (M = 27.65, SE = 2.73), and combination four (M = 33.52, SE = 2.67). These findings suggest that the retrieval practice has a positive effect on foreign vocabulary word retention after a 1-day delay (see Figure 2).
The number of correctly learned German words after 7-day delay was analyzed using a one-way ANOVA with the study-test combinations as a between subject factor. A significant main effect was found F (3, 88) = 10.35, p < 0.001, η2 = 0.26. A multiple comparison Bonferroni suggests that combination one (M = 5.00, SE = 1.36) was significantly different from combination three (M = 11.27, SE = 1.22) and combination four (M = 14.90, SE = 1.40). Combination two (M = 7.32, SE = 1.25) was significantly different from combination four. Combination three was significantly different from combination one. Combination four was significantly different from combination one and combination two. These findings suggest that repeated retrieval practice has a positive impact on longer retention intervals (see Figure 3).
2.4. Discussion (Experiment 1)
The data suggests that the repeated retrieval practice is the most effective way to aid long-term memory retention of German vocabulary words. The more test trials that were included, the greater the number of correct recalls after a 7-day delay. These findings fall in line with Karpicke and Roediger (2007) that highlight the testing effect in memory. But how does repeated retrieval practice compare to other cognitive mnemonic techniques? Some of the popular memory techniques may not be suitable for foreign language learning. For example, mnemonic devices such as elaborative encoding and depth of processing techniques often require the learner to have prior knowledge about a given subject to have a positive impact on learning (Anderson, 2000) . On the other hand, keyword method that incorporates vivid imagery has been found to be effective in foreign language learning (Fritz et al., 2007; Sagarra & Alba, 2006) . In Experiment 2, we continued to investigate the effectiveness of the keyword method compare to repeated retrieval practice.
3. Method (Experiment 2)
3.1. Participants
Forty-four students from the University of Oklahoma participated in Experiment 2 for class credit. Six participants were removed from the experiment for failing to return to part 2 of the study. All participants indicated little or no prior experience with the German language. For the remaining 38 participants, 9 were male and 29 were female with ages ranging from 18 - 22 (M = 18.63, SE = 0.17).
![]()
Figure 1. Mean number of German words learned after a 1-day and 7-day delay by study and test combination type. Individuals in the shorter retention interval (1-day delay) condition retained more words than in the longer retention interval (7-day delay) condition. Combination one (study only) learned significantly fewer German words than all other combinations. Combination two (one test) and three (three tests) learned significantly fewer German words than combination four. Combination four (five tests) resulted in the greatest memory retention out of all combinations. Number of German words learned is out of 60. Error bars represent standard error.
![]()
Figure 2. Mean number of German words retained after a 1-day delay by study and test combination type. Combination one (study only) resulted in lower German word retention than all other combinations types. Combination two (one test), combination three (three test), and combination four (five test) were not significantly different from one another. Number of German words learned is out of 60. Error bars represent standard error.
![]()
Figure 3. Mean number of German words retained after a 7-day delay by study and test combination type. Combination one (study only) resulted in significantly less German words learned than combination three (three test) and combination four (five test). Combination four resulted in significantly more German words learned than combination one and combination two (one test). Number of German words learned is out of 60. Error bars represent standard error.
3.2. Materials and Procedure
Twenty stimuli were taken at random from Fritz et al. (2007) for the Keyword portion of the experiment. Forty stimuli were randomly taken from the top 200 nouns that were most frequently used in the English language between 2005 and 2006 (Shaoul & Westbury, 2006) . The forty English nouns were then translated using dict.cc and verified by a German instructor for accuracy. All other properties of the stimuli were normalized so that they would match the stimuli in Experiment 1. For the second part of the experiment participants were tested over all 60 words they learned on the previous day. Participants were asked to complete a series of metacognition questions assessing the difficulty of each learning method. A Likert scale was used to rate the difficulty of each learning condition: 1 = hard, 2 = somewhat hard, 3 = neither hard nor easy, 4 = somewhat easy, and 5 = easy.
A 3 within (learning method) × 2 between (typing type) mixed factorial design was used. For the first part of the experiment, all participants ran through rote, test (repeated retrieval practice), and keyword conditions. Twenty English German word pair combinations were used for each condition. In all conditions, participants went through a whole wordlist once (20 words) before repeating the same wordlist 5 more times. For the rote learning condition of the experiment each participant was shown an English German word pair combination and asked to type out the German word in the text box (typing condition) or was merely exposed to the word pair (no typing condition). Each rote word pair combination was seen 6 times for a total of 120 trials. For the testing condition of the experiment, each participant was exposed to the stimuli in following fashion: RT RT RT, where R represents a rote trial and T represents a test trial. During the testing condition, participants were given an English word and asked to type equivalent German word (test trials were always typed out, however during the no typing condition participants did not type in the German word during the rote trials). Each rote word pair occurred 3 times and each testing occurred 3 times for a total of 120 trials. For the keyword condition each participant was given an English word as well as its German counterpart within a keyword phrase. Participants were told that the keyword phrase should help them remember the English German translation. Each participant was asked to type out the German word highlighted within the keyword phrase or was merely exposed to each keyword phrase condition 6 times for a total of 120 trials. Rote, testing, and keyword condition orders were counterbalanced. The learning phase took approximately 35 minutes to complete.
For the second part of the experiment participants were tested over all 60 words they learned on the previous day. Participants were asked to complete a series of metacognition questions assessing the difficulty of each learning method. Participants were also asked to report which learning method they preferred and which they thought would result in the best overall learning out of the retrieval practice, keyword method, and rote methods of learning. The second part of the experiment took approximately 10 minutes to complete. Participants were then thanked for their time, given course credit, and debriefed about the nature of the experiment.
3.3. Results (Experiment 2)
A 3 within (learning method) × 2 between (typing type) mixed ANOVA was used to analyze our data. A main effect was found for the method of learning factor F (1, 36) = 78.19, p < 0.001, η2 = 0.69. A multiple comparison Bonferroni suggests that the repeated retrieval practice (M = 8.87, SE = 0.77) method significantly outperformed the keyword (M = 3.47, SE = 0.54) and rote (M = 3.42, SE = 0.55) methods of learning (see Figure 4). No significant difference was found between the keyword and rote methods of learning. No significance was found between the typing (M = 5.81, SE = 0.64) and no typing (M = 4.70, SE = 0.64) conditions. Our hypothesis that the retrieval practice condition would outperform the rote method of learning was supported however our hypothesis that the keyword method would outperform the rote learning condition was not supported.
Metacognition measures revealed no significant differences in the level of difficulty between the rote (M = 2.37, SE = 0.16), keyword (M = 2.68, SE = 0.21), and retrieval practice (M = 2.82, SE = 0.21) methods of learning F (1, 37) = 3.30. No significant preferences were seen between rote (8), keyword (15), and retrieval practice (15) methods of learning χ2 (2, N = 38) = 2.58. It was found that participants tended to think that the keyword (21) method would results in better learning over the rote (6) and retrieval practice (11) conditions χ2 (2, N = 38) = 9.21, p = 0.01 (see Table 1).
3.4. Discussion (Experiment 2)
It was found that the repeated retrieval practice condition outperformed both keyword and rote methods of
![]()
Figure 4. Mean number of German words retained after a 1-day delay by learning condition. The retrieval practice condition significantly outperformed the keyword and rote methods of learning. No significant differences were found between the keyword and rote methods of learning. Number of German words learned is out of 20. Error bars represent standard error.
![]()
Table 1. Participant’s preferences for learning and predicted best for learning.
learning. These findings conflict with Fritz et al. (2007) which found that retrieval practice and keyword methods performed at comparable levels. Differences in methodology may account for the differences in results. Fritz et al. (2007) used both visual and auditory modalities while our study used only a visual modality. Another key component that our experiment incorporated was repeated retrieval practice. Past research has only compared a single instance of retrieval practice to the keyword method while our study compared multiple instances of retrieval practice. In accordance with Karpicke and Roediger (2007) this current study suggests that multiple instances of retrieval practice behooves long term memory retention.
Metacognition measures suggest that participants think that the keyword method will result in the best learning. These findings are troublesome because in reality the repeated testing condition outperformed the keyword and rote methods of learning. These results fall in line with the desirable difficulty literature which states that learners often do not know which learning procedure is best (Yue et al., 2013) . Due to finding no differences in overall difficulty between the three learning methods it is recommended to use repeated testing when possible when training novice foreign language learners because of the methods memory enhancement properties.
4. Summary and Concluding Discussion
The present study highlights the robustness of retrieval practice and repeated retrieval practice in the area of foreign language learning. In both experiments it was found that repeated retrieval practice outperformed rote methods of learning, such findings fall in line with prior research (Karpicke & Roediger, 2007; Pyc & Rawson, 2009) . Retrieval practice and repeated retrieval practice is thought to strengthen retrieval pathways to target to-be-remembered information so that data may be easier retrieved at a later time. Inconsistent with prior research (Fritz et al., 2007) , it was found that the keyword method of learning was no more effective than rote learning conditions.
Metacognitive measurements included in Experiment 2 indicate that participants thought that the keyword method would result in better learning when in reality the repeated retrieval practice (or testing) method resulted in the best learning. These findings with metacognition measures are consistent with prior research that suggests that learners are often unaware which methods of learning are most effective in reality (Yue et al., 2013) . If electronic learning software is properly programmed with effective learning methods then the learners that use these devices will be taught in an effective fashion regardless of what they think is the best way of learning.
Retrieval practice and repeated retrieval practice seem powerful enough to be implemented without the use of prior knowledge; recall that all the learners included this study had little or no knowledge of the German language. As such, future projects should inquire the effectiveness of retrieval practice and repeated retrieval practice in research and applied settings of learning. Future researchers should also include an auditory modality which may further aid learners in foreign language learning. Lastly, the use of feedback has also seen some effectiveness in the area of foreign language learning and should be further inquired in conjunction with repeated retrieval practice (Kang, McDermott, & Roediger, 2007) .
Acknowledgements
We thank the German material reviewer John W. Love from the University of Oklahoma for his contributions to this project.