Learning Behavior Analysis and Learning Effect Evaluation in Open Online Courses

Xiufang Li

Qilu University of Technology (Shandong Academy of Sciences), Jinan, China.

DOI: 10.4236/ce.2022.134081                

Abstract

In the context of globalization, the trend of globalization in the field of education is becoming more and more significant. With the maturity of Internet and cloud computing, a large online open course learning platform providing courses and educational services for global users has emerged in the past few years. This is not only the innovation of Internet applications, but it is also believed that it will trigger a change in higher education and social development. The main body of online open courses is learners, and its biggest feature is that there are a large number of learners and a variety of learner groups. Due to the characteristics of Internet technology, all learning behaviors of learners on the online open course platform will be recorded in the form of rich and diverse data. Therefore, it is necessary to analyze learners’ learning behavior. This paper proposes a dual-channel clustering algorithm, which analyzes and mines a large number of learning behavior data of more than 5000 learners in online open courses of a university. This method takes fine-grained data as the core, obtains the types of learners in different models through dual-channel clustering calculation, and finally characterizes learners based on the fused model. Compared with three state-of-the-art clustering algorithms, the experimental results show that the proposed dual-channel clustering algorithm can enhance the cohesion of clusters, cluster learners more accurately, and characterize learners’ profiles more deeply and comprehensively.

Keywords

Learning Behavior Analysis, Learning Effect Evaluation, Open Online Cours-es, Cluster

1. Introduction

In online learning, teachers can only know the learning effect of students through the examination of each course. It is difficult to obtain the specific learning process of each learner in the teaching process. There are problems such as separation of teaching, difficulty in monitoring and distance. When the learners with a state of decline, there is no reasonable intervention to make the learner develop in the right direction (Hoi et al., 2021; Kuo et al., 2021). Therefore, it is necessary to supervise the learner’s learning process, predict the learner’s learning trend in time, take appropriate intervention measures for different learning effects, and give the learner targeted help and guidance or encouragement, so that the learner can constantly correct the learning route in the learning process.

With the increasing amount of data on the online learning platform, researchers and scholars begin to find ways to make these data comprehensible and meaningful (Ma et al., 2021; Cheon et al., 2021). To analyze and mine more potential educational information, researchers deeply explore the theory, framework, tools and practice of learning analysis (Zhang et al., 2020). In recent years, more and more studies on learning behavior analysis has been carried out, and predicting student performance has attracted scholars’ attention (Sedrakyan et al., 2020). Since 2013, with the continuous development of learning analysis research, researchers have started to use machine learning to study learning prediction. This is also due to the development of Massive Open Online Courses (MOOC) and other platforms (Wang et al., 2019; Jin, 2020). A large number of platform users generate large-scale educational data. In view of the phenomena that more registered users drop out on MOOC platform, researchers begin to explore the rules between user behavior and whether to drop out or whether to get a certificate. By analyzing the behavior information of users, we hope to find out the rules, so as to take early measures to reduce the dropout rate on MOOC platform, so that more excellent resources can be better learned.

Through statistical analysis of behavior data of online learning platform, we can gain a deeper understanding of students and help to provide adaptive learning guidance. There are a lot of hidden educational data stored on the online learning platform, which undoubtedly can make the teachers have a deeper understanding of the students. For teachers, facing tens of thousands of learners, they cannot really understand them in a short time. Through statistical analysis of behavior data, learners’ learning characteristics can be quickly and intuitively understood, thus learners can be classified and analyzed. For different types of learners, teachers can timely adjust their teaching methods and strategies, so as to provide different learners with teaching resources that are as appropriate as possible. For students, by reviewing the results of the analysis, they can learn about the gaps or strengths with other learners, so as to adjust their learning plans and constantly improve themselves (Heymann et al., 2022; Giddens et al., 2021; Sun et al., 2020).

Using data mining method to predict the learning effect can predict the learning trend in advance and facilitate taking appropriate intervention measures. Predicting learner performance by analyzing learning behavior can help developers evaluate online learning system more effectively, continuously improve system availability and expand system functions, so as to visualize learner behavior and future development trend. At the same time, it also helps the teachers to understand the development trend of learner’s behavior and intervene the learners artificially when appropriate. It also helps the teachers to improve the courses and improve the quality of teaching. Online learning platforms can interfere with poor learners at appropriate times, such as dialog box prompts or learning resource recommendations (Wang, 2021; Xue et al., 2021; Ma et al., 2021).

Considering the implicitness of educational resource data, many attributes cannot be obtained directly from data records. Using statistical coarse-grained data as input only will result in a certain degree of information loss, while using fine-grained characteristics can reduce the confusion caused by the implicitness of data. In the process of educational data mining, most of the studies only model single-way data mining based on single-angle coarse-grained data, which is prone to poor classification effect and incomplete analysis (Trakunphutthirak & Lee, 2021). To solve the above problems, a dual-channel clustering modeling method is presented. This method classifies students’ data characteristics based on fine-grained data as the core feature, carries out two-angle clustering modeling on behavior data features and academic data features, finally fuses the model, carries out various analysis and classification on learners’ learning behavior.

The main contribution of this paper is that dual-channel clustering algorithm is proposed to analyze learning behavior of learners in online open courses. The rest of this paper is organized as follows. Section 2 is the related works. In Section 3, dual-channel clustering algorithm is presented. The experimental results are shown in Section 4. Section 5 gives the conclusion and future work of this paper.

2. Related Works

2.1. Study on Learning Behavior Analysis

Learning behavior analysis is a technology that explores the relationship between learner’s learning behavior and learning effect by mining and analyzing the data left by learners on the learning platform or learning system. Scholars at home and abroad have conducted extensive studies on the analysis of learning behavior, mainly focusing on prediction, relationship mining and teaching optimization. Through the collaborative data analysis of online learning behavior, Shou et al. (2020) proposed a learning path planning algorithm based on collaborative analysis of learning behavior. Yang & Chen (2020) used the method of lag sequence analysis to investigate the overlap between the two cognitive style dimensions from the perspective of online learning behavior. Wang et al. (2020) proposed the design of an operating system experimental course with a flat learning curve and an extensible operating system experimental platform to support learning behavior analysis. Zhao et al. (2021) proposed a learning behavior analysis based on result confirmation framework, which added a result confirmation step to explore the reasons behind learning patterns and strategies. Fan et al. (2021) integrated multiple data sources and proposed an interpretable method to analyze students’ learning behavior and recommend MOOC.

2.2. Clustering Algorithms

Clustering refers to the process of dividing datasets into multiple groups (Wang et al., 2021; Son & Hong, 2021). As a useful data analysis tool, clustering is widely used in pattern recognition (Riaz et al., 2021; Lv et al., 2017; Fard et al., 2018), data mining (Jiao & Li, 2021; Zou, 2020; Kausar et al., 2018), image segmentation (Khan et al., 2019; Lei & Ouyang, 2019) and other aspects. The main goal of clustering is to divide the given multi-attribute sample dataset into several groups, so that the samples in the same group are similar to each other and the samples in different groups are different from each other. Cluster analysis can find the distribution characteristics hidden in the data, which lays a foundation for further fully and effectively using the data to obtain useful information. At present, the commonly used clustering methods mainly include partition method (Kang et al., 2020), hierarchical method (Xu et al., 2020), density-based method (Wang et al., 2021), grid-based method (Starczewski et al., 2021) and model-based method (Yang et al., 2019). Liu & Li (2020) proposed a clear hierarchical clustering objective function, called Bayesian hierarchical k-means (BHK means). Tan et al. (2020) proposed a high-order fuzzy clustering algorithm called multi-kernel mean-shift (MKMS-HoFC), using the method based on multi-kernel spatial mean-shift to segment the data, and expands the original dimension into multiple new dimensions in the high-dimensional kernel feature space. Sinaga & Yang (2020) proposed a new unsupervised k-means (U-k-means) clustering algorithm, which can automatically find the optimal number of clusters without any initialization and parameter selection.

3. Proposed Methodology

Through the mining of educational data and clustering of student, students can be personalized analysis and guidance. Combine with student attribute features, the dual-channel clustering modeling method is proposed in this section, and the framework is shown in Figure 1.

3.1. Data Source

The original data of this paper comes from the online platform of a university. This platform integrates computer test platform, course platform and forum. Courses are uploaded to the platform in the form of video recording, and students can start learning after successful application. The operation of students on the platform will leave log files, such as on-demand recording, forum speaking time, learning time, etc. Raw data can be divided into three groups: video

Figure 1. Architecture of dual-channel clustering modeling method.

viewing, forum discussion and online work.

It is important to understand the learner’s interaction with the distance education system. Psychological studies show that through the analysis of behavior, effective information such as human motivation, state and goal can be distinguished, which provides a basis for the development of personalized curriculum and reasonable evaluation of learning effect. Learners’ behavior analysis needs a lot of data support, such as video playing duration, forum discussion, etc. This paper collected 650,773 on-demand records and 157,635 discussion records from 5427 students who participated in the “educational psychology” course from September 1, 2021 to December 31, 2021.

3.2. Feature Construction Layer

The input coding layer mainly performs data cleaning for the features of the original samples. At first, it preprocesses the attributes, such as standardization, discretization and filling missing values. Then, feature construction is performed based on the preprocessed data.

This section constructs the following eight data features for each sample in the dataset: number of video playing (playNum), total video playing time (playTime), number of knowledge points (kNum), number of discussion (dNum), amount of speech (speechAmount) and three fine-grained features (value of learning attitude (lAttitude), knowledge points entropy (kEntropy) and pass rate (kPass).

Value of learning attitude represents the learning attitude of students participating in the course.

${\text{lAttitude}}_k=\log \left(\frac{{d^{\prime }}_k}{d_k}+1\right){d^{\prime }}_k{\gamma }_k$ (1)

where $m_i$ is the difference between the start learning date and end learning date of student k, and represents the learning cycle. ${d^{\prime }}_k$ is the effective learning days of student k in the learning cycle, and ${d^{\prime }}_k/d_k$ is the learning density of student k. ${\gamma }_k$ is the division of student k’s learning time and average learning time.

Entropy of knowledge points represents the breadth of student’s involvement in learning process. Information entropy is commonly used to measure the purity of sample sets. The higher the entropy, the lower the purity. Standardized information entropy is defined as follows.

$\text{SIE}\left(S\right)=\frac{-{\displaystyle {\sum }_{i=1}^{\delta }{p}_i\log p_i}}{\log \delta }$ (2)

where S is the sample set, $p_i$ is the proportion of the th sample in the sample set, and δ is the total number of groups.

Supposing the video played by student k contains m knowledge points, and the learning duration of each knowledge point is $t_1,t_2,\cdots ,t_m$, the knowledge point entropy of student k is defined as follows.

$\text{kEntropy}\left(x\right)=\frac{-{\displaystyle {\sum }_{k=1}^m\frac{t_k}{t}\log \frac{t_k}{t}}}{\log m}$ (3)

$t={\displaystyle {\sum }_{k=1}^m{t}_k}$ (4)

The lower the entropy of knowledge points, the narrower the breadth of knowledge that the student has learned, and the length of learning on one knowledge point is significantly longer than other knowledge points. The higher the entropy of knowledge points is, the wider the breadth of knowledge the student has learned is, and the student is inclined to exert power evenly on the knowledge points he has learned.

The pass rate of knowledge points is used to reflect students’ learning degree of knowledge points in the learning process. The higher the pass rate is, the higher the learning degree is, and its function is defined as follows.

$\text{kPass}\left(x\right)=\frac{\mathrm{kNu}{m}^{\prime }}{\text{kNum}}$ (5)

where kNum is the number of knowledge points learned by learners, and kNum’ is the number of knowledge points effectively completed by learners. In the on-demand record of learners, if the cumulative playing time of a certain knowledge point exceeds 90% of the total video time of the knowledge point, it is regarded as effectively completed.

After the feature construction is completed, each student in the dataset is mapped to an eight-dimensional eigenvector, and finally 5427 ´ 8 dimensional student group feature data is obtained.

3.3. Dual-Channel Clustering Layer

The goal of clustering is to discover clusters naturally formed in data and mine the information contained in data itself. Therefore, the input feature data will have a decisive influence on clustering. In this section, the eight features are further subdivided, in which number of video on-demand, total video playing time, number of discussion, amount of speech and value of learning attitude related to online learning behavior are called Behaviors Features (BF), and number of knowledge points, knowledge points entropy and pass rate related to course knowledge are called Academic Features (AF) (Xu et al., 2019).

The dual-channel clustering layer will cluster from the perspectives of learning behavior feature data and learning academic feature data respectively. Based on the feature data of different data types, eigenmatrix is constructed and data models from different perspectives are established to describe the learning profile of each student more comprehensively. The main process of dual-channel clustering is as follows.

1) Eigenmatrix construction. Generally, it is assumed that the feature set of student k is $f_k$, which consists of a series of features, and $f_k=\left\{x_1,x_2,\cdots ,x_m\right\}$. The eigenmatrix $\mathbb{A}$ is defined as follows.

$\mathbb{A}=\left[\begin{array}{cc}\begin{array}{c}\begin{array}{cc}\begin{array}{cc}{x}_{11}& x_{12}\end{array}& \cdots \end{array}\\ \begin{array}{cc}\begin{array}{cc}{x}_{21}& x_{22}\end{array}& \cdots \end{array}\end{array}& \begin{array}{c}{x}_{1q}\\ x_{2q}\end{array}\\ \begin{array}{cc}\begin{array}{cc}\begin{array}{c}⋮\\ x_{p1}\end{array}& \begin{array}{c}⋮\\ x_{p2}\end{array}\end{array}& \begin{array}{c}\\ \cdots \end{array}\end{array}& \begin{array}{c}⋮\\ x_{pq}\end{array}\end{array}\right]$ (6)

where p is the dimension of each feature column and q is the number of features.

2) Dual-channel clustering modeling. In this section, k-means clustering method is used for clustering modeling of student feature data, and the eigenmatrix of established behavior data and academic data is taken as input, and the sum of square error (SSE) within the cluster and elbow method are combined to determine the optimal number of clustering. Let r ands denote the serial number of the current sample and the serial number of the current cluster respectively, then the SSE within the cluster is defined as follows.

$\text{SSE}={\displaystyle {\sum }_{r=1}^q{\displaystyle {\sum }_{s=1}^i{\Vert x^{\left(r\right)}-c^{\left(s\right)}\Vert }_2^2}}$ (7)

where $c^{\left(s\right)}$ is the center point of cluster s. If sample $x^{\left(r\right)}$ belongs to cluster s, ${\Vert x^{\left(r\right)}-c^{\left(s\right)}\Vert }_2^2=1$, otherwise ${\Vert x^{\left(r\right)}-c^{\left(s\right)}\Vert }_2^2=0$.

3.4. Group Fusion Layer

The dual-channel clustering layer is used to cluster the student samples from two aspects of behavioral features and academic features. Assuming that the set A of cluster categories generated by behavioral feature clustering $A=\left\{a_1,a_2,\cdots ,a_q\right\}$ and the set B of cluster categories generated by academic feature clustering $B=\left\{b_1,b_2,\cdots ,b_p\right\}$, take a and b as cartesian product, then matrix $M=A\times B$. Each element in matrix M is the group fusion. Finally, k groups with the largest number and the most typical groups are selected as the output result.

4. Experimental Results and Analysis

4.1. Comparison Experiment

The dual-channel clustering algorithm is implemented by Matlab R2020a. Experiments are performed on a computer with Intel i7-12700KF 3.6 GHz CPU, 32GB of RAM (3200 MHz). In the dataset collected in this paper, BHK means (Liu & Li, 2020), MKMS-HoFC (Tan et al., 2020), U-k-means (Sinaga & Yang, 2020) and the dual-channel clustering algorithm proposed in this paper are used to compare the classification performance of the four algorithms through running time, intra-cluster sum error and contour coefficient. According to the experimental results (Table 1), dual-channel clustering algorithm has the smallest intra-cluster SSE and the highest contour coefficient, indicating that dual-channel clustering algorithm can effectively enhance the cohesion of clusters and make clustering more accurate. The running time of the dual-channel clustering algorithm is slightly higher than that of BHK means, MKMS-HoFC and U-k-means algorithms, and the reason is that although algorithm dual-channel clustering adopts parallel clustering in dual-path clustering to accelerate its running speed, the algorithm needs to be coded in the feature construction layer before clustering, which consumes a certain amount of time.

Table 1. Performance metrics between four algorithms on the same dataset.

4.2. Clustering Results and Analysis

The algorithm proposed in this paper classifies 5427 learners into five groups. To further test the clustering effect, this paper uses Principal Component Analysis (PCA) dimensionality reduction algorithm to visualize the clustering effect. It can be seen from Figure 2 that learners are obviously clustered into five groups, which are Cluster 1 (C1), Cluster 2 (C2), Cluster 3 (C3), Cluster 4 (C4) and Cluster 5 (C5).

According to the clustering results, C1, C2, C3, C4 and C5 contain 2362, 1178, 876, 706 and 305 students respectively. To better show the differences between different types of learners, the statistical characteristics of learners of different types are calculated (Table 2), such as minimum, maximum and mean number.

The behaviors of the five types of learners are analyzed as follows. In terms of learning attitude, C4 has the best learning attitude with an average of 18, which is much higher than the other four types. In contrast, the learning attitude of other types of learners is relatively low, and the learning attitude value is lower than 2. From the perspective of video playing, C4 is still in a prominent position. The average number of playing is close to 300 times and the average playing time is about 68,000 seconds, which is several to tens of times longer than that of other learner., indicating that C4 plays learning videos more frequently and also confirming that C4 has the best learning attitude. C5 shows outstanding performance

Figure 2. Visualization of clustering results.

Table 2. Overview of statistical features of learners.

in participating in discussions, far outperforming other types of learners, indicating that C5 is very active after class and often uses forums to communicate with classmates and teachers.

To further analyze the performance of learners in academic characteristics, C4 has the most knowledge points, the highest entropy of knowledge points and the highest pass rate of knowledge points, indicating that C4 has comprehensive learning knowledge with emphasis and good learning effect. C2 does not learn many knowledge points, but the pass rate of knowledge points is only next to C4, indicating that C2 performs selective learning in a targeted way, and such students are often students with basic knowledge. All academic characteristics of C3 are at a low level, indicating that the learning state is in urgent need of adjustment. C1 has learned more knowledge points than C3, but it is still in an unsatisfactory state, and the pass rate is only 26%, indicating that C1’s learning is partial and not deep enough. C5 watches more courses than C1, but the pass rate is only 17%, and the learning status is slightly worse than C1.

It can be seen from Figure 3 that the proportion of different types of learners registering at different times. 1) The proportion of C2 registering at the beginning of the course was as high as 60%, and then dropped rapidly to the following one month. 2) C1 accounts for about half of the total learners, which gradually increases after the beginning of the semester, and finally stabilizes at about 50%. 3) The curves of C3 and C4 are in a steady and slow decline state, while the curves of C5 remain in a stable state. 4) In the stage after Dec 20, the curves of

Figure 3. Distribution of registration proportion of all types of learners at different time.

C1, C2 and C4 fluctuate greatly. This is because this stage is at the end of the semester, and there are few students joining the course. Some small increase or decrease in the data will cause a relatively large shock.

As can be seen from Figure 4, the classification accuracy of the algorithm in this paper is the highest for each cluster, and the lowest accuracy is above 90%. However, BHK means algorithm has the lowest accuracy, which is due to the poor clustering effect of hierarchical clustering on high-dimensional data features, and poor clustering effect if the dataset distribution cluster is not similar to hypersphere or convex. Because of the time complexity of the algorithm, the result of hierarchical clustering depends on the selection of the merging point and the splitting point. Moreover, the most obvious feature of hierarchical clustering is irreversibility. After the objects are merged or split, the next cluster will continue to merge or split on the basis of the previous one. That is to say, once the clustering result is formed, it is impossible to re-merge to optimize the performance of the cluster. However, MKMS-HoFC and U-k-means algorithms have different classification accuracy for each cluster, and their classification accuracy is better than that of BHK means algorithm.

As indicated in Figure 5, the classification time of the dual-channel clustering algorithm proposed in this paper is the lowest, which is lower than 70 ms on average. This is because dual-channel clustering algorithm can enhance the cohesion of clusters and cluster learners more accurately. MKMS-HoFC algorithm has the highest classification time, because although mean-shift clustering does

Figure 4. Comparison of classification accuracy.

Figure 5. Comparison of classification time.

not need to select the number of clusters, the selection of window size has a great impact on the classification time. On the other hand, the classification time of BHK Means algorithm is relatively low, because hierarchical clustering can obtain multi-level clustering structure with different granularity by setting different related parameter values, thus reducing the classification time.

Given the above, the characteristics of C1, C2, C3, C4 and C5 are summarized as follows.

C1: Partially learning. Such students learn less knowledge points, resulting in insufficient learning time. The characteristics of learning attitude, knowledge points entropy and pass rate are mediocre, and there are a large number of such learners, indicating that the level of online education still has a large space for improvement.

C2: Selective learning. These types of learners are mostly students with basic knowledge and are similar to C1 in behavioral characteristics, so they are not dominant in learning. However, they should be distinguished from C1, because C2 students selectively learn some knowledge points in depth.

C3: Bystander. Such students are on the edge of quitting learning, and their characteristics are in a bad state, showing a state of passive neglect. Therefore, we should fully mobilize the learning enthusiasm of such students.

C4: Comprehensive learning the characteristics of such students show that their learning state is excellent, their knowledge points are studied comprehensively, their continuous learning time is long, and they can also distinguish key knowledge and have a certain subjective initiative.

C5: Discussion-oriented. Such students actively participate in the discussion in the forum, and they are the main force of the active forum, but their learning is not as comprehensive as C4. Therefore, it is can be concluded that there may be problems in their learning methods, which need to be corrected by teachers in time.

Through the personalized analysis of five typical students and combined with the advantages of online education, the following suggestions are put forward.

1) The learning effect of online learning can be further improved. Teachers should pay more attention to the breadth and depth of students’ learning and reduce some flexibility in the design of learning platform. For example, learning videos can be designed so that the progress bar cannot be dragged, but it can be fast forward and rewind appropriately.

2) Students pay little attention to the forum in the learning platform, and the role of the forum is not brought into full play. Teachers should actively guide students to ask questions, give full play to students’ subjective initiative, or appropriately improve the difficulty of the subject.

3) For excellent students, an improvement class can be set up to learn knowledge more in line with their own level. At the same time, the learning platform can add a “traffic light” function to warn students. When a red light appears, it means that their learning status has been lower than the average level of students in the same batch. This way of internal competition among students may be more effective than external pressure.

5. Conclusion and Future Work

To solve the problem of insufficient log file mining of online education platform, this paper mines 650,773 on-demand records and 157,635 discussion records of 5427 students, trying to find the rules and characteristics of students’ learning behavior, so as to improve the teaching level of online education. Based on log records, this paper explores multiple fine-grained features, proposes a dual-channel clustering modeling method, analyzes all kinds of learners in detail, and characterizes five typical learners. The results show that the proposed dual-channel clustering modeling method with fine-grained features as the core can effectively subdivide different types of learners, and the time complexity is low, which is conducive to its application in large datasets.

Further research could focus on how to better evaluate students’ learning outcomes, rather than relying solely on statistical data. At the same time, personalized test recommendation will be a research focus. As for teacher-level data mining, there are still few research materials at present. It is also worth paying attention to explore the pattern matching between “teaching” and “learning” through the interaction between teachers and students.

Acknowledgements

This paper is supported by Key Teaching Research Project of Qilu University of Technology (Shandong Academy of Sciences) (Granted NO. 2019zd16).

Conflicts of Interest

The authors declare no conflicts of interest regarding the publication of this paper.

References

[1]	Cheon, J., Cheng, J. M., & Cho, M. H. (2021). Validation of the Online Learning Readiness Self-Check Survey. Distance Education, 42, 599-619. https://doi.org/10.1080/01587919.2021.1986370
[2]	Fan, J., Jiang, Y., Liu, Y., & Zhou, Y. (2021). Interpretable MOOC Recommendation: A Multi-Attention Network for Personalized Learning Behavior Analysis. Internet Research, 32, 588-605. https://doi.org/10.1108/INTR-08-2020-0477
[3]	Fard, E. R., Aghayar, K., & Amniat-Talab, M. (2018). Quantum Pattern Recognition with Multi-Neuron Interactions. Quantum Information Processing, 17, Article No. 42. https://doi.org/10.1007/s11128-018-1816-y
[4]	Giddens, J., Curry-Lourenco, K., Miles, E., & Reeder E. (2021). Enhancing Learning in an Online Doctoral Course through a Virtual Community Platform. Journal of Professional Nursing, 37, 184-189. https://doi.org/10.1016/j.profnurs.2020.05.007
[5]	Heymann, P., Bastiaens, E., Jansen, A., van Rosmalen, P., & Beausaert, S. (2022). A Conceptual Model of Students’ Reflective Practice for the Development of Employability Competences, Supported by an Online Learning Platform. Education and Training. (In Print) https://doi.org/10.1108/ET-05-2021-0161
[6]	Hoi, S. C. H., Sahoo, D., Lu, J., & Zhao, P. (2021). Online Learning: A Comprehensive Survey. Neurocomputing, 459, 249-289. https://doi.org/10.1016/j.neucom.2021.04.112
[7]	Jiao, G., & Li, W. (2021). Neural Network Data Mining Clustering Optimization Algorithm. IETE Journal of Research. https://doi.org/10.1080/03772063.2021.1965043
[8]	Jin, C. (2020). MOOC Student Dropout Prediction Model Based on Learning Behavior Features and Parameter Optimization. Interactive Learning Environments.
[9]	Kang, Z., Zhao, X., Peng, C., Zhu, H., Zhou, J., Peng, X. et al. (2020). Partition Level Multiview Subspace Clustering. Neural Networks, 122, 279-288. https://doi.org/10.1016/j.neunet.2019.10.010
[10]	Kausar, S., Xu, H., Hussain, I., Zhu, W., & Zahid, M. (2018). Integration of Data Mining Clustering Approach in the Personalized E-Learning System. IEEE Access, 6, 72724-72734. https://doi.org/10.1109/ACCESS.2018.2882240
[11]	Khan, Z., Yang, J., & Zheng, Y. (2019). Efficient Clustering Approach for Adaptive Unsupervised Colour Image Segmentation. IET Image Processing, 13, 1763-1772. https://doi.org/10.1049/iet-ipr.2018.5976
[12]	Kuo, T. M., Tsai, C. C., & Wang, J. C. (2021). Linking Web-Based Learning Self-Efficacy and Learning Engagement in MOOCs: The Role of Online Academic Hardiness. Internet and Higher Education, 51, Article ID: 100819. https://doi.org/10.1016/j.iheduc.2021.100819
[13]	Lei, X., & Ouyang, H. (2019). Image Segmentation Algorithm Based on Improved Fuzzy Clustering. Cluster Computing, 22, 13911-13921. https://doi.org/10.1007/s10586-018-2128-9
[14]	Liu, Y., & Li, B. (2020). Bayesian Hierarchical K-Means Clustering. Intelligent Data Analysis, 24, 977-992. https://doi.org/10.3233/IDA-194807
[15]	Lv, J., Wang, X., Ren, K., Huang, M., & Li, K. (2017). ACO-Inspired Information-Centric Networking Routing Mechanism. Computer Networks, 126, 200-217. https://doi.org/10.1016/j.comnet.2017.07.004
[16]	Ma, L., Huang, M., Yang, S., Wang, R., & Wang, X. (2021). An Adaptive Localized Decision Variable Analysis Approach to Large-Scale Multiobjective and Many-Objective Optimization. IEEE Transactions on Cybernetics, 52, 1-13. https://doi.org/10.1109/TCYB.2020.3041212
[17]	Riaz, M., Habib, A., Khan, M. J., & Kumam, P. (2021). Correlation Coefficients for Cubic Bipolar Fuzzy Sets with Applications to Pattern Recognition and Clustering Analysis. IEEE Access, 9, 109053-109066. https://doi.org/10.1109/ACCESS.2021.3098504
[18]	Sedrakyan, G., Malmberg, J., Verbert, K., Jarvela, S., & Kirschner, P. A. (2020). Linking Learning Behavior Analytics and Learning Science Concepts: Designing a Learning Analytics Dashboard for Feedback to Support Learning Regulation. Computers in Human Behavior, 107, Article ID: 105512. https://doi.org/10.1016/j.chb.2018.05.004
[19]	Shou, Z., Lu, X., Wu, Z., Yuan, H., Zhang, H., & Lai, J. (2020). On Learning Path Planning Algorithm Based on Collaborative Analysis of Learning Behavior. IEEE Access, 8, 119863-119879. https://doi.org/10.1109/ACCESS.2020.3005793
[20]	Sinaga, K. P., & Yang, M. S. (2020). Unsupervised K-Means Clustering Algorithm. IEEE Access, 8, 80716-80727. https://doi.org/10.1109/ACCESS.2020.2988796
[21]	Son, S., & Hong, S. (2021). Multiple Group Analysis in Multilevel Data across Within-Level Groups: A Comparison of Multilevel Factor Mixture Modeling and Multilevel Multiple-Indicators Multiple-Causes Modeling. Educational and Psychological Measurement, 81, 904-935. https://doi.org/10.1177/0013164420987899
[22]	Starczewski, A., Scherer, M. M., Ksiazek, W., Debski, M., & Wang, L. (2021). A Novel Grid-Based Clustering Algorithm. Journal of Artificial Intelligence and Soft Computing Research, 11, 319-330. https://doi.org/10.2478/jaiscr-2021-0019
[23]	Sun, Z., Anbarasan, M., & Kumar, D. P. (2020). Design of Online Intelligent English Teaching Platform Based on Artificial Intelligence Techniques. Computational Intelligence, 37, 1166-1180. https://doi.org/10.1111/coin.12351
[24]	Tan, D., Zhong, W., Jiang, C., Peng, X., & He, W. (2020). High-Order Fuzzy Clustering Algorithm Based on Multikernel Mean Shift. Neurocomputing, 385, 63-79. https://doi.org/10.1016/j.neucom.2019.12.030
[25]	Trakunphutthirak, R., & Lee, V. C. S. (2021). Application of Educational Data Mining Approach for Student Academic Performance Prediction Using Progressive Temporal Data. Journal of Educational Computing Research. https://doi.org/10.1177/07356331211048777
[26]	Wang, J., Wang, X., Yu, G., Domeniconi, C., Yu, Z., & Zhang, Z. (2021). Discovering Multiple Co-Clusterings with Matrix Factorization. IEEE Transactions on Cybernetics, 51, 3576-3587. https://doi.org/10.1109/TCYB.2019.2950568
[27]	Wang, L., Zhen, Z., Wo, T., Jiang, B., Sun, H., & Long, X. (2020). A Scalable Operating System Experiment Platform Supporting Learning Behavior Analysis. IEEE Transactions on Education, 63, 232-239. https://doi.org/10.1109/TE.2020.2975556
[28]	Wang, S. (2021). Online Learning Behavior Analysis Based on Image Emotion Recognition. Traitement Du Signal, 38, 865-873. https://doi.org/10.18280/ts.380333
[29]	Wang, W., Guo, L., & Sun, R. (2019). Rational Herd Behavior in Online Learning: Insights from MOOC. Computers in Human Behavior, 92, 660-669. https://doi.org/10.1016/j.chb.2017.10.009
[30]	Xu, Q., Zhang, Q., Liu, J., & Luo, B. (2020). Efficient Synthetical Clustering Validity Indexes for Hierarchical Clustering. Expert Systems with Applications, 151, Article ID: 113367. https://doi.org/10.1016/j.eswa.2020.113367
[31]	Xu, X., Wang, J., Peng, H., & Wu, R. (2019). Prediction of Academic Performance Associated with Internet Usage Behaviors Using Machine Learning Algorithms. Computers in Human Behavior, 98, 166-173. https://doi.org/10.1016/j.chb.2019.04.015
[32]	Xue, Y., Qin, J., Su, S., & Slowik, A. (2021). Brain Storm Optimization Based Clustering for Learning Behavior Analysis. Computer Systems and Engineering, 39, 211-219. https://doi.org/10.32604/csse.2021.016693
[33]	Yang, M. S., Chang-Chien, S. J., & Nataliani, Y. (2019). Unsupervised Fuzzy Model-Based Gaussian Clustering. Information Sciences, 481, 1-23. https://doi.org/10.1016/j.ins.2018.12.059
[34]	Yang, T. C., & Chen, S. Y. (2020). Investigating Students’ Online Learning Behavior with a Learning Analytic Approach: Field Dependence/Independence VS. Holism/Serialism. Interactive Learning Environments. https://doi.org/10.1080/10494820.2020.1817759
[35]	Zhang, J., Zuo, L., Miao, J., Zhang, Y., Hwang, G., & Zhu, Y. (2020). An Individualized Intervention Approach to Improving University Students’ Learning Performance and Interactive Behaviors in a Blended Learning Environment. Interactive Learning Environments, 28, 231-245. https://doi.org/10.1080/10494820.2019.1636078
[36]	Zhao, F., Hwang, G., & Yin, C. (2021). A Result Confirmation-Based Learning Behavior Analysis Framework for Exploring the Hidden Reasons behind Patterns and Strategies. Educational Technology & Society, 24, 138-151.
[37]	Zou, H. (2020). Clustering Algorithm and Its Application in Data Mining. Wireless Personal Communications, 110, 21-30. https://doi.org/10.1007/s11277-019-06709-z