COVID-19: How Do Engineering Students Assess its Impact on Their Learning?

COVID-19 pandemic forced a rapid transition from face-to-face learning to an online format at virtually all universities worldwide. The Department of Civil, Environmental, and Construction Engineering (CCEE) at San Diego State University (SDSU) had to make such a transition in mid-March 2020 within a ten-day period. The Civil Engineering program needs to comply with strict ABET accreditation rules. One of them is an indirect assessment made by students in each course: they judge to which degree ABET-mandated Student Outcomes (SO) were actually met in any course designated to deliver them. The unprecedented pandemic situation created an opportunity to administer a student survey twice to detect any differences between the SO coverage before the COVID lockdown in the middle of the Spring 2020 semester and the parallel coverage of the same SOs during the pandemic period throughout the rest of the semester. With all other conditions remaining un-changed, this creates an opportunity to examine the actual impacts of COVID in a relatively unbiased way. The purpose of this research is to find out whether COVID-19 created any significant changes in students’ assessment of their learning as mandated by ABET. The research method applied is statistical analysis of student scores submitted by a self-administrated online survey. The two-sided t-test was used to assess the statistical significance of the differences observed. This paper does not address any issue related to the students’ or instructors’ perceptions of problems created for them by the COVID pandemic: they were real and challenging for both parties. Instead, it focuses exclusively on numbers resulting from students’ surveys. The scores for SO coverage during the pandemic remained quite high although generally coverage in their courses. Courses and SOs that were significantly negatively affected by the COVID restrictions were the ones that had to rely on lab ex-perimentations and student teamwork.


Introduction
COVID-19 outbreak created unprecedented transitions in all aspects of life around the globe. Universities had to comply with the directives from the national and local health authorities to switch from traditional face-to-face instruction to remote learning. San Diego State University and its College of Engineering had to make a quick and as-smooth-as-possible transition into e-learning in mid-March of 2020, almost exactly in the middle of its Spring semester.
The literature on COVID's impact on education results is still very limited because of the short observation period since the early months of 2020. Most of those publications use online venues to offer faster access to them as compared with the print journals. Although the online journals have enjoyed increasing popularity before the COVID pandemic, its outbreak appears to accelerate this process.
Concerns about the potentially negative impact of the pandemic on student learning are reflected in several publications. Some authors focus on policies introduced centrally for their countries. P. K. Jena (2020) lists highlights of measures taken by the government of India to provide seamless education in the country during the pandemic realizing also the fact that some corners of the country are not fully equipped with the digital platforms. The author sees the benefits of online education and suggests that it should be continued after the COVID lockdown. K. Naji et al. (2020) report their experience with the COVID-based transition for engineering students in Qatar, listing effective communication, proper pedagogical mode (e.g., student group projects), interactive mode of instruction, and availability of early assistance for students who struggle as the critical factors to overcome the problems with the online course transitions in response to COVID-19. Pradeep Sahu (2020) and P. Hiremath et al. (2020) address the impact of COVID on the mental health and economic conditions of the students. They list regular and timely information about the outbreak, availability of the proper counseling services, food and accommodation assistance for international students, and a helpful attitude of instructors as important factors in reducing the pandemic-related stress among students. Marcus L. George (2020) presents his experience with effective teaching and examination under the pandemic at the Advances in Applied Sociology University of West Indies. He lists an effective e-mail consultation and practices mock quizzes as ways to prepare students for an effective transition to a new learning model. The application of those strategies resulted in the student performance that was not worse than what was experienced in the past five years. However, he is not sure whether extensive student collaboration on the projects was also a factor.
M. Park et al. (2020) discuss their experience with engineering education under COVID focusing on disruptions affecting primarily courses with lab activities and related stress that may affect students' enrollment in those courses. J.
Qadir and A. Al-Fuqaha (2020) offer a primer for students who need to take courses under the pandemic times that are: "violate, uncertain, complex and ambiguous", offering adherence to seven steps students can take to make remote learning as effective as the traditional face-to-face instruction.
S. Asgari et al. (2020) present results of an extensive qualitative and qualitative study performed at SDSU's sister campus at the California State University at Long Beach, based on a survey of 110 faculty and 627 students in various Engineering fields. They discovered serious negative impacts of COVID-19 on issues related to learning and teaching as well as proper assessment methods affecting primarily vulnerable, disadvantaged, and underrepresented groups. They addressed numerous technical challenges facing students and instructors and proposed strategies to cope with those issues. They found out that the top challenge facing students taking remote courses was the difficulty in maintaining focus.
There have also been more opportunities for students to cheat on exams. Seventy

The SDSU Study
The SDSU study reported in this paper focused on Civil Engineering undergra-Advances in Applied Sociology duate students who took courses during the Spring semester 2020. Selection of this group of students was logical as the first author of this paper as Department Chair was able to administer an additional, mid-semester survey for the Civil Engineering students to mirror similar surveys the students answer routinely at the end of each semester. The intended study of the COVID impact on Civil Engineering students at San Diego State University had the following starting objectives: 1) to be related to the ABET accreditation procedures the program needs to comply with; 2) be limited in scope yet based on a large database; 3) as a pure before/after study, be unlikely to be biased by unaccounted factors, and 4) be suitable to statistical tests of significance. All those conditions appeared feasible when a sudden COVID-19 transition was to be implemented.
Each respectable Engineering program aspires to be accredited by the Accreditation Board of Engineering and Technology (ABET). ABET accreditation is valued highly as it is demanding and rigorous. It examines the fulfillment of nine major evaluation criteria by the program at hand. The key component of the ABET evaluation is the expectation that each program has procedures aimed at Continuous Improvement (CI). CI documentation needs to deliver evidence that the education process in any program relies on assessment and evaluation of student learning data to continuously improve the program. Evidence of improvement is typically quantitative and should be performed at 1) Individual course level; 2) Student Outcome (SO) level; 3) Program Educational Objective (PEO) level; and 4) Department Mission level.
Assessment of SOs is the heart of the ABET accreditation. There are two main kinds of assessment: direct and indirect. Direct assessment relies on instructors' own grading results of assignments designated to any given Student Outcome. The indirect assessment uses students' own perceptions about their learning. Although not as reliable as the direct assessment results, indirect evidence is still a useful tool in the continuous improvement paradigm.
One of such ABET-inspired indirect assessment procedures performed every semester at the Department of Civil, Construction, and Environmental Engineering at SDSU for each course is a survey that asks each student to assess the degree to which Student Outcomes assigned for that course were delivered as planned. The newest version of ABET rules specifically requires that all Engineering courses have to cover and evaluate student learning in seven well-defined Student Outcomes through graded assignments, tests, and/or projects. In addition, for various Engineering disciplines, additional Student Outcomes are defined by the respective professional engineering societies. Although those additional Student Outcomes are not formally required by ABET, they are enforced by ABET. In the case of Civil Engineering programs, the American Society of Civil Engineers is the organization defining those additional Student Outcomes for ABET.
The Civil Engineering program at SDSU currently requires compliance with 16 well-defined Student Outcomes. The first seven of them are specifically mandated by ABET for all Engineering programs; they were adopted without any Advances in Applied Sociology modification. The remaining nine were produced by the constituencies relevant to this process: faculty, Industrial Advisory Board members, and student representatives. The list of those SOs is presented in Appendix.
Coverage of those SOs was assigned to individual courses in a logical way. For example, SO 6 requiring performing experiments was assigned to courses with lab experimentation such as CivE 463, Geotechnical Engineering Lab. In order to have a robust set of data, it was decided that each course would need to cover at least three SOs, and each SO needs to be tested in at least two courses.
The data collecting procedure during the Spring semester 2020 was done in a consistent way: students were evaluating each course from the point of view of the coverage level of all SOs designated for that course on a scale from 1 to 5. On March 16, 2020, the Chancellor of the CSU system issued an executive order to switch all face-to-face classes to the remote instruction mode. Because of the incoming Spring Break, instructors and students were given just 10 days to adapt to the new teaching paradigm. It was a truly unprecedented and very challenging task for professors and students alike. There were adaptation issues to be sure.
However, the CCEE Department with a vast majority of technology-savvy professors handled that unique situation remarkably well.
At the time of that dramatic COVID-19-based transition, students were approximately halfway through the semester, and all courses have already had an opportunity to cover the respective portions of the SOs assigned to them. That created an opportunity to perform the indirect student assessment before the new remote mode was about to start. An identical second survey was performed online toward the end of the semester to capture potential differences in student perceptions about their SO-related learning.

Analysis and Results
The intention of the study was to capture the potential impact COVID-19 pandemic had on students' learning in the context of ABET-mandated Student Outcomes. The following null hypotheses were formulated: Ho #1: Coverage of SOi was not affected by the pandemic-related course mode transition.
Ho #2: SO-related instructor performance in course j was not affected by mode transition In addition, it is of interest to investigate whether courses that require hands-on lab experience were affected by the pandemic restrictions the same way as the lecture-based courses.
To analyze the statistical significance of the observed changes, the two-sided t-test was applied in order not to prejudge the direction (decrease or increase) of the changes in student scores.
Combined student score averages for sixteen SOs received in relevant courses before and after the COVID-transition are presented in Table 1. In addition, a number of students' responses in each case is presented. Clearly, the results are All COVID scores are lower than the pre-COVID ones. However, only four out of 16 (25%) score differences are statistically significant at the 5% significance level. Table 2 and Table 3 show the results of the COVID versus pre-COVID score comparisons for all contributing courses separately. Differences in student scores that were statistically significant at the 5% level (tcalc > 1.96) are labeled y; those not significant are labeled n; empty space indicates that a given course was not designated to deliver that specific SO. Letters y or n in italics indicate that the COVID period scores are actually higher rather than lower than the pre-COVID scores.
Generally, Hypothesis 1 cannot be rejected at the 5% level of significance for 12 out of 16 Student Outcomes analyzed. Student scores lowering under the COVID period was observed for all 16 student Outcomes but was significant in only four of them.
Also, Hypothesis 2 cannot be rejected at the 5% level of significance in 11 out of 18 Civil Engineering courses, showing again a rather limited impact of COVID on the students' assessment of SO coverage in individual courses.  Table 3. T-statistic significance summary of scores of student outcome coverage in civil engineering courses J through S at SDSU: Pre-COVID versus COVID periods.  Examination of Tables 1-3 leads to the following, more detailed comments: 1) The COVID-19-mandated sudden transition of the teaching mode from face-to-face (F2F) to online (OL) in mid-March 2020 did affect perceptions of Civil Engineering students at SDSU about the effectiveness of the coverage of the ABET-related Student Outcomes in courses offered to them during the Spring semester 2020. However, only 17 of the 80 changes in Student Outcome scores analyzed (21.3%) were statistically significant at the 5% level. This would suggest that the overall impact of COVID-19 on student's opinions about their learning experience during Spring 2020 was somewhat limited as the COVID scores for the vast majority courses of were high by a standard adopted by the CCEE department: they were typically higher than the 4.0/5.0 which is well above the 3.5/5.0 acceptance level. In fact, while the lowest overall score for any SO in the pre-COVID survey was 4.38/5.00, the lowest score for the COVID period survey was 4.25/5.00.
2) Out of the 80 cases analyzed, 62 (77.5%) of the COVID scores for Student Outcomes coverage were lower on average than the comparable pre-COVID scores for the same Student Outcomes. Three cases (3.8%) reported no change in scores (t = 0), and 15 cases (18.7%) reported the COVID scores to be actually higher than the symmetrical pre-COVID scores. Thus, the COVID impact on students' perception of their learning was not uniformly negative. None of the positive changes was statistically significant at the 5% level.
3) Twelve out of those 15 positive changes were found in CIVE 495 Senior Design course. That course is responsible for covering 13 out of 16 adopted Student Outcomes. Contrary to some other Engineering disciplines, Civil Engineering students do not produce a fabricated model or similar. It appears that the ZOOM format for their group interactions created a disciplined, working model for their designs.
4) In four out 16 Student Outcomes (25.0%), student scores drops were statistically significant. As expected, most negatively affected by COVID were SO 6 and SO 11 that expect students to "develop and conduct experiments". The remote versions of courses that normally offer a "hands-on" experience were unable to deliver those Outcomes fully.

5) Student
Outcome 5 "an ability to function effectively on a team…" appears to be also negatively affected by the COVID restrictions for ad hoc team assignments. The Senior Design is a noted exception here as teams are created even before the projects start and are formed in such a way as to have all the necessary expertise for the chosen project. Interactions among team members are indispensable and frequent. It appears that these interactions when done remotely via pre-scheduled ZOOM meetings were actually more efficient than face-to-face contacts that are more affected by outside factors such as traffic jams or similar.
6) Statistically significant scores drops were also found in SO 10 "ability to analyze and solve problems in at least four technical areas.

General Conclusions
According to students' scores, main accreditation expectations in the form of the mandated Student Outcomes were met satisfactorily not only before the COVID-19 restrictions but also under the COVID-19 restrictions. In both cases, the average student scores for all SOs were above 4.0/5.0. Only in 21% of cases, the COVID-caused transition led to scoring lowering that was statistically significant at the 5% level.
Student Outcomes that emphasize lab experience and teamwork were the ones most negatively affected by the transition from the face-to-face to remote instruction mode. It is not clear whether the drop in the student scores represented their perception that that new mode of instruction was ultimately inferior or because the transition to the new mode of instruction was so dramatic and abrupt.
COVID's impact on the performance of individual instructors in delivering the mandated Student Outcomes varied. A majority of instructors effectively used the Spring Break week to adjust their course instruction to the new mode of instruction and was able to continue quality teaching. The most difficult and demanding adjustments were in the courses with significant lab components where the drops in students' scores were statistically significant at the 5% level.
Judging by the SDSU case, COVID-19 will likely lead to changes in the way Engineering courses are to be taught in the future as several instructors originally "forced" to teach in the remote mode indicate now that they want to continue their online or hybrid teaching even after the COVID pandemic is finally over.
New models for teaching some critical laboratory classes such as blending or flipped learning that appear to work satisfactorily at some universities are likely to emerge.
In summary, the course assessments made by the Civil Engineering students at San Diego State University suggest that the negative impact of COVID-19 restrictions on student learning was noticeable but lower than expected and in most cases not significant statistically. Both instructors and students adapted remarkably well to the rapid transition of all courses to the remote mode. De-