Journal of Software Engineering and Applications, 2013, 6, 571-581
Published Online November 2013 (http://www.scirp.org/journal/jsea)
http://dx.doi.org/10.4236/jsea.2013.611069
Open Access JSEA
571
Standardization of Quality Evaluation of Educational
Software and Electronic Learning Tools—Analysis of
Opinions of Selected Experts*
Štefan Karolčík, Elena Čipková, Milan Veselský, Helena Hrubišková
Department of Didactics in Science, Psychology and Pedagogy, Faculty of Natural Sciences, Comenius University, Bratislava,
Slovak Republic.
Received September 27th, 2013; revised October 24th, 2013; accepted October 31st, 2013
Copyright © 2013 Štefan Karolčík et al. This is an open access article distributed under the Creative Commons Attribution License,
which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
ABSTRACT
Our research was focused on the identification of features, which was essential for educational digital products and the
determination of their quality. The introductory analytical part of our research is focused on the analysis of existing
sources of information related to the problems of research, production, appropriate use and evaluation of educational
software environments. Consequently, we have divided the existing software products into three basic groups according
to our main distinguishing feature. Second part of our paper is focused on various aspects, which are to be considered
when assessing the quality of software solutions. The final part contains the presentation of results of our findings re-
lated to the most important features expected and required from digital learning tools by professional experts and spe-
cialists in given field.
Keywords: Educational Technologies; Learning Tools; Evaluation Tools
1. Introduction
Reliable and in-depth quality evaluation of electronic
learning tools presupposes the existence of specific
evaluation tools, which are able to evaluate the specific
content and functionality of software applications with
educational ambitions. Only the clearly defined, impar-
tial, professionally correct and understandable criteria
can distinguish a high quality educational product from
other products—less appropriate or even socially dan-
gerous electronic didactic tools. In spite of the fact that
educational technologies influence the current educa-
tional process, professional and methodically elaborated
assessment of their quality has not been a subject of ma-
jor research projects by now. One reason is the excep-
tional dynamics of technologies development and con-
stantly expanding list of new devices and their functions.
It disables the standardization of evaluation related ex-
clusively to one specific software product or device.
Which features characterize a modern and attractive
digital learning tool? Which is useful for educational
practice? What distinguishes it from other less appropri-
ate software applications? Is it possible to measure the
quality of any software solution by existing tools? Efforts
to obtain relevant answers to these questions prompted us
to carry out this research.
2. Educational Technologies and Their
Categorization
Determination of clear criteria, by which we could dis-
tinguish the educational (pedagogical, academic) soft-
ware, learning environment or more comprehensive in-
formation system from other digital products, is an ex-
tremely demanding task. From the point of view of
pedagogical practice, this is a critical issue. When defin-
ing the terms associated with the use of digital technolo-
gies in the teaching process, most of the authors focuses
on the pedagogical character of a product, which was
designed, created, tested in practice and further devel-
oped especially for educational purposes—as a tool for
teaching and learning [4,6,14,17,20,27]. In this sense we
understand the educational technologies as technological
solutions, tools and devices designed specifically for the
*This work was supported by the Slovak Research and Development
Agency under the contract No. APVV-0266-11.

Standardization of Quality Evaluation of Educational Software and Electronic
Learning Tools—Analysis of Opinions of Selected Experts
572
needs of education and various educational and training
activities. When studying the problems of introduction of
the digital technologies into the educational process more
deeply, we can also find the expressions considering any
strict delimitation of a set of educational software as
dangerous and incorrect. These attitudes are largely
based on the premise, that any software becomes really
educational only at the moment of its application in the
educational process, or learning process, it means by the
way of its usage [12,17,21,29]. Taking into account these
opinions, it is necessary to enlarge the group of educa-
tional technologies with specific software products,
which in spite of the fact, that they were not specifically
designed to support the pedagogical process, they have
naturally found its application and became popular
among teachers or students.
A huge range and variety of educational software ap-
plications and interactive educational environments dis-
ables the introduction of uniform templates for evalua-
tion of their quality. It is necessary to define the basic
criterion/criteria for the categorization of existing soft-
ware solutions and to evaluate all the products within one
defined group. In addition there are several software
classifications, and at the highest level we can divide the
software products into free products (free of charge,
shareware, they can be copied) and proprietary products
(their usage is conditioned by the purchase of license;
they cannot be modified or copied). Other divisions de-
pend on the selection of criteria. Educational software
can be divided by the way of presentation of processed
information, role of the computer in the educational
process, learning subject for which it is primarily in-
tended, by educational paradigm, cognitive process phase
of a learning person, for whom the software is primarily
intended, by the software’s main function, recipient’s age,
way of work etc. [1,3,20]. If we use a primary purpose of
usage as a main criterion, most of the educational soft-
ware environments used in schools is intended for prac-
tising and studying of curriculum. Their popularity
emerges from the fact, that they help teachers save time,
facilitate the preparation of teaching aids (interactive
tests, quizzes a exercises) and simplify the organization
of learning process (individual and group work of stu-
dents). The evaluation of results is accurate, fast and
automatic. Reverse side of their frequent application in
the educational process are repetitive activities and ques-
tions focused most frequently on the knowledge of facts.
When designing the propositions of standards for
quality measuring of software with educational usage, we
can start from the basic division of software solutions
into three main groups: complex learning environments;
instructive programs and Applets and finally software
modules and accessories. As a main distinguishing fea-
ture, we have defined the extent of provided and elec-
tronically processed information and the associated range
of application possibilities of usage of products in the
educational process. A separate category consists of
software tools for management, administration and sup-
port of learning activities—LMS (learning management
system), video conference systems, presentation programs
and other unilaterally focused and highly specified digital
teaching aids, e.g. interactive worksheets, crosswords,
gap-fills, electronic forms etc. Into this category we can
also add the web portals with video lectures, which can-
not be changed and which are very often a part of e-
learning courses and lessons.
Complex learning environments contain robust web
portals with a large database of educational materials
(objects) for a number of learning subjects and classes,
and also tools, which simplify the work with them
(searching, authentication, preparation of students/teach-
ers, education management, creation of individual tasks,
interactive tasks, methodical guidelines, recommended
procedures, worksheets, automated monitoring of student
results etc.). Technical support and continuous updates of
the electronic content is a matter of course.
Instructive programs are specific software solutions,
which partially replace a teacher in the interpretation of
new curriculum, its practising and automatic evaluation
of a level of student skills and knowledge. The electronic
content processed by the program is usually limited to
one specific discipline, thematic unit or teaching subject.
Typical instructive programs include various comer-
cially sold learning CD and DVD media with installa-
tion .exe files (Terasoft, LANGMaster, SILCOM Multi-
media...).
Applets, software modules and accessories represent
the smallest program units, which can be launched sepa-
rately or as plugins of web pages. Their didactic usage is
limited mainly to the mathematical calculations in more
complex tasks, addition of missing data, explanation of
problems or simulation of phenomena or processes. The
interactive presentation mainly consists of animations
supplemented with graphical and sound effects. Their
progress can be influenced, e.g. by the change of the in-
put quantities. There are also quite well-known Java ap-
plets focused on the explanation of mathematical and
physical relations (GeoGebra, FyzWeb-applets...).
A special group of software products designed to sup-
port the distance forms of education is so called LMS
(Learning management system). It is an e-learning envi-
ronment intended for creation, edition and management
of educational projects (lessons, courses). Its application
into the educational process requires not only a good
knowledge of chosen environment, but also the complete
preparation of the electronic content. This is not a part of
Open Access JSEA

Standardization of Quality Evaluation of Educational Software and Electronic
Learning Tools—Analysis of Opinions of Selected Experts 573
an installation package and its quality depends exclu-
sively on skills and experiences of the teacher. The most
widely used system in Slovakia is LMS Moodle. Other
used systems are LMS Claroline, Microsoft Class Server
or Webcity.
Video conference systems also do not provide the me-
thodically elaborated electronic content. Only the topic
of a virtual meeting, discussion character and ways of the
provision of the relevant data can transform a video con-
ference to the highly effective education tool. Video
conference is actually a web-oriented environment,
which enables the mutual interconnection of several par-
ticipants in one video conference room. Voice and image
transfer is supplemented with possibility of writing the
text messages (chat) and presentation of multimedia data
files. The high-quality products enable to share the run-
ning applications and sophisticated tools of a team work.
E. g. a widely used system EVO received a large support
in the scientific circles. Its commercially sold alternative
Adobe Connect Pro offers a simple access to the video
conference.
Presentation programs conclude a special group of
software programs, which do not provide the users with
materials of the educational nature. However, they help
teachers and students present the information available in
various digital formats in a very interesting form. Thanks
to the programs the outputs have a multimedia character
with interactive features. In this way they become dy-
namic and visually attractive for students. An example of
a typical presentation program in schools is the most
widely used Microsoft PowerPoint. Good competition
represent programs like Prezi, Adobe Captivate, Asham-
poo Presentations and programs intended for the creation
of concept maps—Xmind, MindMap, SmartDraw etc.
3. Criteria for the Quality Evaluation of
Educational Software and Electronic
Educational Tools
Despite the fact that digital technologies are more and
more applied in the educational and learning process,
selection of an appropriate (professionally and didacti-
cally correct) software is becoming an increasingly de-
manding pedagogical problem—there is no thorough re-
search in this field. Most of the published papers are fo-
cused on the assessment of specific technological solu-
tions (e.g. selected Open Source Software, LMS, CAL
software, Zooming Editor Software, Electronic mobile
devices, Cloud Learning, Cloud Computing etc.) and
determination of their learning potential (see e.g. [2,5,15,
33]). There are many results presented, which analyze
the effectiveness of application of selected digital tech-
nologies in a specific learning situation and specific
educational space (see e.g. [16,48]). The research pro-
jects considering all the general principles of creation
and objectivity of quality evaluation of modern electronic
educational tools are, however, missing. In order to
evaluate the quality of the educational software and elec-
tronic learning materials, it is necessary to consider sev-
eral aspects. From the point of view of the teacher and
education, psychological-didactic and professional aspect
of the software is extremely important. The evaluation
from this point of view should include the fact how the
software supports the management of education and
evaluation of students. Equally important are the soft-
ware technical parameters and user perspective on the
evaluated digital product taking into account its individ-
ual peculiarities.
Educational aspect assesses the software value as a
means of support of student various cognitive processes
within the meaningful learning process and also the ex-
tent, to which it contributes to their involvement in the
learning process and to stimulation of their cognitive
interests. The software should particularly respect the
basic relations of perception, memory and thinking. From
this perspective it is, for example, important to evaluate
the clearness and intensity of incentives, mutual arrange-
ment and use of various classes of multimedia operating
objects, text, graphics (including animations and video)
and sound. For efficient perception, understanding and
learning it is important to simultaneously represent both
basic modalities of provided information—it means vis-
ual and verbal. To avoid overloading the attention and
memory of a learning person, it is necessary to avoid the
division of students’ attention to overlapping sources.
Educational objects should also support cognitive proc-
esses of selection, organization and integration. Illustra-
tion of hierarchy and reciprocality between the blocks of
information helps better understand their organization
(principle of hierarchy). Organization of information
helps learning person create the adequate mental repre-
sentations (see e.g. [24]).
From an educational point of view, it is essential to
find out a purpose, validity, content complexity and ade-
quacy, technical correctness of provided information,
their compliance with curriculum, and also a range of
possibilities of the software use in the educational proc-
ess, motivation impulses and activities created by the
software, provided feedback tools, as well as multimedia
variety of the electronic content and preferred style of
teachers and students work with the software. Attention
should be paid to the fact, whether the software is able to
take into account the variability of students, which is
determined by their various abilities, cognitive style or
preferred learning style, or by their handicap (see e.g.
[22,35]). Another evaluated issue is also the interactivity
of a software work, which can be understood as an ability
Open Access JSEA

Standardization of Quality Evaluation of Educational Software and Electronic
Learning Tools—Analysis of Opinions of Selected Experts
574
to influence the behaviour and the course of the events in
dependence on user’s requirements, and its openness, it
means to what extent it is possible to change the default
software environment and to enter its database (possibil-
ity of adding own objects, of editing already created ob-
jects etc.).
User’s point of view pays attention mainly to the fea-
tures of user environment also known as graphical user
interface (GUI). An important thing is an overall clarity
of a symbolic language and tools used by the software
when communicating with user, as well as the transpar-
ency of drivers’ location. We evaluate the software’s
control, various settings options, appearance, quality of
multimedia presented, overall stability (resistance to in-
correct (unsupported) interventions, localization, soft-
ware price, as well as the availability of trial version be-
fore purchase and use of current technological trends.
Another advantage can also be an obligation of the soft-
ware producers to upgrade, extend and update the origi-
nal product and its compatibility with other products.
The learning process is also connected with teacher’s
administrative activities, e.g. students data archiving,
creation of various statistical summaries and final evalua-
tions. Therefore it is important to find out how the edu-
cation software helps teachers in the education manage-
ment, students records and evaluation. We are monitor-
ing the software performance, capacity, support of import,
export, backup and archiving of student records, the abil-
ity to record the structured data about students (classes),
to manage and control the access to information sources,
provision of possibility to evaluate the partial perform-
ance, monitor students progress, create the personal stu-
dent reports, summarizing tables, statistics, schemes and
graphs.
The real software application also depends on its tech-
nical parameters (technical aspect). The subject of eva-
luation should be in this case the software compatibility
(start-up under various operating systems and their ver-
sions), its hardware requirements, installation method,
technical support, security and level of protection against
unauthorized attacks.
Specific objectives and various methods of use of in-
dividual types of educational technologies require vari-
ous approaches and evaluation criteria, which take into
account their particularities. Regarding the instructive
programs intended for the curriculum practising, the es-
sential fact is whether they provide various levels of
complexity of tasks or issues, whether the program set-
tings enable to determine their number, order or varia-
tions in the test, as well as the method by which they will
be managed (options of restart, variation of time of the
response, evaluation of answers correctness in blocks or
by individual test items etc.). Concerning the simulation
programs, the important thing is whether they allow the
user to make decisions on every critical step, whether
they provide realistic and credible consequences (effects)
of selected actions and to what extent the factors facili-
tating the proper understanding of a simulated pheno-
menon or process are highlighted.
4. Research Methodology
For the purpose of determining the users’ expectations
and requirements in relation to the digital learning tools,
we realized a questionnaire survey. We used the ques-
tionnaires of own design with only open items. Since it is
a very specific field of research (educational technolo-
gies), only a relatively small sample of respondents was
included–it means those who have met the requirements
of experts or professional users of digital technologies.
We have selected the Delphi method as a basis of the
methodology. The questionnaires were provided inten-
tionally to 3 selected groups of surveyed experts regu-
larly using the digital technologies in everyday work or
personal life. The experts gave their opinions on 3 areas
of problems related to the most important expected fea-
tures of software applications and electronic materials
intended and created specifically for the needs of educa-
tion and learning process support. We have addressed
several experts and the following ones have participated
in the research: 12 specialists on information technolo-
gies, 11 teachers, who have been experimenting with
application of digital technologies into the learning proc-
ess in the long term and 12 respondents—users of infor-
mation technologies, who are regularly using various
digital tools and technologies in work. In order to ensure
the respondents a quick and simple access to the ques-
tionnaires, we have published them on our website:
for IT specialists:
http://www.evaluedu.sk /sk/home/formular-it-specialista,
for users:
http://www.evaluedu.sk /sk/home/formulare/formular-pou
zivatel/
for teachers:
http://www.evaluedu.sk /sk/home/formulare/formular-ucit
el.
5. Research Results
5.1. IT Specialists Evaluation
When taking into account all the selections of features
made by IT specialists during the evaluation of the im-
portance of the educational software features for its users,
the following features occurred most frequently: clarity
and simplicity, graphical processing, design variety, pos-
sibility to access the software through the internet, clear-
ness expressed by illustrative examples, videos or pic-
Open Access JSEA

Standardization of Quality Evaluation of Educational Software and Electronic
Learning Tools—Analysis of Opinions of Selected Experts 575
tures, software interactivity and also its attractiveness for
students-motivational function. The specific frequencies
of selected features are stated in the Table 1.
Table 1. Frequencies of answers of IT specialists to item 1:
Try to identify at least 5 most important features of educa-
tional software (electronic learning material), which you as
a user would expect from this learning tool.
Feature Frequency Frequency %
clarity/simplicity/user friendly 9 75.0
graphical processing/design variety 6 50.0
possibility to access the software
through the internet 5 41.7
clearness with illustrative examples,
animations, pictures 3 25.0
interactivity 3 25.0
motivational/attractive 3 25.0
completeness of content with
multimedia elements 2 16.7
intuitiveness 2 16.7
compatibility/universality for all OS 2 16.7
factuality/sim plicity of the text 2 16.7
perfection of data and
information/proficiency 2 16.7
meets a learning
objective/meaningfulness 2 16.7
contains a number of tasks for practising 2 16.7
Undo function 2 16.7
ergonomics 2 16.7
enables further work with information 2 16.7
free/open 1 8.3
Slovak language
mutation/comprehensibility 1 8.3
simple and quick installation 1 8.3
quick searching of topics and
information/quick feedback 1 8.3
intersubject relations/interconnection1 8.3
interestingly formulated text 1 8.3
possibility to deal with
multiple functions 1 8.3
up-to-dateness 1 8.3
possibility to create the simple outputs 1 8.3
contains the interactive tasks 1 8.3
possibility to download the tasks 1 8.3
didactically consistent 1 8.3
Total 12 100.0
An interesting fact is that one third of respondents
stated on the first position the software clarity and sim-
plicity, what can be freely interpreted as an expression of
preference of these features. The answers of other re-
spondents related to the features stated on this position
differed.
The subject of our research was also focused on fea-
tures of the educational software, which are important for
the education manager, it means for teacher, instructor,
lecturer etc. Most often, the IT specialists considered
such features to be clarity, simplicity and comprehensi-
bility, then features related to free sharing and software
financial affordability, then features like testing of stu-
dents knowledge, intermediation of results of student
tests in a structured form, meeting the requirements for
graphical processing and design of presented information,
focus on essential content and forms of the curriculum,
intermediation of curriculum in a form, which presup-
poses its meaningful learning and remembering, then up-
to-dateness, overall software attractiveness for students,
options of editing of intermediated content and compati-
bility of software with various operating systems and
devices.
Regarding the item No. 1: Try to identify at least 5
most important features of educational software (elec-
tronic learning material), which you as a user would ex-
pect from this learning tool, 33.3% of respondents stated
the features clarity and simplicity in the first position,
other responses occurred in this position only once.
Taking into account all the selected features regardless
of order, the most frequently reported features were clar-
ity and simplicity, followed by graphical processing, de-
sign variety, then option of accessing the software
through the internet, clearness expressed by illustrative
examples, videos or pictures, software interactivity and
finally its attractiveness for students-motivational func-
tion.
Item No. 2 revealed the similar findings: Try to iden-
tify at least 5 most important features of educational
software (electronic learning material), which you as an
education manager would expect from this learning tool
(instructor, teacher, lecturer etc.).
The most important feature of the educational software
stated by the respondents was clarity and simplicity. This
was stated by 27.3% of respondents. Other features were
listed as preferred only once.
In general, the most frequently occurring features were
again clarity, simplicity and comprehensibility, then fea-
tures related to free sharing and software financial af-
fordability, then features like testing of students’ knowl-
edge, intermediation of results of student tests in a struc-
tured form, meeting the requirements for graphical proc-
essing and design of presented information, focus on
Open Access JSEA

Standardization of Quality Evaluation of Educational Software and Electronic
Learning Tools—Analysis of Opinions of Selected Experts
576
essential content and forms of the curriculum, interme-
diation of curriculum in a form, which presupposes its
meaningful learning and remembering, then up-to-date-
ness, overall software attractiveness for students, options
of editing of intermediated content and compatibility of
software with various operating systems and devices.
Item No. 3: Try to identify at least 5 most important
technical features of educational software (electronic
learning material), which you as a PC classroom admin-
istrator (technician) would expect from this tool. This
item revealed the following results:
First features mentioned most frequently were re-
quirements for simplicity of control, start-up and installa-
tion; remaining responses were mentioned only once.
In general, the most frequently mentioned feature ex-
pected by IT specialists was simple administration of
users and their accounts, followed by requirements for
simplicity of control, installation and start-up of the
software, then its compatibility with various operating
systems and devices, easy upgrade, update, protection
against hackers, system security against the attacks from
the outside, minimalization of a need for service calls
and finally financial affordability.
A group of IT specialists considers a new educational
software as a well-arranged and user-friendly educational
tool, which is financially affordable for schools, teachers
as well as for students. They expect from the software to
be graphically interesting and attractive for students, with
motivational features and examples using various multi-
media tools. The software should be able to intermediate
the curriculum on the professional level in a form, which
would enable teachers to meet the educational objectives
and help students facilitate the meaningful process of
learning. It should be compatible with various operating
systems and multiple devices such as tablet, iPad, inter-
active table etc. PC classroom administrator should dis-
pose of simple user’s administration and easy software
upgrade with a minimum number of service calls. The
software should by stable with sufficient protection
against hackers.
5.2. Teachers Evaluation
A group of teachers involved in the research consisted of
teachers who are very well oriented in given field of
digital technologies and who are regularly using the
available technological solutions in their teaching prac-
tice.
Regarding the item No 1: Try to identify at least 5
most important features of educational software (elec-
tronic learning material), which you as a user would ex-
pect from this learning tool, 36.4% of respondents chose
clarity and simplicity as a first feature, followed by Slo-
vak language mutation, comprehensibility (27.3%) and
finally completeness of content with multimedia ele-
ments (18.2%).
Taking into account all the mentioned features regard-
less of order, the most frequently reported features were
clarity and simplicity, followed by completeness of con-
tent with multimedia elements, interactivity, Slovak lan-
guage mutation, comprehensibility and graphical proc-
essing, design variety. The specific frequencies of se-
lected features are stated in the Table 2.
Regarding the item No. 2: Try to identify at least 5
Table 2. Responses of teachers to item 1: Try to identify at
least 5 most important features of educational software
(electronic learning material), which you as a user would
expect from this learning tool.
Feature Frequency Frequency %
clarity/simplicity/user friendly 9 81.8
completeness of content with
multimedia elements 5 45.5
interactivity 5 45.5
Slovak language
mutation/comprehensibility 4 36.4
graphical processing/design variety 4 36.4
shareware/financial affordability 3 27.3
clearness with illustrative examples,
animations, pictures 3 27.3
quick searching of topics and
information/auick feedback 3 27.3
compatibility/universality for all OS 2 18.2
factuality/simplicity of the text 2 18.2
free/open 1 9.1
localization 1 9.1
suitability for students
considering their age 1 9.1
simple and quick installation 1 9.1
intuitiveness 1 9.1
up-to-dateness 1 9.1
intersubject relations/interconnection 1 9.1
consistent preparation for
practising/verifying of knowledge 1 9.1
perfection of data and
information/proficiency 1 9.1
interestingly formulated text 1 9.1
possibility to deal with
multiple functions 1 9.1
information about experiments 1 9.1
Total 11 100.0
Open Access JSEA

Standardization of Quality Evaluation of Educational Software and Electronic
Learning Tools—Analysis of Opinions of Selected Experts 577
most important features of educational software (elec-
tronic learning material), which would help you in the
management (organization) of the educational process;
the respondents stated different first features and opin-
ions on the intuitively most important feature differed.
In general, the most frequently occurring feature of the
educational software was clarity, simplicity, comprehen-
sibility and interactivity. These were followed by fea-
tures like multimedia character, possibility to keep re-
cords of students work, appropriate graphical processing,
design and clearness with illustrative examples, anima-
tions and pictures. Furthermore, from high quality soft-
ware the teachers expect that it will be a shareware pro-
gram, financially affordable and it will provide several
difficulty levels and function “Undo”.
Regarding the item No. 3: Try to identify at least 5
most important technical features of educational software
(electronic learning material), including its formal proc-
essing, which you as a teacher would expect from this
tool, the opinions of respondents on the feature men-
tioned as a first one, were for each respondent different.
The most frequently mentioned features were an inter-
esting design of user interface and its attractive graphical
processing. These were followed by up-to-dateness and
compliance of content with curriculum and textbooks.
Another features expected by teachers from the software
are financial affordability, content completeness, simple
control, start-up and installation. Segmentation of the text
should be well-arranged with logical interconnection of
related topics and use of interactive elements.
A group of innovative teachers, who are regularly us-
ing the digital technologies in their learning process fo-
cuses on the clarity and simplicity of educational soft-
ware, not only on the control, but also on the start-up and
orientation in it, logically interconnected topics with
complete up-to-date content using the multimedia tools.
The software should be graphically interesting with sev-
eral difficulty levels and financially affordable for teach-
ers as well as for students. The teacher should have the
possibility to keep records of students work and interac-
tively use all the existing electronic media formats, e.g.
animations, pictures and videos. An important feature is
Slovak language mutation of the software.
5.3. Users Evaluation
Participated users daily use the available digital tech-
nologies in their professional work and in this regard
may be considered as sufficiently informed and experi-
enced. Their opinions on the features of electronic edu-
cational aids are therefore considered to be qualified.
Regarding the item No. 1: Try to identify at least 5
most important features of educational software (elec-
tronic learning material), which you as a user would ex-
pect from this learning tool, 25% of respondents stated
clarity and simplicity as a first feature, perfection of data
and proficiency occurred less frequently (16.7%). Other
responses occurred only once.
When evaluating all the mentioned features regardless
of order, the most frequently reported features were clar-
ity and simplicity as well as perfection of data and in-
formation and proficiency. These were followed by fea-
tures like suitability for students considering their age,
clearness with illustrative examples, animations, pictures,
interesting graphical processing, design variety, up-to-
dateness and ability to activate a user. The specific fre-
quencies of selected features are stated in the Table 3.
Other results were revealed by a question No. 2: Try to
identify at least 5 most important features of presented
content of educational software (electronic learning ma-
terial), including its formal processing, which you as a
user would expect from this learning tool. The respon-
dents considered proficiency and quality to be the most
important features in this case. This was stated by 33.3%
of respondents.
In general, the most frequently occurring responses
were proficiency and quality followed by focus on stu-
dents and clearness with illustrative examples, anima-
tions and pictures. From an electronic learning tool the
users expect multimedia character, clarity, simplicity,
comprehensibility and meaningful content. Segmentation
of the text, graphical processing and design of a tool
should be appropriate for age of the student and should
be focused on what it is supposed to teach.
Item No. 3: Try to identify at least 5 most important
technical features of educational software (electronic
learning material), which you as a user would expect
from such product, revealed the following results:
First most frequently occurring response was simplic-
ity of control, start-up and installation, together with
perfect functionality and compatibility with various op-
erating systems.
In general, the most frequently mentioned feature was
simplicity of control, start-up and installation together
with compatibility with various operating systems. Fur-
ther features were undemanding requirement for hard-
ware and possibility of creation of various outputs. The
remaining responses were mentioned 2 times or only
once.
A group of experts actively using the digital technolo-
gies would appreciate a perfect and professional educa-
tional software with simple control, start-up and installa-
tion, which offers—with regard to age of the student—il-
lustrative examples with animations, pictures and videos,
and which is graphically interesting and attractive, com-
patible with various operating systems, as well as with
devices like tablet, iPad, interactive table etc. The users
Open Access JSEA

Standardization of Quality Evaluation of Educational Software and Electronic
Learning Tools—Analysis of Opinions of Selected Experts
578
Table 3. Responses of users to item 1: Try to identify at
least 5 most important features of educational software
(electronic learning material), which you as a user would
expect from this learning tool.
Feature Frequency Frequency %
clarity/simplicity/user friendly 6 50.0
perfection of data and
information/proficiency 6 50.0
suitability for students considering
their age 5 41.7
clearness with illustrative exam ples,
animations, pictures 4 33.3
graphical processing/design variety4 33.3
up-to-dateness 4 33.3
ability to activate a user 4 33.3
free/open 3 25.0
shareware/financial affordability 3 25.0
interactivity 3 25.0
meets a learning
ob_iective/meaningfulness 2 16.7
up-to-dateness 2 16.7
motivational/attractive 2 16.7
Slovak language
mutation/comprehensibility 1 8.3
completeness of content with
multimedia elements 1 8.3
factuality/simplicity of the text 1 8.3
intersubject relations/interconnection1 8.3
consistent preparation for
practising/verifying of knowledge 1 8.3
occupies a small capacity 1 8.3
efficiency/help teachers better
organize their work 1 8.3
possibility to access the software
through the internet 1 8.3
possibility to combine it with another
educational software 1 8.3
various difficulty levels 1 8.3
portable 1 8.3
contains a number of tasks
for practising 1 8.3
constructive learning process 1 8.3
ecological 1 8.3
possibility to create the simple outputs1 8.3
update options 1 8.3
Total 12 100.0
also emphasize a well-arranged segmentation of the text,
its meaningfulness as well as the possibility to create
various kinds of outputs. According to their opinion, the
software should not require a demanding hardware, so it
will not be necessary to buy new devices and accessories
for PC.
5.4. Results Summary
The item No. 1: Try to identify at least 5 most important
features of educational software (electronic learning ma-
terial), which you as a user would expect from this learn-
ing tool, was answered by all 3 groups—a total of 35
respondents.
The most important feature stated by respondents in
the first position was clarity and simplicity. This was
mentioned by 31.4% of respondents.
Taking into account all the mentioned features regard-
less of order, the most frequently reported features were
clarity and simplicity, followed by graphical processing,
design variety, interactivity, clearness with illustrative
examples, videos and pictures, perfection of presented
data and information, proficiency and completeness of
content with multimedia elements. The specific frequen-
cies of selected features are stated in the Table 4.
Item No. 2: Try to identify at least 5 most important
features of educational software (electronic learning ma-
terial), which you as an education manager would expect
from this learning tool (instructor, teacher, lecturer etc.),
or which would help you in the management of the edu-
cational process, was formulated identically for the group
of IT specialists and teachers. This group consisted of 23
respondents.
According to the respondents, the most important fea-
ture is clarity and simplicity. It was stated by 14.3% of
respondents.
In general, the most frequently occurring features were
clarity, simplicity and comprehensibility. These were
followed by interactivity, requirement for a free sharing
and financial affordability of the software, high-quality
graphical processing, design and possibility to keep re-
cords of students work.
Item No. 3: Try to identify at least 5 most important
technical features of educational software (electronic
learning material), which you would expect from this
tool, was identical for the group of IT specialists and
users. This group consisted of 24 experts.
First most frequently mentioned responses were sim-
plicity of control, start-up and installation, which was
started by 27.3% of respondents.
In general, the most frequently mentioned features
were simplicity of control, start-up and installation and
compatibility with various operating systems and devices,
followed by simple users’ administration, undemanding
Open Access JSEA

Standardization of Quality Evaluation of Educational Software and Electronic
Learning Tools—Analysis of Opinions of Selected Experts
Open Access JSEA
579
Table 4. Responses of respondents to item 1: Try to identify at least 5 most important features of educational software (elec-
tronic learning material), which you as a user would expect from this learning tool.
Features Frequency Frequency %
clarity/simplicity/user friendly 24 68.6
graphical processing/design variety 14 40.0
interactivity 11 31.4
clearness with illustrative examples, animations, pictures 10 28.6
perfection of data and information/proficiency 9 25.7
completeness of content with multimedia elements 8 22.9
Slovak language mutation/comprehensibility 6 17.1
shareware/financial affordability 6 17.1
suitability for students considering their age 6 17.1
possibility to access the software through the internet 6 17.1
free/open 5 14.3
factuality/simplicity of the text 5 14.3
up-to-dateness 5 14.3
motivational/attractive 5 14.3
quick searching of topics and information/quick feedback 4 11.4
compatibility/universality for all OS 4 11.4
meets a learning objective/meaningfulness 4 11.4
ability to activate a user 4 11.4
intuitiveness 3 8.6
intersubject relations/interconnection 3 8.6
up-to-dateness 3 8.6
contains a number of tasks for practisin g 3 8.6
simple and quick installation 2 5.7
consistent preparation for practising/verifying of knowledge 2 5.7
interestingly formulated text 2 5.7
possibility to deal with multiple functions 2 5.7
possibility to create the simple outputs 2 5.7
Undo function 2 5.7
ergonomics 2 5.7
enables further work with information 2 5.7
localization 1 2.9
information about experiments 1 2.9
occupies a small capacity 1 2.9
efficiency/help teachers better organize their work 1 2.9
possibility to combine it with another educational software 1 2.9
various difficulty levels 1 2.9
portable 1 2.9
constructive learning process 1 2.9
ecological 1 2.9
update options 1 2.9
contains the interactive tasks 1 2.9
possibility to download the tasks 1 2.9
didactically consistent 1 2.9
Total 35 100.0

Standardization of Quality Evaluation of Educational Software and Electronic
Learning Tools—Analysis of Opinions of Selected Experts
580
requirement for hardware, protection against hackers and
overall product security.
In regard to individual participated groups of respon-
dents, we found out a significant similarity in evaluating
the importance of features of educational software or
educational learning tools. IT specialists, teachers and
qualified IT users consider the basic and most important
features of these tools to be simplicity and clarity in in-
stallation, start-up and orientation, then interesting and
attractive graphical interface containing the elaborated
technical and visual elements, e.g. animations, videos
and pictures, compatibility with various operating sys-
tems and devices like tablet, iPad, etc. The respondents
also think that the educational software should be finan-
cially affordable for schools, teachers and students. It
should be complex in terms of the content, and should
also meet the professional criteria. When we are talking
about individual items, each of the groups stated some-
thing special, what should not be missing from its point
of view. However, in all the groups and all the items
these features were almost identical.
6. Conclusion
The introduction of standardized evaluation tools, which
are able to competently evaluate the quality of educa-
tional software and electronic learning materials, is com-
plicated because of a number of factors. One of the most
important factors is the determination of appropriate,
expected or directly required features, which are crucial
for the classification of software product as a recom-
mended and high quality learning tool. The research re-
alized among respondents—specialists revealed rela-
tively clear results. A high level of respondents’ agree-
ment defined the simplicity and clarity in installation,
start-up and orientation as most important features of
educationally focused software products. Other important
features were represented by interactivity, multimedia
content, various forms of intermediation of information,
e.g. texts, pictures, videos, animations, technical and fi-
nancial affordability for users, etc. The research also con-
firmed that the extent of expected software features de-
pended on its content, focus and purpose, for which it
was intentionally created.
REFERENCES
[1] S. Bostock, “Classification of Educational Software,”
Keele University, 1995.
http://www.keele.ac.uk/depts/aa/landt/lt/docs/atcbttyp.htm
[2] S. Bleck, M. Bullinger, A. Lude and S. Schaal, “Elec-
tronic Mobile Devices in Environmental Education (EE)
and Education for Sustainable Development (ESD) Eva-
luation of Concepts and Potentials,” Procedia-Social and
Behavioral Sciences, Vol. 46, 2012, pp. 1232-1236.
[3] Open Learning Technol- ogy Corporation Limited, “CAL
Software Classifications,” 1996.
http://www.educationau.edu.au/archives/CP/07.htm
[4] co_hodnotit, 2008.
http://pdf.uhk.cz/kch/evaluace_SW/co_hodnotit.htm
[5] C. Conboy, S. Fletcher, K. Russell and M. Wilson, “An
Evaluation of the Potential Use and Impact of Prezi, the
Zooming Editor Software, as a Tool to Facilitate Learning
in Higher Education,” In: Inovations in Practice, Vol. 7,
2012, pp. 31-45.
[6] Criteria for Evaluating Software, 2001.
http://www.colleges.org/software_eval/criteria.html
[7] Educational Software Evaluation, 2009.
http://www.humboldt.edu/~ra7001/pages_courses/sed718
/formb.html
[8] Evaluace Výukového Software, 2007.
http://pdf.uhk.cz/kch/evaluace_SW/evalindex.htm
[9] Educational Software Evaluation, 2002.
http://www.sc.edu/etc/eval.etceval.htm
[10] Educational Software Evaluation Criteria, “Freshpond
Software Review,” 2001.
http://www.freshpond.net/treasures/technology/softreview
s/softeval.htm
[11] R. Fisher, “Učíme děti Myslet a učit se,” Praha, Portál,
2011, 176 p.
[12] R. Foshat and M. I. Ahmed, “A Practical Process for Re-
viewing and Selecting Educational Software,” PLATO
Learning, Inc., 2003.
[13] “Kritériá na hodnotenie kvality učebnice pre všeobecno-
vzdelávacie predmety,” 2008.
http://www.statpedu.sk/buxus/docs//vyskum/ucebnicova_
politika/hodnotenie_kvality_ucebnice.pdf
[14] “IT232—Pedagogical Software,” 2009.
http://www.idi.ntnu.no/~terjery/IT232/EnglishIT232.html
[15] J. Jeff, “Evaluation of CAL Software for Higher Educa-
tion: A Task for Three Experts,” Enhancing Learning,
Teaching & Curricula with a University-Wide Integrated
World Wide Web Framework. The Hong Kong Polytech-
nic University, Educational Development Centre, 2013.
[16] K.-T. Yang and T.-H. Wang, “Interactive White Board:
Effective Interactive Teaching Strategy Designs for Bi-
ology Teaching,” In-tech, E-Learning-Engineering, On-
Job Training and Interactive Teaching, 2012, pp. 139-
154.
[17] I. Kalaš, “Mýty a Vízie O Informatizácii Školy,” In:
Zborník Príspevkov z 5. Celoštátnej Konferencie Infovek,
Bratislava, 2005, pp. 35-42.
[18] S. Kemmis, R. Atkin and E. Wright, “How Do Students
Learn?” Working Papers on Computer Assisted Learning,
Centre for Applied Research in Education, Norwich,
1977.
[19] Z. Kolář and A. Vališová, “Analýza Vyučování,” Grada,
Praha, 2009, 232 p.
[20] D. Lehotská, “Edukačný Softvér,” In: Matematika Infor-
Open Access JSEA

Standardization of Quality Evaluation of Educational Software and Electronic
Learning Tools—Analysis of Opinions of Selected Experts 581
matika Fyzika, Roč. 16, č. 30, 2007, pp. 16-23.
[21] J. Lever-Duffy and B. J. Mc. Donald, “Teaching and
Learning with Technology,” 4th Edition, Pearson Educa-
tion, Boston, 2011.
[22] J. Mareš, “Styly Učení Žáků a Studentů,” Portál, Praha,
1998, 239 p.
[23] S. Papert, “Jean Piaget,” The TIME 100.
http://www.time.com/time/time100/scientist/profile/piage
t03.html
[24] J. Pavlíček, “Projektování Výuky pro Inovativní Výukové
Prostředí,” 2. vyd, Ostravská Univerzita v Ostravě, Os-
trava, 2008, 100 p.
[25] J. Piaget, “Logic and Psychology,” Manchester Univer-
sity Press, New York, 1953.
[26] J. Piaget, “Psychologie Inteligence,” SPN, Praha, 1970.
[27] Program, “Definition and Much More from Answers.com,”
2009. http://www.answers.com/program
[28] J. Prucha, “Encyklopedie Pedagogiky,” Portál, Praha,
2009.
[29] M. Resnick, “Rethinking Learning in the Digital Age,” In:
G. Kirkman, Ed., The Global Information Technology
Report: Readiness for the Networked World, Oxford Uni-
versity Press, Oxford, 2002.
[30] L. P. Rieber, “Microworlds,” In: D. Jonassen, Ed., Hand-
book of Research for Educational Communications and
Technology, 2nd Edition, Lawrence Erlbaum Associates,
Mahwah, 2004, pp. 583-603.
[31] M. Roblyer, “Integrating Educational Technology into
Teaching,” Pearson Education, Boston, 2012.
[32] K. L. Seifert and R. J. Hoffnung, “Child and Adolescent
Development,” Hougton Mifflin Comp, New York, 1994.
[33] D. B. Shinde and S. D. Gaikwad, “An Evaluation of Se-
lected Open Source Software for Digital Libraries,” Seva
Sadan’s College of Education, Ulhasnagar, 2013.
[34] M. Simpson and F. Payne, “Using Information and Com-
munications Technology as a Pedagogical Tool: Who Ed-
ucates the Education?” 2009.
http://www.leeds.ac.uk/educol/documents/00001292.htm
[35] V. Smékal, “Pozvání do Psychologie Osobnosti. Člověk v
zrcadle Vědomí a Jednání,” Barrister & Principal, Brno,
2002, 517 p.
[36] “Software Evaluation”.
http://www.ed.brocku.ca/~jkerr/sftwreva.htm
[37] “Software: Definition and Much More from Answers.com”.
http://www.answers.com/topic/computer-software
[38] R. Taylor, “The Computer in the School: Tutor, Tool,
Tutee,” Teachers College Press, New York, 1980.
[39] “Teachernet, Learning, Teaching and Managing Using
ICT”.
http://www.teachernet.gov.uk/wholeschool/ictis/
[40] “The State of Children’s Software Evaluation”.
http://www.childrenssoftware.com/evaluation.html
[41] “Transforming Teaching and Learning through ICT in
Schools,” 2004-2007.
http://www.northamptonshire.gov.uk/NR/rdonlyres/301B
8C79-2F78-4A6A-B9E6-24E6AC3A0136/0/TTandLICTi
nSchoolsStrategy200407finalversionpagesinorder.pdf
[42] M. Veselský, “Pedagogická Psychológia 1. Teória a Prax,”
Univerzita Komenského, Bratislava, 2004.
[43] M. Veselský, “Pedagogická Psychológia 2. Teória a Prax,”
Univerzita Komenského, Bratislava, 2008.
[44] A. Vitovský, “Moderní Slovník Softwaru. Výkladový.
Anglicko-Český a Česko-Anglický,” AV software, Praha,
2006.
[45] “Výkladový Terminologický Slovník Electronických Ko-
munikácií”.
http://www.vus.sk/iecd/new/Vyklad.asp
[46] “What Is Software?—A Word Definition from the We-
bopedia Computer Dictionary”.
http://www.webopedia.com/TERM/s/software.html
[47] “What the Research Says about Using ICT in Science”.
http://www.becta.org.uk/page_documents/research/wtrs_s
cience.pdf
[48] Z. H. Li, “Superiority of Multimedia Technology in Ge-
ography Teaching,” In: T. Zhang, Ed., Future Computer,
Communication, Control and Automation, Springer-Ver-
lag Berlin Heidelberg, 2012, pp. 581-585.
Open Access JSEA