RESEARCH OF METHODS OF EVALUATION OF EDUCATIONAL INFORMATION SYSTEMS - Научное сообщество
Главная
UA RU EN

Вас приветствует Интернет конференция!

Приветствуйем на нашем сайте

Рік заснування видання - 2011

RESEARCH OF METHODS OF EVALUATION OF EDUCATIONAL INFORMATION SYSTEMS

07.05.2024 13:10

[1. Информационные системы и технологии]

Автор: Dmytro Miroshnychenko, Bachelor of Computer Science, National Technical University "Kharkiv Polytechnic Institute", Kharkiv



Introduction. Relevance of educational information systems. Today, educational information systems are developing very rapidly due to the rapid pace of technology development. They are relevant for students and teachers during the period of distance education, which can be carried out due to, for example, quarantine or military and political actions. But now there are a lot of such systems, almost all of them have an intuitive interface, a large amount of information, automatic student assessment and storage of this information, presentation of statistics on the learning process, and feedback between system users. But you need to choose the best one. 

The purpose of the research is to compare four different methods of assessing the quality of educational information systems to determine their effectiveness and objectivity.

The first approach is the methodology “Assessment by means of a user survey”. This approach involves the creation of questionnaires to assess the satisfaction of NIS users. Users answer questions about various aspects of the system, such as usability, content quality, system responsiveness, etc. 

Advantages:

• Easy to use and understandable for survey participants.

• Can provide direct feedback from users.

Disadvantages:

• Subjective and subjective to users' personal preferences.

• Does not take into account the actual use of the system or its effectiveness.

The next approach is the “User Activity Based Evaluation” methodology [1]. This approach involves tracking user activity in the NIS, such as time spent in the system, materials viewed, interaction with the functionality, etc. Based on this data, a conclusion is made about the system's effectiveness.

Advantages:

• Uses objective data on user activity.

• Provides information about the actual use of the system.

Disadvantages:

• Can be difficult to implement due to the need to collect and analyze a large amount of data.

• Does not take into account user satisfaction or their personal impressions of the system.

There is also a methodology called “Assessment based on learning outcomes” [2]. This approach is based on studying learning outcomes, such as students' academic achievements, their progress in learning, knowledge gained, etc.

Advantages:

• Uses specific test or training program results.

• Allows you to assess the actual knowledge and skills gained from using the system.

Disadvantages:

• It may not take into account other aspects.

The Kirkpatrick model [3] is used to evaluate the quality of a product or service in terms of five aspects: functionality, reliability, usability, efficiency, and appearance. By applying this model to the evaluation of educational information systems, it is possible to take into account not only technical aspects but also to ensure high quality of use of the system.

Among the advantages, I have identified a wide range of evaluation aspects, the ability to assess the quality of the system from many perspectives.

The disadvantages are the difficulty of defining and measuring each aspect and the need for specialized tools for evaluation.

Choosing the Kirkpatrick Model Assessment methodology may be the best choice for several reasons:

1. Broad coverage of evaluation aspects: The Kirkpatrick Model takes into account five key aspects of evaluation, including functionality, reliability, usability, efficiency, and appearance. This allows for a more complete picture of the quality of the learning information system.

2. Ensuring high quality of use: An evaluation based on the Kirkpatrick model allows you to focus not only on the technical aspects of the system, but also on its interaction with the user. This will ensure a high quality of use of the system, which is critical for successful implementation and user acceptance.

3. Consideration of user needs: Kirkpatrick's model allows you to take into account the needs and expectations of system users. The assessment according to this model will help determine how well the system meets the needs of users and in what aspects improvements are needed.

Functionality determines how well a software product performs its main functions. It is estimated using the following formula:




where Fc - is the number of correctly executed functions,

Ft - is the total number of functions.

Functionality is measured on a scale from 1 to 100%, where 100% is perfect functionality and 1% is the worst.

Reliability reflects the stability and reliability of the system. It is determined as follows:




where NC - is the number of correct executions,

NT - is the total number of tests.

Scale: 0-100%, where 100% is absolute reliability.

Ease of use is evaluated by the proportion of users who were able to solve the task without additional support.




where NS - number of users who successfully completed the task,

NU - total number of participants.

Scale: 0-100%, where 100% is maximum convenience.

Efficiency is defined in terms of time and resources allocated to achieve certain goals. Formula:




where TA - actual time to achieve the goals,

TT - theoretically required time.

Scale: 0-100%, where 100% is maximum efficiency.

Appearance is evaluated based on the appearance and aesthetic design of the software product. This can be calculated as the average of the user ratings.

Scale: 1-10, where 10 is the most attractive appearance.

A comprehensive assessment of the quality of an educational information system can be calculated as the arithmetic mean of the scores for each of the assessment aspects. After the composite score is calculated, it can be measured using a scale that corresponds to the range of values of the composite score.

The composite score (C) is calculated as the arithmetic mean of these values:




where F – functionality (1), 

R – reliability (2), 

U – ease of use (3),

E – efficiency (4),

A – appearance.

The composite score is measured on a scale from 0 to 100%, where 100% corresponds to the ideal quality of the system and 0% corresponds to the lowest level of quality. This scale makes it easy to compare different systems in terms of their overall quality.

References

1 Activity-based assessment // URL: https://www.taasa.org/wpcontent/uploads/2014/09/Activity-Based-Assessment-Toolkit-Final.pdf. (дата звернення: 01.05.2024).

2 Learning Outcome Types and Recommended Assessment Methods. // URL: https://teaching.cornell.edu/learning-outcome-types-and-recommended-assessment-methods. (дата звернення: 01.05.2024). 

3 Kirkpatrick's model for assessing training effectiveness // URL: https://www.startexam.ru/journal/likbez/model-kirkpatrika-dlya-otsenki-effektivnosti-obucheniya. (дата звернення: 01.05.2024).

_______________________

Supervisor: Iryna Liutenko, Ph.D., National Technical University "Kharkiv Polytechnic Institute", Kharkiv

Creative Commons Attribution Ця робота ліцензується відповідно до Creative Commons Attribution 4.0 International License
допомога Знайшли помилку? Виділіть помилковий текст мишкою і натисніть Ctrl + Enter

Другие научные работы даной секции

Конференции

Конференции 2024

Конференции 2023

Конференции 2022

Конференции 2021



Міжнародна інтернет-конференція з економіки, інформаційних систем і технологій, психології та педагогіки

Наукова спільнота - інтернет конференції

:: LEX-LINE :: Юридична лінія

Інформаційне суспільство: технологічні, економічні та технічні аспекти становлення