Usability Testing on QR Code Scanner Application for Lecture Presence

-Universities are obliged to facilitate lectures. Many students require universities to provide a fair teaching and learning services with a minimum of student cheating. In order to improve the quality of teaching and learning, each university develops a lecture presence system electronically. The QR code scanner application became a solution offered for leak problems that previously existed on the magnetic card system. Before the application applied on a large scale, developers needed to conduct an assessment of the usability of the QR scanner application. The assessment aimed to make lectures go smoothly and to maintain the good reputation of the university. The method used is usability testing. The result of this study is a usability system at the level of 65%. This value consists of an effectiveness value of 70%, an efficiency value of 54.31%, and a satisfaction value of 70.85%. The improvements of user interface recommended in this study include adding of placeholders to inform the correct NIM format, changing the QR scanner icon into a titled icon and choosing a stimulating color, providing a zoom feature on the scanner camera, and applying a more familiar logout icon according to the mental model of the user.


Introduction
Universities and colleges are required to facilitate lectures. Higher education institutions have a different lecture system, but still refer to the 2015 government regulation on Permenristekdikti 44 concerning SN Dikti Article 3 [1] where the University must meet the national standards of higher education to achieve the specified quality. The quality in the teaching and learning process can be achieved, one of which is the facilities and infrastructure that can be accessed by students and lecturers. In terms of teaching and learning, the facility that must be provided is a student presence system in the classroom. Presence is the evidence of the teaching and learning process that needs to be well-conducted.
This study engaged a private university as a case study, which is the University of AMIKOM Yogyakarta. AMIKOM University is a university that has 16 study programs and more than 10,000 active students. Many students demanded that AMIKOM provides fair teaching services with a minimum of student cheating. Hence, the presence system for students had already applied using an electronic system. As time goes by, the magnetic cards were considered less capable to ensure that students are actually present or only entrust their magnetic cards to others. This leak problem must be resolved, so that fair lectures could be conducted to generate quality college graduates. Therefore, the QR Code scanner application was developed to replace the magnetic cards. The QR code scanner applications required testing to ensure that the application could actually be used. Then, the usability testing was carried out to determine the usability level of the application. The testing was conducted to maintain the good reputation and sustainability of the university.
All actions in the human factor were considered to have their impact towards sustainability [2], [3], [4]. The usability testing was an important step in the preparation of scanner applications, because the success or failure of the implementation would greatly affect sustainability in the teaching and learning process at the university. The problem formulation of this research is "what is the usability level of the QR code scanner application for student attendance?". The purpose of this study is to provide some recommendations for improving the QR code scanner application for student attendance.
Research on usability testing was conducted by Shafrida, R et al [5]. This study applied cognitive walkthrough methods to test applications for blind people, namely B-Smart. This research focused on the interface testing. Shafrida's study used one testing method. The number of respondents from Shafrida's study were 30 respondents -who were potential users of the B-Smart application. The next research was a study conducted by Zuntriana, A [6]. This study examined the Indonesian national library web portal. The method used is the Remote Usability Test. The findings in this study indicate that there were difficulties in finding the service schedules. Another finding is the dissatisfaction of the user interface design and layout on the home page. Another study was conducted by Farouqi et al [7] who tested usability testing in online transportation applications in Indonesia. Farouqi used the SUS questionnaire as well as measuring effectiveness and efficiency. Nielsen [8] defines usability as a quality that assesses how easy the user interface is used. International Standards (ISO) [9] defines usability as effectiveness, efficiency, and user satisfaction in achieving certain goals in certain environments. Usability testing of computer systems for complex tasks must include measures of efficiency, effectiveness, and user satisfaction [10]. Usability testing refers to evaluating a product or service by testing it with a representative user. The aim is to identify usability problems, gather qualitative and quantitative data, and determine participant satisfaction with the product [11]. Usability is a trait that depends on interactions between users, products, tasks, and the environment [12] p. 1267. Usability is very important for all types of interfaces [13]. This helped engineers to design user interfaces [14]. This study refers to the usability handbook written by Rubin J, and Chisnell, D [15]. This book guides how usability researchers begin the research. One of the methods discussed in the book is usability testing implemented in this study.

Method a. Usability Testing Method
According to the book "Handbook of Usability" [15], methods for evaluating usability can be done in various ways, one of which is Usability Testing. This method was held by observing the user. Researchers provided realistic tasks using both formal and informal approaches.
The usability testing process is explained in Figure 1. The test in Figure 1 consists of 8 steps, including:

Develop a test plan
At this stage, the researcher planned an overall test of how, when, where, who, why.

Prepare a test environment
At this stage, the researcher established a "usabilitylab" to accommodate testing by respondents.
3. Find and select participants Participants in this study were determined at this stage.

Prepare test material
This step contains the implementation of step 2.
Step 4 aimed to reduce testing failures due to lack of equipment preparation in testing.

Running a test session
This step is the beginning of testing. 6. Question and answer by participants and observers At this stage, participants began to be explored with questions from the observer. 7. Analysis and observation of data Data obtained at the question and answer stage, was followed by analysis to produce findings.

Report findings and recommendations
Reports on the findings of the analysis were written and documented.

b. Sampling technique
There are 2 sample groups required in this study, including:

1) Qualitative sample group
This sample was taken from students who had never used the QR scanner application. This sample aimed to observe the mistakes made by participants from the level of success and efficiency. The sample required for this group was 5 people. The number of samples recommended by Nielsen is 3-5 people [16]. Students -who were participants -were asked to carry out the following three task scenarios: -T01 - Task  There were three assessments of the success rate of the participants that became the measurements in this study, including: -Success: the participants completed the task with the right steps. -Delayed: the participants completed the task, but with steps that do not fit the scenario. -Failed: the participant failed to complete the task.
Participants were arranged to sit in the front, middle and back area of the class. The first and second rows were the front area category, the third and fourth rows were the middle area category, and the fifth and sixth rows were the rear area category. Participants were observed by observers in terms of their level of success and efficiency.

2) Quantitative sample group
Quantitative sample groups in this study were used to measure satisfaction of users of the application. The sampling technique in this quantitative group is the Probability Sampling Technique. This technique was taken since the population is countable, so that the sample data could become a representation of population data. Participants in this study were students who were active and still doing lectures in the theoretical class. The list of students who took lectures in class can be seen in the Study Plan Card (KRS) which is taken in the even semester of the 2018/2019 academic year. Slovin formula was applied to determine the number of samples. The data obtained was used to measure application user satisfaction. However, the satisfaction itself could only be measured on users who had already used the application.

c. Data analysis method
The level of success, efficiency, and satisfaction were measured using the measurements conveyed by Nielsen [17]. The measurements to be analyzed were the success rate (1), efficiency (2), and satisfaction.
The calculation formula is shown as follows: Success Rate = (Success + (Delayed Success x 0.5)) / (Total Tasks) x100% (1) Efficiency was determined by the amount of time required to complete the given task [18]. The comparison of time between novice users and standard time to complete tasks became a percentage of efficiency. The efficiency was based on time, which can be calculated by the formula: Satisfaction was calculated using the SUS (System Usability Scale) [19] formula, informed in the following steps: 1. Odd-numbered questions' score was reduced by 1. 2. Even-numbered questions' score was calculated by subtracting a value of 5 with the respondent's answer. 3. SUS score = the scores of each question were added up, then multiplied by 2.5. 4. The average of the score was calculated.

Results and Discussion
QR code scanner application could be applied in a theoretical class with QR code processing as follows:

Lecturer Login
Each class was already provided a PC for a lecturer to login on the class presence system and projector.

Generating QR Code
When the lecturer logged in, the system would generate a QR code.

Displaying the QR Code
The projector would display the generated code.

Refreshing the QR Code Display
The system would periodically update the displayed QR code in every 15 seconds. 5. Displaying students who had successfully attended. Students could send the information regarding name, NIM (Student Identification Number), and scan time, by logging in to the QR code scanner application -installed in each student's smartphone. After that, the process of scanning the code on the projector screen by pointing the smartphone's camera to the screen was conducted. Finally, the data was sent to server via the internet.
Based on the process in Figure 2, the usability test can be started with the following steps:

a. Develop a test plan
The test plan was divided into several steps: 1) Testing Purpose The purpose of testing a QR code scanner application is to find out whether the application could be used by Amikom students for attendance (marked by success).
Another purpose is to find out which part of the application became the point of the most errors, and then creating recommendations for improvement.

2) Research Questions
What is the usability level of the QR code scanner application for lecture attendance? 3) Testing Method (Table 1)

4) Task List -
Login to the QR code scanner application -Presence using the QR code scanner application -Logout from QR code scanner application 5) Testing, Equipment and Logistics Environment -Testing environment: Classroom -Equipment: smartphone, internet connection, projector screen, presentation computer.

6) Role of Testing Moderators -
Introducing the testing objectives -Giving assignments according to the assignment plan 7) Data collected and evaluation measurements The data collected is in the form of qualitative data and quantitative data. Qualitative data was obtained from interviews with participants; the goal was to explore the failure reasons of the application. Quantitative data was obtained from questionnaire results. Usability in this study was measured from the ratio of success, efficiency, and satisfaction conducted by the participants during completing the task. The data collected was in the form of qualitative and quantitative data. Qualitative data was taken from direct observation to participants who had never used the application. The tasks given to the participants include login, try presence, and logout. Quantitative data was obtained from the questionnaire answers from 384 active students who were taking theoretical courses and had used the QR Code scanner application for the presence before.

Inter-Subject Plan
Each participant was given the task to do a presence in class using the presence application. Participants sat on a chair that was prearranged in a sequence from the front to the back side of the classroom. The order of the seat shows whether the distance determines presence failure or not. The participant was asked to carry out the task, when the participant had completed the tasks given by the observer.
The form was filled in by the observer, containing how long the participant completed the task, what failures were encountered, and what made it difficult for the participants to complete the tasks.

Time
The total time required is 25 minutes. The introduction session was held in 5 minutes, the task implementation was conducted in 10 minutes, and the interview session upon the test completion was carried out in 10 minutes. The test is located in the regular classroom at Amikom University in Yogyakarta.

Introduction (5 minutes)
Introduction to observer and explanation of the observation activities. Filling in participant's personal data.

Execution of tasks (10 minutes)
The participants began to work on the tasks by using a QR code scanner for class presence.
Interview after test (10 minutes) Asking participants regarding the problems they encountered during the presence.

8) Report content and presentation
The report contains a Google form spreadsheet and photo documentation.

9) Set up the test environment
At this stage, the researcher arranged "usabilitylab" to accommodate the testing performed by the participants ( Table 2).

b. Participant Selection 1) Qualitative Group
There were 5 participants who became respondents in this study. Participants 1-5 were observed in a class. Each participant represents the front area of the right and left sides, the middle of the right and left sides, and the back of the right side. application. The number of students represents the total population in this study.
The sampling technique employed is simple random sampling method. This method was conducted by determining the number of samples using the Slovin formula. The simple random sampling method was preferred since the population is large and each member of the population has the same opportunity to be sampled. The number of samples taken was based on the Slovin formula, explained as follows: n= 9659/(9659*(0,05)^2+1) n= 9659/(24,1475+1) n= 384,093846 The number of samples used was 384.093846, rounded up to 384 students.

c. Test Materials Preparation
This step contains the implementation of step 2. This step aimed to reduce the failure of the test due to lack of equipment preparation in the test.
The materials prepared for the testing session includes: -Projector Screen -Monitoring Form -Internet connection

d. Running a Test Session
This step is the initial step of the test session. The testing session was carried out in parallel between success, efficiency, and satisfaction. The testing session is divided into 2 groups, including the success and efficiency testing as well as the satisfaction testing.
The success and efficiency testing was held in class with 5 participants and 1 observer. The testing process of the 5 participants was not conducted simultaneously, but one at a time. First, the participants were given a briefing regarding the tasks that need to be completed by the participants. Participants did not receive an explanation or clue related to completing the given tasks.
The satisfaction testing was conducted online by filling out questionnaires that are accessible for 10 days -to meet the number of samples, 384 respondents.

e. Participants and Observer Questions and Answers
At this stage, participants began to be explored with questions from the observer. After the participant completed the task, the observer conducted an interview with the participant. The interview questions are mentioned as follows: 1) Did you successfully login to the application? 2) Were there any difficulties experienced in the login task? 3) Did you successfully use QR code? 4) Were there any difficulties in assigning attendance? 5) Did you successfully logout from the application? 6) Were there any problems experienced in the logout task? The data obtained in Task 1 is that participants were asked to login to the QR code scanner application. Participant 1 has a status of D or Delayed. It means that participant 1 successfully completed the task but did the wrong steps. An error occurred when participant 1 tried to enter a NIM that was supposed to apply a dot. Participant 1 did not use a dot, so that participant 1 was unable to enter the application successfully. The application did not inform the user regarding the location of the error. However, participant 1 tried to add a dot to the NIM (Student Identification Number) text field and finally succeeded. Scan the Code S D D D F Success = (1+ (3x0.5))/5 x100% = 50% Participants 2, 3, and 4 are categorized as Delayed due to the same mistake, i.e. participants mistakenly tapped on the "presence" icon that contains the attendance history. Participants thought that the icon was an icon for attendance using QR scanner. After realizing that the participant was wrong, the participant returned and then chose the QR code scanner icon and finally succeeded. Participant 5 failed because participant 5 could not scan the QR code from the point where the participant sat. Participant 5 sat at the back-right side of the classroom. Participant 5 tried to enlarge the camera, and expected that the application has a zoom feature. The success rate for this task was only 50% because there were 3 delayed and 1 failed. Participants 2 and 4 conducted the logout process without any problems. Participant 1 was delayed because participant 1 thought that the logout button was on the profile menu, so that participant 1 chose the profile photo icon. Participant 3 lack knowledge regarding which icon to choose to exit the application. Participant 3 tried to logout and finally succeeded. Participant 5 tapped the code icon when being asked to exit. The error that occurred KHAZANAH INFORMATIKA | ISSN: 2621-038X, Online ISSN: 2477-698X Vol. 6 No. 1 | April 2020 in participants 1, 3, and 5 is basically due to the lack of information about the logout icon. An uncommon icon is used to log out from the application. This application uses icons that are less familiar to users. According to Table 7, the lowest success rate is task 2 with the success rate of only 50%. Assignment 2 is the main feature, where students were able to scan QR codes for their presence. There were two mistakes that were often made including a mistake of placing the presence history in the presence menu. The next fatal error is that the QR code could not be scanned from the seat at the back of the classroom. The longest time needed to complete task 1 is 22 seconds and the fastest is participant 3, completed in 11 seconds. The standard time is the time taken by people who have already used the application.  Logout was able to be completed in 3 seconds, if the icon in the application was already known. The most time required by participant 1 is 20 seconds because participant 1 did not recognize the logout icon.

2) Efficiency
The average overall time to complete task 1 to task 3 for novice users is 52.4 seconds. Whereas, the standard time is 29 seconds or less. The application is stated to be efficient when the value is close to 100%. In this application, the efficiency value is 54.31% -obtained because the participant undertook the wrong steps in selecting the menu for completing the task. Most of the data belongs to class D, which is 137 respondents. Class D contains respondents whose score between 51-67. The number of respondents and the scores of respondents are informed in the following table. The  individual values obtained in table 13 were then calculated altogether, resulting in a SUS score of 70.85. The score implies OK/Fair satisfaction level, which means good but required an improvement to become Excellent.

g. Report Findings and Recommendations 1) Findings
The findings gained from this study were some errors that arose due to the design of the user interface that did not sufficiently depict the function of each icon. Some error findings are informed in Table 15. The user lack knowledge that inputting NIM must apply dots to separate numbers 2 Presence The user chose a larger icon and can attract more attention.
The user thought the icon is an icon to start scanning the QR code 3 Presence The user failed to scan the QR code because the seat is located at the back side of the classroom The camera could not reach the code, so that it could not read the code properly 4 Logout The user was trying around by selecting the profile icon The user thought logout is situated in the profile menu; the logout icon is not familiar.

2) Recommendation
The recommendations of the study findings were produced according to errors found in this study. The recommendations are informed in Table 16. Changing the QR scanner icon with a titled icon and change the color to a stimulating color 3. Error during scanning QR code Providing a zoom feature on the scanner camera 4. Error during logging out Using a logout icon that is more familiar according to the user's mental model

Conclusion
Usability testing showed the usability level of the QR Code scanner results of this study is 65% -with effectiveness at the level of 70%, efficiency at the level of 54.31% and satisfaction at the level of 70.85%. There were four mistakes that were often made by novice users. The first mistake occurred when user entered NIM (Student Identity Number). The second mistake occurred when user trying to scan a QR code, but the user chose a more attracting icon. The third mistake occurred when user tried to scan a QR code that was too far from the seat. The fourth mistake was a mistake in selecting the profile icon when the user tried to exit. These errors resulted in inefficient use of time. The recommendations of developing a QR code scanner application in lectures include adding a placeholder to inform the correct format of NIM, changing the QR scanner icon to a titled icon and choosing a stimulating color, providing a zoom feature on the scanner camera, and using a logout icon that is more familiar according to user's mental model. The future studies are expected to provide comparative results after some interface improvements based on the results of this study.