Academic Information System Assessment of AKRB Yogyakarta Using UTAUT

- Implementation of Academic Information System (AIS) at the Radya Binatama Communication Academy (AKRB) had some problems for some users. That problems are known from interviews with several AIS users. This caused a delay in data exchange with other divisions. This study aims to assess the implementation of AIS to the AKRB by using the Unified Theory of Acceptance and Use of Technology (UTAUT) method. UTAUT has four main constructs that affect user acceptance namely performance expectations, effort expectations, social influences, and facilitating conditions. The data were obtained from distributing questionnaires to all AIS users as many as 40 respondents. Then the data is processed using Structural Equation Modeling (SEM) techniques with the help of SmartPLS. The results of the analysis show that only construct facilitating conditions is valid with a t-statistic value of 2.733. While the other three constructs have values that are in the range of invalid values between -1.96 to 1.96 with the values of each construct being 1.891, 0.050, 1.440. It can be concluded that the application of SIA in AKRB has not been well received by all AIS users. Therefore, it is necessary to conduct an evaluation that represents the other three constructs.


Introduction
Academic activities are an important part of educational institutions which include student data, starting from registration, tuition fees, to graduation. Academic Information System (AIS) used to store data and manage online academic activities. AIS aims to process all live information in an integrated manner so that the existed data is always real time and up to date [1]. With AIS, academic data management can be done easily and quickly as an information center and used to monitor the progress of student studies.
Radya Binatama Communication Academy (AKRB) Yogyakarta, Indonesia has been using AIS to assist in the implementation of academic activities since 2018 to replace the previous one to carry out a lecture contract using a paper form filled out by students and submitted to the university. Every system used, of course, requires changes in line with user needs. The use of AIS is intended to improve services to students and make it easier for academic staff in the operator section t o carry out routine reporting to the government.
Since the implementation of AIS at the AKRB in 2018, there has never been a test on its implementation. Based on interviews, it is known that there are still staff who do not fully understand how to use AIS. However, the campus management did not know this and did not carry out full socialization regarding the use of AIS. This is deemed necessary considering that the training on the use of this information system is only conducted once and has not been followed by all staff. In addition, there have been several recruitments of new staff who directly use AIS without any detailed instructions regarding its use. This happened caused by AIS users less in optimization of AIS. The effect is that the completion of the work takes time and the flow of information between divisions or to students is hampered. The worst thing that might happen is that the data provided is wrong and results in wrong decision making.
User acceptance greatly affects the application of a technology or system. The extent to which users can utilize and accept the technology is important to know the success rate. This is as done by Widiyanti [2] and Ekayanti [3] who conducted an evaluation to determine the acceptance of AIS UNISRI Surakarta and LMS Gandhi Memorial Intercontinental School.
The system can be accepted if the user feels the benefits in its use such as helping to complete work quickly and simplifying the work [4]. User acceptance is a important factor to determine the success of the application of system information. An analytical model is needed to measure the level of success of the application of system information, one of which is The Unified Theory of Acceptance and Use Technology (UTAUT) which in this study was chosen to evaluate the implementation of AIS AKRB.

a. Unified Theory of Acceptance and Use Technology (UTAUT)
UTAUT is a theory used to determine the level of user acceptance of an information system. The UTAUT model is a theory developed by Venkatesh, et.al in 2003 [5]. UTAUT combines the successful features of eight leading technology acceptance theories into one theory. The eight leading theories that are combined in UTAUT are Model of PC Utilization (MPTU), the Theory of Reasoned Action (TRA), Motivational Model (MM), Innovation Diffusion Theory (IDT), Technology Acceptance Model (TAM), Social Cognitive Theory (SCT), Theory of Planned Behavior (TPB) and combined TAM and TPB [6]. UTAUT proved to be more successful than the other eight theories in explaining up to 70% of user variance [7]. There are several studies that have tested the acceptance of technology using the UTAUT method and ensured the effectiveness of this model, including the evaluation of the use of WhatsApp for learning during the COVID-19 period at the University of Zimbabwean [8], factors influencing lecturers' intention to use e-learning inenvironment hybrid [9], evaluation of the application of MyUMN as an academic information system using UTAUT at Multimedia Nusantara University [10], and the use of UTAUT to understand academics' perceptions of the Online Faculty Development Program [11]leading to learning through online portals. In the prevailing covid-19 situation, countries experienced lockdown, work from home, and online learning has emerged in a big way. From traditional classroom teachers to an online learning environment, the role of academicians has changed drastically. Hence, Faculty Development Programs became an important component in this lockdown period for skill and knowledge up-gradation. This study used a quantitative methodology in which a theoretical model UTAUT was tested using hypotheses to assess the interaction between the study variables, performance expectancy (PE . The purpose of implementing UTAUT is to assist universities in understanding user reactions to the implementation of new technologies. Initially, UTAUT was developed from the Technology Acceptance Model (TAM) with four main constructs that influence it or Behavioral Intention (BI), namely Facilitating Conditions (FC), Social Influence (SI), Effort Expectancy (EE), and Performance Expentancy (PE) as can be seen in Figure 1. PE is the level of expectation that each individual has that the use of the system can improve performance at work. SI is defined as the extent to which a person perceives the importance that other people believe will influence using the new system. EE is defined as where the comfort of the user in using the system. FC represents the degree to which a person believes that technical infrastructure and the organizational is in place to support the system. Behavioral intention is a behavior of using a technology in the long term, the ability of an application to provide benefits to its users and foster satisfaction for its users so that they want to convey positive things to others and recommend the application to other users and continue to use the application.
A similar study was conducted by Handayani and Sudiana in 2017 [12]including institutions of higher education. With the presence of Academic information systems (SiAkad, by examining the application of the Academic Information System at STTNAS Yogyakarta using the UTAUT method. The study used 325 respondents consisting of lecturers, staffs, and employees. This study also uses four constructs as instruments in the study. The difference lies in Handayani and Sudiana who use SPSS Amos and this study uses Smart PLS as the respondent's data processor. From the results of the analysis, it was concluded that PE, SI, and FC had a significant effect on BI, while the EE variable gave results insignificant. From this research, this study will test the four constructs of the UTAUT model with the following hypotheses: H1: EE has an important influence on BI H2: FC has an important influence on BI H3: PE has an important influence on BI H4: SI has an important influence on BI In facilitating the research process, in this phase, the research phase is divided into several parts. a) Introduction. There are several activities carried out, including determining the location of the research, determining the object of research, and setting the title of the research. The research location was determined, namely the AKRB with the object of research focusing on AIS.
b) Data Collection. There are four stages, namely literature study, observation and interviews, making questionnaires, and distributing questionnaires. c) The last step is to analyze the data that has been collected with Smart PLS. The analysis stages are grouped into several parts including testing the outer model and testing the inner model.

b. Academic Information System (AIS)
Academic Information System (AIS) is a system built to process campus data with the help of hardware and software so that the stages of academic activities can be analyzed into useful data. The data from the analysis is expected to help manage information and can be used as a reference for policy makers in making decisions for university development. With the existence of SIA, people from inside and outside the university are expected to receive more effective and better information services through the internet so that the education process is carried out properly. Educational institutions generally have many needs and have academic management regulations that are quite complicated and take time, thought and energy. Therefore, AIS is applied to provide a fast, effective and efficient response to the needs and problems of universities to process university data.

Structural Equation Modeling (SEM)
Structural Equation Modeling (SEM) is a technique developed to analyze many variables so that the limitations of the analytical models used in the past are covered and this technique has also been used in many statistical studies. The advantages of SEM in research include: a) Can test the relationship of causality, validity, and reliability at the same time. b) Can be used to see the direct and indirect effects between variables. c) Testing several dependent variables at once with several independent variables. d) Can measure how much the indicator variables affect the respective factor variables. e) Can measure factor variables that cannot be measured directly through the indicator variables.
SEM modeling steps start from theoretical model development, path diagram development, flowchart conversion into SEM equations, input matrix selection and estimation techniques, assessing problem identification, model evaluation to interpretation, and model modification [13].
The analysis stage begins with forming an initial model of each method using the adapted constructs. After that, enter the testing stage, Outer Model first which is to test the convergent validity of each indicator of the construct. Validity testing is useful so that all research indicators measuring research variables that have been distributed are valid or not. In SmartPLS, convergent validity is seen through the results of the loading factor of all indicators in the construct related. If the standard value of the correlation coefficient is 0.5 or more then the data is declared valid. If all indicators of each construct are valid, then the test continues, Outer Model second phase is namely the Discriminant Validity Test.
Discriminant Validity Test reviewed through the indicator value Fornell-Larcker Criterion. A construct is said to be valid if the calculation value indicated by the correlation value between one construct and another is smaller than the Fornell-Larcker Criterion of each construct. If the test results show an invalid value, the test model must be changed again and testing is carried out from the beginning, namely testing convergent validity.
The final testing stage of the Outer Model is the Reliability Test. This testing phase is carried out to see the level of reliability or consistency of a research instrument. To test the reliability of the instrument, technique can be used Cronbach's Alpha and composite reliability. The reliability test is declared valid if it produces Cronbach's Alpha 0.6 and composite reliability 0.7. In this study, the test was carried out with only one measurement or one shot.
After testing the Outer Model has been valid, then the testing can be done Inner Model. This analysis is carried out so that the structural model made is strong and accurate. The stages inner model tests the significance of the effect of each variable by looking at the t-statistical value [14]. The prediction model of the research is said to be good if the r-square value is high. The results of path coefficients represent the level of significance at the hypothesis testing stage. The results of the analysis at this stage are used to review the impact of the independent variable on the dependent variable.
Still in the Inner Model Test, the next test is Hypothesis Testing. At this stage, the focus is on the magnitude of the T-statistic value using a significance level of 95%. The result of the T-table with this number is 1.96. If the results of the T-table are in the range of -1.96 to 1.96, then the hypothesis is rejected or said to be invalid [15]. In SmartPLS there is a bootstrap feature that can be used to get the t-value and path coefficient.
There are several ways to use SEM analysis. The first way is-based SEM Covariance (CB-SEM) which is generally practiced by adopting software packages such as Lisrel, EQS, AMOS, and Mplus. Another way is Partial Least Squares (PLS), which targets analysis of variance and can be done using Visual PLS, PLS-Graph and SmartPLS. In this study, Smart PLS is the choice in analyzing the data to be studied.

Result a. Respondent
This research is similar to Sani and Salim [16] which only takes respondents from academic staff because the system between staff and students is a different system. Compared with Kristiawan and Harisno's research [10], the study used 423 respondent data distributed via email to AIS users, in this case students. Kristiawan  research processes the respondent's data with different software from the one used in this study. The study used AMOS version 23 for windows but still used the SEM flow in its research. The results of this study indicate that the constructs of Performance Expectancy and Social Influence have a significant impact on the construct of Behavioral Intention.
Another research that is still related to this research is Ulfa's research [17]. Ulfa conducted research with the same object as this research, namely AIS and processed it with the same software, namely SmartPLS. The difference lies in the model used in the analysis. Ulfa's research uses themethod DeLone and McLeanand shows the results that the application of AIS has been well received by its users. The constructs that have a significant impact on the implementation of SIMAK are information quality, service quality, system quality, self-efficacy, top management support, user satisfaction, and net benefits. Ulfa's research used 147 respondents data collected from questionnaires via email.
There are 60 respondents used in this research. The characteristics of respondents assessed in this research were age, gender, education, and knowledge of AIS AKRB. Details of the characteristics of 40 respondents can be seen in Table 1. The majority of respondents came from the male group of 26 people. In terms of age 30-39 as many as 17 people. In terms of education, the majority of respondents passed graduate. The method for collecting data is using a questionnaire which is a set of questions used to obtain information from respondents related to their personality and other matters related to the research material [18]. Questionnaire distribution is done by distributing questionnaires directly to respondents and using Google Form.

b. Outer Model Test
Partial Least Square (PLS) becomes the method used in conducting the analysis and then the data is processed with SmartPLS. PLS is a prophetic method that allows to analyze various independent variables, although there is a close correlation between two or more independent variables on these variables [11]. PLS is a method similar to SEM with the use of finding solutions to problems in fairly complex variable relationships. a) Convergent Validity Test. Validity testing is carried out so that all research indicators measuring research variables that have been distributed are valid or not. In PLS, convergent validity is seen through the results of factor loading of all indicators in the related construct. If the standard value of the correlation coefficient is 0.5 or more then the data is declared valid.

Figure 2. UTAUT Model first step
In Figure 2, it can be seen that the loading factor of each instrument is above 0.5 which indicates that the value of all instruments is valid. The next validity test is to test convergence by paying attention to the Average Variance Extracted (AVE) value of each instrument. In Table 2, it can be seen that the AVE value of each indicator of this model is worth more than 0.5 so that the indicators used have reached the minimum standard of convergent validity test.  Table 3.  Table 3 contains several invalid Fornell-Larcker Criterion values. Therefore, the existing UTAUT model must be modified so that the discriminant validity test is met. The results of the modification of the UTAUT model are shown in Figure 3.

Figure 3 UTAUT Model Step 2
From this modified UTAUT model, a convergent validity test is carried out again to ensure that this model has met the specified valid requirements. The loading factor value of each instrument > 0.5 indicates a valid instrument such as the AVE value of this modified model in Table 4.  Table 5. the Fornell Larcker Criterion value has met the specified assessment value so that this model is valid and can be continued in the analysis to the next step.

c. Inner Model Test
Next step if the outer test phase the model has been fulfilled, namely to test the validity of structural models . This analysis is carried out so that the structural model made is strong and accurate. The inner model stage tests the significance of the effect of each variable by looking at the t-statistics or t-value [14]. The prediction model of the research is said to be good if the r-square value is high. The path coefficient results represent the significance grade at the hypothesis testing step. a) Determination Test.  Table 7, it can be seen that the significance of the relationship between variables is by comparing the path coefficient value with the results on the T-Statistic.

Conclusion
The results of the analysis of the application of AIS to AKRB using the UTAUT model show that of the four hypotheses tested, only 1 hypothesis is valid or accepted. The construct is a Facilitating Condition (FC) with a T-Statistic value of 2.733.
Based on these values, it is concluded that the implementation of AIS on the AKRB Campus is still not good. This is indicated by the presence of three constructs that are still invalid, namely Performance Expectancy (PE), Effort Expectancy (EE) and Social Influence (SI). The results of the analysis using the UTAUT model can be used as recommendations to the AKRB in making policies as needed. For example, The PE construct representing user beliefs regarding the ease provided from using AIS is still rejected. As well as with SI which describes the influence of the environment in using AIS. It is necessary to hold socialization or training from the Campus to new users, and if there is an update in AIS. The EE construct relates to the user interface of AIS, which means that it is necessary to evaluate and improve AIS. Future research are expected to be able to use other methods in conducting research or examining other objects in the AKRB such as AIS for students.