Factors Influencing User Satisfaction with Generative Artificial Intelligence Power Chat System
DOI:
https://doi.org/10.21632/irjbs.18.1.67-84Keywords:
Artificial Intelligence, User SatisfactionAbstract
This research aims to explore the factors that affect user satisfaction with AI chat systems, particularly within the Indonesian context, to create a comprehensive evaluation scale for user experience. A quantitative approach using surveys with a Likert scale model was employed. Questionnaires were distributed online to AI chat users in Indonesia. A total of 365 valid questionnaires were collected, and all items were recorded using a seven-point Likert scale. Factor analysis was adopted to identify user satisfaction. Then, multiple regressions analysis was used to examine the impact of the multiple factors on AI chat system user satisfaction. The data analysis identifies factors that directly affect user satisfaction with the Generative Artificial Intelligence Powered Chat System. These factors are credibility, interactivity, ease of use, usefulness, convenience, growth, logical inference, and enjoyment. In contrast, the factors that indirectly affect user satisfaction are security, creativity, anthropomorphism, and accuracy. This study aims to provide valuable insights that will inform the design and implementation of more effective and user-friendly AI chat technology.
References
Adam, M., Wessel, M. and Benlian, A. (2020), "AI-based chatbots in customer service and their effects on user compliance", Electronic Markets, Vol. 31 No. 3, pp. 427-445. https://doi.org/10.1007/s12525-020-00414-7.
Akturan, U. and Tezcan, N. (2012), "Mobile banking adoption of the youth market: percep-tions and intentions", Marketing Intelligence & Planning, Vol. 30 No. 4, pp. 444-459. https://www.emerald.com/insight/content/doi/10.1108/02634501211231928/full/html.
Al-Adwan, A.S., Li, Na, Al-Adwan, A., Abbasi, G.A., Albelbisi, N.A. and Habibi, A. (2023), “Extending the technology acceptance model (TAM) to predict university students’ intentions to use metaverse-based learning platforms”, Education and Information Technologies, Vol. 28 No. 11, pp. 15381-15413, https://doi.org/10.1007/s10639-023-11816-3.
Alan, B. (2013), “The interaction of representation and reasoning”, Proceedings of the Royal Society A, 469: 20130194, https://doi.org/10.1098/rspa.2013.0194.
Alkhaqani, A.L. (2023), “ChatGPT and nursing education: challenges and opportunities”, Al-Rafidain Journal of Medical Sciences, Vol. 4, pp. 50-51, ISSN 2789-3219, https://doi.org/10.54133/ajms.v4i.110.
Anderson, C. L., and Agarwal, R. (2010). Practicing safe computing: a multimethod empirical examination of home computer user security behavioral intentions. MIS Q. 34, 613–643. https://doi.org/10.2307/25750694.
Athilingam, P. and He, H.-Gu (2023), “ChatGPT in nursing education: opportunities and chal-lenges”, Teaching and Learning in Nursing, Vol. 19 No. 1, pp. 97-101, https://doi.org/10.1016/j.teln.2023.11.004.
Benichou, L. (2023), “The role of using ChatGPT AI in writing medical scientific articles”, Journal of Stomatology, Oral and Maxillofacial Surgery, Vol. 124 No. 5, 101456, https://doi.org/10.1016/j.jormas.2023.101456.
Bin-Nashwan, S.A., Sadallah, M. and Mohamed, B. (2023), “Use of ChatGPT in academia: academic integrity hangs in the balance”, Technology in Society, Vol. 75, 102370, https://doi.org/10.1016/j.techsoc.2023.102370.
Brown, Jordan (2023). “Comparing GPT-3 vs GPT-4: An AI Expert‘s In-Depth Guide”, https://www.33rdsquare.com/comparing-gpt-3-vs-gpt-4-an-ai-experts-in-depth-guide/
Chatelan, A., Clerc, A. and Fonta, P.-A. (2023), “ChatGPT and future artificial intelligence chatbots: what may be the influence on credentialed nutrition and dietetics practitioners?”, Journal of the Academy of Nutrition and Dietetics, Vol. 123 No. 11, pp. 1525-1531, https://doi.org/10.1016/j.jand.2023.08.001.
Comrey, A.L., & Lee, H.B. (1992). A First Course in Factor Analysis (2nd ed.). Psychology Press. https://doi.org/10.4324/9781315827506.
Dell’Acqua, F., McFowland, E., Mollick, E.R., Lifshitz-Assaf, H., Kellogg, K., Rajendran, S., Krayer, L., Candelon, F. and Lakhani, K.R. (2023), “Navigating the jagged technological frontier: Field User experience in AI chat systems experimental evidence of the effects of AI on knowledge worker productivity and quality”, Harvard Business School Technology & Operations Mgt. Unit, Working Paper (24-013).
Duong, C.D., Bui, D.T., Pham, H.T., Vu, A.T. and Nguyen, V. (2023), “How effort expectan-cy and performance expectancy interact to trigger higher education students’ uses of ChatGPT for learning”, Interactive Technology and Smart Education, https://doi.org/10.1108/itse-05-2023-0096.
Febiandini, V.V. and Sony, M.S. (2023), “Analysis of public administration challenges in the development of artificial intelligence industry 4.0”, IAIC Transactions on Sustainable Digi-tal Innovation (ITSDI), Vol. 4 No. 2, pp. 164-168, https://doi.org/10.34306/itsdi.v4i2.586.
Foroughi, B., Senali, M.G., Iranmanesh, M., Khanfar, A., Ghobakhloo, M., Annamalai, N. and Naghmeh-Abbaspour, B. (2023), “Determinants of intention to use ChatGPT for educa-tional purposes: findings from PLS-SEM and fsQCA”, International Journal of Human-Computer Interaction, pp. 1-20, https://doi.org/10.1080/10447318.2023.2226495.
Golan R, Ripps S J, Reddy R, et al. (July 20, 2023) ChatGPT's Ability to Assess Quality and Readability of Online Medical Information: Evidence From a Cross-Sectional Study. Cure-us 15(7): e42214, https://doi.org/10.7759/cureus.42214.
Hair, J.F., Black, W.C., Babin, B.J. and Anderson, R.E. (2014), Multivariate Data Analysis, 7th ed., Pearson Education, Harlow.
Hasfar, M., Militina, T. and Achmad, G.N. (2020), “Effect of customer value and customer ex-perience on customer satisfaction and loyalty PT meratus samarinda”, International Journal of Economics, Business and Accounting Research (IJEBAR), Vol. 4 No. 01, https://doi.org/10.29040/ijebar.v4i01.909.
Hiwa, D., Abdalla, S., Muhialdeen, A., Hamasalih, H., Karim, S. (2024), “Assessment of Nursing Skill and Knowledge of ChatGPT, Gemini, Microsoft Copilot, and Llama: A Comparative Study”, Barw Medical Journal, https://doi.org/10.58742/bmj.v2i2.87.
Hosseini, M., Gao, C.A., Liebovitz, D.M., Carvalho, A.M., Ahmad, F.S., Luo, Y., MacDonald, N., Holmes, K.L. and Kho, A. (2023), “An exploratory survey about using ChatGPT in education, healthcare, and research”, medRxiv:2023.03. 31.23287979, Vol. 18 No. 10, e0292216, https://doi.org/10.1371/journal.pone.0292216.
Ismulyana Djan, Siti Rubbiah Adawiyyah. (2020). The Effect of Convenience and Trust to Purchase Decision and Its Impact to Customer Satisfaction. International Journal of Business and Economics Research, 9(4), 263269, https://doi.org/10.11648/j.ijber.20200904.23.
James, G., Witten, D., Hastie, T., and Tibshirani, R. (eds.). (2013). An introduction to statistical learning: with applications in R. New York: Springer.
Junghwan Kim, Jinhyung Lee, Kee Moon Jang, Ismini Lourentzou,Exploring the limitations in how ChatGPT introduces environmental justice issues in the United States: A case study of 3,108 counties, Telematics and Informatics, Volume 86, 2024, 102085, ISSN 0736-5853, https://doi.org/10.1016/j.tele.2023.102085
Joo, E. and Yang, J. (2023), "How perceived interactivity affects consumers' shopping inten-tions in live stream commerce: roles of immersion, user gratification and product involve-ment", Journal of Research in Interactive Marketing, Vol. 17 No. 5, pp. 754-772. https://doi.org/10.1108/JRIM-02-2022-0037
Kaufman, S. B. (2023). Self-Actualizing People in the 21st Century: Integration With Contem-porary Theory and Research on Personality and Well-Being. Journal of Humanistic Psy-chology, 63(1), 51-83. https://doi.org/10.1177/0022167818809187
Keebler, J.R., Shelstad, W.J., Smith, D.C., Chaparro, B.S. and Phan, M.H. (2020), “Validation of the GUESS-18: a short version of the game user experience satisfaction scale (GUESS)”, Journal of Usability Studies, Vol. 16 No. 1, p. 49.
Keshta, I. (2022), “AI-driven IoT for smart health care: security and privacy issues”, Informat-ics in Medicine Unlocked, Vol. 30, 100903, https://doi.org/10.1016/j.imu.2022.100903.
Kivijärvi, Hannu, and Karoliina Pärnänen. “Instrumental Usability and Effective User Experi-ence: Interwoven Drivers and Outcomes of Human-Computer Interaction.” International Journal of Human-Computer Interaction, vol. 39, no. 1, 2023, pp. 34–51, https://doi.org/10.1080/10447318.2021.2016236.
Kliestik, T., Nagy, M. and Valaskova, K. (2023), “Global value chains and industry 4.0 in the context of lean workplaces for enhancing company performance and its comprehension via the digital readiness and expertise of workforce in the V4 nations”, Mathematics, Vol. 11 No. 3, p. 601, https://doi.org/10.3390/math11030601.
Koo, M. (2023), “Harnessing the potential of chatbots in education: the need for guidelines to their ethical use”, Nurse Education in Practice, Vol. 68, 103590, https://doi.org/10.1016/j.nepr.2023.103590. https://dx.doi.org/10.1016/j.nepr.2023.103590 doi: 10.1016/j.nepr.2023.103590
Kopplin, C.S. (2022), “Chatbots in the workplace: a technology acceptance study applying us-es and gratifications in COWORKING spaces”, Journal of Organizational Computing and Electronic Commerce, Vol. 32 Nos 3-4, pp. 232-257, https://doi.org/10.1080/10919392.2023.2215666.
Kutner, M.H., Nachtsheim, C.J. and Neter, J. (2004), Applied Linear Regression Models, 4th ed., McGraw Hill/Irwin, Chicago, IL.
Li, Y. and Wang, C.L. (2022), “Effect of customer’s perception on service robot acceptance”, International Journal of Consumer Studies, Vol. 46 No. 4, pp. 1241-1261, https://doi.org/10.1111/ijcs.12755.
Li, H. (2023), “Rethinking human excellence in the AI age: the relationship between intellectu-al humility and attitudes toward ChatGPT”, Personality and Individual Differences, Vol. 215, 112401, https://doi.org/10.1016/j.paid.2023.112401.
Liu, P., Chen, S., Liu, J., Xu, Y., Cao, F. and Wu, G. (2023), “An efficient contradiction sepa-ration based automated deduction algorithm for enhancing reasoning capability”, Knowledge-Based Systems, Vol. 261, 110217, https://doi.org/10.1016/j.knosys.2022.110217.
Ma, X. and Huo, Y. (2023), “Are users willing to embrace ChatGPT? Exploring the factors on the acceptance of chatbots from the perspective of AIDUA framework”, Technology in So-ciety, Vol. 75, 102362, https://doi.org/10.1016/j.techsoc.2023.102362.
Malhotra, N.K. (2020), Marketing Research: An Applied Orientation, 7th ed., Pearson Educa-tion Limited, Harlow.
McCade, Simon (2024). “Chat GPT Artificial Intelligence: A UX Design Review”. https://www.simonmccade.com/blog/chat-gpt-artificial-intelligence-a-ux-design-review
Ngoc, N.M., Phuong, H.L.H., Manh, N.D., Duong, K.A., Tung, T.T., Hao, H.V. and Hieu, N.M. (2023), “Exploring continuance intention to use electric motorcycles among students in Hanoi using expectation confirmation theory”, Tạp Chı Khoa Học Giao Thong Vận Tải, Vol. 74 No. 1, pp. 58-71, https://doi.org/10.47869/tcsj.74.1.6.
Nunnally, J.C. (1994), Psychometric Theory 3E, Tata McGraw-Hill Education, New York.
Obeng, A.Y. (2023), “Consequential effects of using competing perspectives to predict learning style in e-learning systems”, Cogent Education, Vol. 10 No. 1, p. 16, https://doi.org/10.1080/2331186X.2023.2218960.
OpenAI (2022), “Introducing ChatGPT”, available at: https://openai.com/blog/chatgpt (ac-cessed 13 October).
Orden-Mejía M, Carvache-Franco M, Huertas A, Carvache-Franco O, Carvache-Franco W (2023) Modeling users’ satisfaction and visit intention using AI-based chatbots. PLoS ONE 18(9): e0286427. https://doi.org/10.1371/journal.pone.0286427.
Oscar Oviedo-Trespalacios, Amy E Peden, Thomas Cole-Hunter, Arianna Costantini, Milad Haghani, J.E. Rod, Sage Kelly, Helma Torkamaan, Amina Tariq, James David Albert Newton, Timothy Gallagher, Steffen Steinert, Ashleigh J. Filtness, Genserik Reniers, The risks of using ChatGPT to obtain common safety-related information and advice, Safety Science, Volume 167,2023, 106244, ISSN 0925-7535, https://doi.org/10.1016/j.ssci.2023.106244.
Park, J.-Y. and kim, D (2020), “The effect of shopper’s time perception and creativity on perceived usefulness and shopping happiness in omni-channel”, Journal of Channel and Retailing, Vol. 25 No. 2, pp. 99-126, https://doi.org/10.17657/jcr.2020.04.30.5.
Pei, X.-L., Guo, J.-N., Wu, T.-Ju, Zhou, W.-X. and Yeh, S.-P. (2020), “Does the effect of cus-tomer experience on customer satisfaction create a sustainable competitive advantage? A comparative study of different shopping situations”, Sustainability, Vol. 12 No. 18, p. 7436, https://doi.org/10.3390/su12187436.
Rice, S., Crouse, S.R., Winter, S.R. and Rice, C. (2024), “The advantages and limitations of using ChatGPT to enhance technological research”, Technology in Society, Vol. 76, 102426, https://doi.org/10.1016/j.techsoc.2023.102426
Romero-Rodrıguez, J.M., Ramırez-Montoya, M.S., Buenestado-Fernandez, M. and Lara-Lara, F. (2023), “Use of ChatGPT at university as a tool for complex thinking: students’ per-ceived usefulness”, Journal of New Approaches in Educational Research, Vol. 12 No. 2, pp. 323-339, https://doi.org/10.7821/naer.2023.7.1458
Ross O'Hagan, Randie H. Kim, Brian J. Abittan, Stella Caldas, Jonathan Ungar, Benjamin Ungar; Trends in Accuracy and Appropriateness of Alopecia Areata Information Obtained from a Popular Online Large Language Model, ChatGPT. Dermatology 5 December 2023; 239 (6): 952–957. https://doi.org/10.1159/000534005
Rossettini, G., Rodeghiero, L., Corradi, F., Cook, C., Pillastrini, P., Turolla, A., Castellini, G., Chiappinotto, S., Gianola, S., and Palese, A. (2024), “Comparative accuracy of ChatGPT-4, Microsoft Copilot and Google Gemini in the Italian entrance test for healthcare sciences degrees: a cross-sectional study”, BMC Medical Education 24, 694, https://doi.org/10.1186/s12909-024-05630-9.
Saha, S.K., Duarte, P., Silva, S.C. and Zhuang, G. (2023), “The role of online experience in the relationship between service convenience and future purchase intentions”, Journal of Internet Commerce, Vol. 22 No. 2, pp. 244-271, https://doi.org/10.1080/15332861.2022.2045767.
Sallam, M., Salim, N.A., Barakat, M., Al-Mahzoum, K., Al-Tammemi, A.’a B., Malaeb, D., Hallit, R. and Hallit, S. (2023), “Assessing health students’ attitudes and usage of ChatGPT in Jordan: Validation study”, JMIR Medical Education, Vol. 9, e48254, https://doi.org/10.2196/48254.
Shahsavar, Y. and Choudhury, A. (2023), “User intentions to use ChatGPT for self-diagnosis and health-related purposes: cross-sectional survey study”, Jmir Human Factors, Vol. 10, p. e47564, https://doi.org/10.2196/47564.
Shen, Y., Heacock, L., Elias, J., Hentel, K.D., Reig, B., Shih, G. and Moy, L. (2023), “ChatGPT and other large language models are double-edged swords”, Radiology, Vol. 307 No. 2, e230163, https://doi.org/10.1148/radiol.230163.
Singla, Alex, Alexander Sukharevsky, Lareina Yee, and Michael Chui, Bryce Hall(2024), “The state of AI in early 2024: Gen AI adoption spikes and starts to generate value”, https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai#/
Sun, Y., Chen, J. and Sundar, S.S. (2024), “Chatbot ads with a human touch: a test of anthro-pomorphism, interactivity, and narrativity”, Journal of Business Research, Vol. 172, 114403, https://doi.org/10.1016/j.jbusres.2023.114403.
Wang, M. (2012), “Determinants and consequences of consumer satisfaction with self-service technology in a retail setting”, Managing Service Quality, Vol. 22 Iss: 2 pp. 128 – 144, https://doi.org/10.1108/09604521211218945.
Wang, Z., Deng, R. and Jiang, Q.-ling (2022a), “Exploration and study of the factors influenc-ing users’ adoption of games for fitness behavior”, International Journal of Human–Computer Interaction, Vol. 39 No. 17, pp. 1-12, https://doi.org/10.1080/10447318.2022.2097985.
Wang, C.L., Dai, J., Zhu, K.K., Yu, T. and Gu, X.Q. (2023a), “Understanding the continuance intention of college students toward new E-learning spaces based on an integrated model of the TAM and TTF”, International Journal of Human-Computer Interaction, Vol. 14, pp. 1-14, https://doi.org/10.1080/10447318.2023.2291609.
Wang, Y., Xu, Z.-L., Lou, J.-Y. and Chen, K.-Da (2023c), “Factors influencing the complex problem-solving skills in reflective learning: results from partial least square structural equa-tion modeling and fuzzy set qualitative comparative analysis”, BMC Medical Education, Vol. 23 No. 1, pp. 1-17, https://doi.org/10.1186/s12909-023-04326-w.
Xing, J., Jiang, Q. (2024), “Factors influencing user experience in AI chat systems – a satisfac-tion study based on factor analysis and linear regression”, Kybernetes, 0368-492X, https://doi.org/10.1108/K-10-2023-2237.
Xue, V.W., Lei, P. and Cho, W.C. (2023), “The potential impact of ChatGPT in clinical and translational medicine”, Clinical and Translational Medicine, Vol. 13 No. 3, e1216, https://doi.org/10.1002/ctm2.1216.
Zheng, L. (2020), "The role of consumption emotions in users’ mobile gaming application con-tinuance intention", Information Technology & People, Vol. 33 No. 1, pp. 340-360, https://doi.org/10.1108/ITP-04-2018-0197.
Zhu, D.H., He, Q. and Chang, Y.P. (2021), “How SoLoMo-based product recommendations influence consumers’ acceptance intention: the moderating role of gender”, International Journal of Services Technology and Management, Vol. 27 Nos 1-2, pp. 129-142, https://doi.org/10.1504/ijstm.2021.113579.
Downloads
Submitted
Accepted
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Danny Hutagalung, Nadia Sancayawati, Ria Aryanty, Tommy Hutabarat, Widhianto Almumin (Author)
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Journal Author(s) Rights
For IRJBS to publish and disseminate research articles, we need publishing rights (transferred from the author(s) to the publisher). This is determined by a publishing agreement between the Author(s) and IRJBS. This agreement deals with the transfer or license of the copyright of publishing to IRJBS, while Authors still retain significant rights to use and share their own published articles. IRJBS supports the need for authors to share, disseminate and maximize the impact of their research and these rights, in any databases.
As a journal Author, you have rights to many uses of your article, including use by your employing institute or company. These Author rights can be exercised without the need to obtain specific permission. Authors publishing in IRJBS journals have comprehensive rights to use their works for teaching and scholarly purposes without needing to seek permission, including:
- use for classroom teaching by Author or Author's institution and presentation at a meeting or conference and distributing copies to attendees;
- use for internal training by the author's company;
- distribution to colleagues for their research use;
- use in a subsequent compilation of the author's works;
- inclusion in a thesis or dissertation;
- reuse of portions or extracts from the article in other works (with full acknowledgment of the final article);
- preparation of derivative works (other than commercial purposes) (with full acknowledgment of the final article);
- voluntary posting on open websites operated by the author or the author’s institution for scholarly purposes,
(But it should follow the open access license of Creative Common CC-by-SA License).
Authors/Readers/Third Parties can copy and redistribute the material in any medium or format, as well as remix, transform, and build upon the material for any purpose, even commercially. Still, they must give appropriate credit (the name of the creator and attribution parties (authors' detail information), a copyright notice, an open access license notice, a disclaimer notice, and a link to the material), provide a link to the license, and indicate if changes were made (Publisher indicates the modification of the material (if any) and retain an indication of previous modifications.
Authors/Readers/Third Parties can read, print and download, redistribute or republish the article (e.g. display in a repository), translate the article, download for text and data mining purposes, reuse portions or extracts from the article in other works, sell or re-use for commercial purposes, remix, transform, or build upon the material, they must distribute their contributions under the same license as the original Creative Commons Attribution-ShareAlike (CC BY-SA).
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
