Shan Huang
Ph.D. Candidate
Computer Science Department
University of Illinois Urbana-Champaign
Joint work with Jeffrey Herman and Alan Sherman, et al.
12 noon–1pm
Friday, January 31, 2024
Remotely via WebEx: https://umbc.webex.com/meet/sherman
Abstract:
We evaluated the performance of five LLMs (Llama a, GPT-3.5-turbo, GPT-4, GPT-4O, and GPT-O1) on two
cybersecurity concept inventories: Cybersecurity Concept Inventory (CCI) and Cybersecurity Curriculum Assessment
(CCA). Using a zero-shot setting to minimize external influencing factors, we compared the performance of these LLMs
with that of students previously studied, and we conducted a qualitative analysis of GPT-O1’s output to examine if it
exhibits misconceptions. Quantitative analysis reveals that, for the CCI and CCA, GPT-O1 significantly outperformed
other models and students, correctly answering 92% of CCI and 72% of CCA test items. These results indicate GPT-O1’s
strong proficiency in foundational topics (CCI) but reveal its limitations in addressing these concepts in more technically
advanced scenarios (CCA). Qualitative analysis of GPT-O1’s reasoning patterns uncovered instances of insightful
reasoning but also highlighted ways in which GPT-O1’s answers reflect persistent student mistakes, such as biases,
overgeneralizations, and logical inconsistencies. This work highlights the significant potential of GPT-O1 as a tool for
introductory cybersecurity education in its ability to provide detailed explanations and structured reasoning for novice
learners.
About the Speaker:
Shan Huang (sh69@illinois.edu) is a Ph.D. candidate in Computer Science at the University of Illinois Urbana-
Champaign. She is broadly interested in how educational games can improve student learning. Current work includes
improving student learning in cybersecurity with educational games and accessing student knowledge of cybersecurity
concepts. Shan is also involved in various educational data mining projects.
Host:
Alan T. Sherman, sherman@umbc.edu
Support for this event was provided in part by the National Science Foundation under SFS grant DGE-1753681.
The UMBC Cyber Defense Lab meets biweekly Fridays 12-1pm. All meetings are open to the public.
Upcoming CDL meetings:
Feb 14 (Shengwei An will speak to CSEE at 11:30am)
Feb 28, Justin M. Pelletier,
Mar 14,
(Mar 17-21 spring break)
Mar 28, Christian Badolato, 2025 UMBC SFS Research Study
Apr 11, Keke Chen (UMBC), Adaptive Domain Inference Attack
(May 2 – CSEE Research Day)
May 9