Evaluation of GPT-generated conceptual models based on verbal descriptions: accuracy and quality analysis

Abstract

This paper examines the use of GPT for extracting entities and relationships from textual descriptions of conceptual models. Using ten models of varying complexity, outputs were evaluated for consistency and accuracy, with Jaccard similarity measuring uniformity. GPT exhibited strong entity recognition but struggled with implicit relationships, often requiring explicit definitions for accurate interpretation. It also occasionally misclassified attributes as entities, resulting in minor inconsistencies. While parameter adjustments had minimal impact, clear and detailed input significantly improved reliability. The findings highlight GPT’s potential in automating conceptual modeling, with future work focused on improving input clarity and consistency.