Evaluating the Role of Heuristics in LLM-Based Extraction of Entity-Relationship Models

Abstract

Translating natural language requirements into precise conceptual models is a challenging and very important step in software development. This paper investigates the effectiveness of combining modelling heuristics with large language models for extracting Extended Entity-Relationship (EER) models from text. Two prompting strategies were compared: zero-shot LLM prompting and heuristic-guided prompting. The evaluation, based on 14 manually annotated models, compares precision, recall, and F1-score for modeling constructs: entities, attributes, and relationships. Results suggest that integrating heuristics can improve the reliability and consistency of LLM-generated models, particularly in entity and attribute extraction. Although the heuristic-guided method requires more processing time, it shows stronger alignment with established modeling practices and offers better resilience to textual ambiguity. These findings indicate that heuristics can provide valuable structure and guidance to LLMs, and highlight the potential for further refinement to address recurring errors and improve performance across diverse domains.