Hailong Jiang

Hailong Jiang

he/him

Tenure Track Assistant Professor in Computer Science at Youngstown State University. Research focuses on HPC systems, compiler technologies, and the intersection of program analysis and large language models (LLMs).

LLM Understanding of Intermediate Representations

Published:

Project Overview

This research project investigates the capabilities of Large Language Models (LLMs) in understanding Intermediate Representations (IRs), which are essential in compiler design and program analysis. The study addresses a critical gap in our understanding of how modern AI models handle low-level code representations.

Research Objectives

  • IR Comprehension Analysis: Evaluate LLM understanding of IR syntax and semantics
  • Task Performance Assessment: Test LLMs across four critical IR-related tasks
  • Model Comparison: Analyze performance differences between various LLM architectures
  • Enhancement Recommendations: Propose improvements for IR-specific LLM capabilities

Methodology

  • Models Tested: GPT-4, GPT-3, Gemma 2, LLaMA 3.1, and Code Llama
  • Evaluation Tasks: Control Flow Graph (CFG) reconstruction, decompilation, code summarization, and execution reasoning
  • Analysis Framework: Comprehensive empirical study with structured evaluation metrics
  • IR Dataset: Diverse Intermediate Representation samples for testing

Key Findings

  • Strengths: LLMs demonstrate competence in parsing IR syntax and recognizing high-level structures
  • Limitations: Struggle with control flow reasoning, execution semantics, and loop handling
  • Common Errors: Misinterpretation of branching instructions, omission of critical IR operations
  • Reasoning Patterns: Heavy reliance on heuristic-based reasoning rather than deep understanding

Research Impact

This study provides foundational insights into LLM capabilities for compiler-level tasks and program analysis, with implications for:

  • AI-assisted compiler development
  • Program analysis automation
  • LLM training for low-level code understanding
  • Integration of AI in software engineering tools

Technical Contributions

  • Novel evaluation framework for IR comprehension
  • Comprehensive analysis of LLM limitations in IR tasks
  • Specific recommendations for IR-specific LLM enhancements
  • Framework for future research in AI-driven program analysis

Future Directions

The research identifies several areas for improvement:

  • IR-specific fine-tuning on structured datasets
  • Integration of explicit control flow models
  • Enhanced training on compiler-level tasks
  • Development of specialized IR understanding models

Publication

This work has been published as an arXiv preprint: "Can Large Language Models Understand Intermediate Representations?"