Python - Python Abstract Syntax Trees (AST) and Code Analysis

Introduction

An Abstract Syntax Tree (AST) is a structured representation of Python source code. When Python reads a program, it does not simply treat the code as a sequence of characters. It analyzes the structure of the code and represents different elements, such as variables, functions, expressions, loops, and conditions, in a tree-like structure.

Python provides the built-in ast module for working with Abstract Syntax Trees. Developers can use this module to inspect Python code, understand its structure, perform static analysis, detect specific coding patterns, and even transform parts of a program without executing it.

For example, consider this simple Python statement:

total = price + tax

The AST represents this statement as different connected components. The assignment is one node, total is represented as a variable node, and price + tax becomes a binary operation containing two variable nodes.

This makes AST particularly useful when a program needs to understand what code means structurally rather than simply treating it as text.

What Is an Abstract Syntax Tree?

An Abstract Syntax Tree is a hierarchical representation of the syntax of a program.

The word "abstract" means that the tree does not preserve every detail of the original source code. For example, unnecessary spaces, indentation formatting, and comments are generally not represented as normal syntax nodes.

The word "syntax" refers to the rules governing how Python code is written.

The word "tree" describes the hierarchical relationship between different elements of the program.

Consider:

x = 10

Conceptually, the AST contains an assignment node with two important parts:

Assignment
├── Target: x
└── Value: 10

Here:

Assignment represents the assignment operation.
x represents the variable receiving the value.
10 represents the constant value.

For a more complex statement:

result = a + b * 2

the structure becomes hierarchical:

Assignment
├── Target: result
└── Value: Add
    ├── a
    └── Multiply
        ├── b
        └── 2

The tree structure also reflects Python's operator precedence. Multiplication is evaluated before addition, so b * 2 forms a nested operation within the addition.

The Python `ast` Module

Python includes the ast module as part of its standard library. It allows programs to parse Python source code and work with its syntax tree.

A simple example is:

import ast

code = "x = 10"

tree = ast.parse(code)

print(ast.dump(tree, indent=4))

The ast.parse() function converts Python source code into an AST.

The resulting tree contains nodes representing the module, assignment, variable, and constant.

ast.dump() is useful for displaying the internal structure of the tree in a readable form.

Parsing Python Source Code

The first step in AST-based analysis is usually parsing.

Consider:

code = """
x = 10
y = 20
total = x + y
"""

tree = ast.parse(code)

The tree variable now contains the AST representation of the complete program.

The source code has not been executed. This is an important characteristic of AST analysis.

For example:

code = """
print("Hello")
"""

tree = ast.parse(code)

Parsing this code does not print Hello. The code is only converted into a structural representation.

This makes AST useful for analyzing potentially large amounts of source code without running that code.

Common AST Node Types

The ast module contains many node types representing different Python constructs.

Some commonly encountered nodes include:

AST Node	Represents
`Module`	Complete Python source
`FunctionDef`	Function definition
`ClassDef`	Class definition
`Assign`	Assignment
`Name`	Variable or identifier
`Constant`	Literal value
`Call`	Function or method call
`Return`	Return statement
`If`	Conditional statement
`For`	For loop
`While`	While loop
`Import`	Import statement
`ImportFrom`	`from ... import ...`
`BinOp`	Binary operation
`Compare`	Comparison
`Attribute`	Attribute access
`List`	List expression
`Dict`	Dictionary expression

Understanding these nodes is essential for building AST-based analysis tools.

Examining an AST

The ast.dump() function provides a convenient way to inspect a tree.

For example:

import ast

code = "x = 5 + 10"

tree = ast.parse(code)

print(ast.dump(tree, indent=4))

The output contains nested nodes representing:

the module
the assignment
the variable x
the addition operation
the constants 5 and 10

This allows developers to see how Python internally structures the statement.

Traversing an AST

After creating an AST, a program can traverse its nodes.

One of the simplest approaches is ast.walk().

import ast

code = """
x = 10
y = 20
print(x + y)
"""

tree = ast.parse(code)

for node in ast.walk(tree):
    print(type(node).__name__)

This visits nodes throughout the syntax tree.

AST traversal can be used to answer questions such as:

How many functions does a file contain?
Which modules are imported?
Are particular functions being called?
How many loops exist?
Which variables are assigned?
Does the code contain a particular programming pattern?

Using `NodeVisitor`

For more organized analysis, Python provides ast.NodeVisitor.

A custom visitor class can define methods for specific node types.

For example:

import ast

class FunctionVisitor(ast.NodeVisitor):
    def visit_FunctionDef(self, node):
        print("Function:", node.name)
        self.generic_visit(node)

code = """
def add(a, b):
    return a + b

def multiply(a, b):
    return a * b
"""

tree = ast.parse(code)

visitor = FunctionVisitor()
visitor.visit(tree)

The visitor identifies function definitions without executing the program.

This technique is particularly useful for source-code analysis tools.

Analyzing Function Definitions

AST analysis can extract information about functions.

Consider:

def calculate(a, b, c):
    return a + b + c

An AST analyzer can determine:

the function name
the number of parameters
whether the function has a return statement
what operations it performs
which functions it calls
whether nested functions exist

For example:

import ast

class FunctionAnalyzer(ast.NodeVisitor):
    def visit_FunctionDef(self, node):
        print("Function:", node.name)
        print("Arguments:", len(node.args.args))
        self.generic_visit(node)

code = """
def calculate(a, b, c):
    return a + b + c
"""

tree = ast.parse(code)
FunctionAnalyzer().visit(tree)

This type of analysis can be extended to create automated code-quality tools.

Detecting Imports

AST can also identify imported modules.

For example:

import os
import json
from pathlib import Path

An analyzer can inspect Import and ImportFrom nodes.

This can be useful when auditing a project to determine:

which libraries are used
whether unwanted modules are imported
whether deprecated modules are still present
which files depend on particular packages

Detecting Function Calls

Consider:

print("Hello")
open("data.txt")
calculate()

Each function invocation is represented by a Call node.

An AST analyzer can inspect these calls and identify the functions being used.

This can be useful for detecting potentially problematic coding patterns.

For example, an organization might want to identify use of a particular function throughout a large codebase.

Static Code Analysis

One of the most important applications of AST is static code analysis.

Static analysis examines source code without executing it.

For example, a tool can search for:

if x == True:

and identify it as a potentially unnecessary comparison.

Another analyzer might identify unused imports, overly complex functions, suspicious constructs, or prohibited function calls.

AST provides the structural information needed to build such tools.

Many code-quality systems use concepts similar to AST analysis to understand source code.

AST-Based Security Analysis

AST can also be useful for security-oriented source analysis.

For example, an organization could scan source code for potentially dangerous constructs such as certain function calls or unsafe coding patterns.

Suppose a policy prohibits a particular operation:

eval(user_input)

An AST analyzer can search for Call nodes where the called function is eval.

The analyzer does not need to execute the program to identify the pattern.

However, AST analysis alone does not guarantee that code is safe. Determining security risks often requires additional analysis, because dangerous behavior can be hidden behind variables, imported functions, dynamic execution, or other techniques.

Modifying Python Code with AST

AST is not limited to reading code. Python also provides mechanisms for modifying syntax trees.

For example, an analyzer could locate a particular expression and replace it with another expression.

The general process is:

Python Source Code
        |
        v
     Parse
        |
        v
      AST
        |
        v
   Analyze/Modify
        |
        v
   Generate Code

A modified AST can then be converted back into Python source code.

This is useful for automated code transformations and refactoring tools.

Using `NodeTransformer`

Python provides ast.NodeTransformer for modifying AST nodes.

For example:

import ast

class NumberTransformer(ast.NodeTransformer):
    def visit_Constant(self, node):
        if isinstance(node.value, int):
            return ast.Constant(value=node.value * 2)
        return node

code = "x = 10"

tree = ast.parse(code)

transformer = NumberTransformer()
new_tree = transformer.visit(tree)

ast.fix_missing_locations(new_tree)

In this example, integer constants can be transformed.

The important point is that the transformation operates on the syntax tree rather than performing a simple text replacement.

Converting an AST Back to Python Code

After modifying an AST, Python provides ast.unparse() in modern Python versions.

For example:

import ast

code = "x = 10 + 20"

tree = ast.parse(code)

new_code = ast.unparse(tree)

print(new_code)

This produces Python source code from the AST.

A typical transformation workflow therefore looks like:

Source Code
     |
     v
ast.parse()
     |
     v
AST
     |
     v
Analyze or Transform
     |
     v
ast.unparse()
     |
     v
Modified Source Code

AST Versus Regular Text Processing

A common question is why AST should be used instead of searching through source code as plain text.

Consider:

print("eval() is mentioned here")

A simple text search might detect the string eval() and incorrectly conclude that the program calls eval.

AST analysis understands the difference between a string containing text and an actual function call.

For example:

eval(data)

contains a Call node.

But:

message = "eval(data)"

contains a string constant.

This structural understanding makes AST-based analysis much more reliable than basic text searching for many programming-language tasks.

Practical Applications of AST

AST has numerous practical applications.

Code Quality Tools

AST can identify:

unnecessary constructs
excessive nesting
complex functions
specific coding patterns
duplicate structural patterns

Automated Refactoring

Development tools can use syntax trees to safely identify and modify code structures.

Linters

Linters can inspect source code and report violations of coding standards.

Dependency Analysis

AST can identify imported modules and help understand relationships between source files.

Educational Tools

AST can help programming education platforms visualize how Python interprets different language constructs.

Code Metrics

AST can be used to calculate metrics such as:

number of functions
number of classes
number of conditional statements
number of loops
number of imports
nesting depth

Code Migration

AST-based transformations can assist in automatically updating source code when APIs or coding conventions change.

Limitations of AST Analysis

Although AST is powerful, it has limitations.

First, AST describes the syntactic structure of code rather than its complete runtime behavior.

For example:

function_name = "calculate"

and dynamically calling a function through a variable may be difficult to understand purely from the AST.

Second, AST does not automatically tell you what every expression will produce at runtime.

Third, dynamically generated code can make static analysis more difficult.

Fourth, converting an AST back to source code may not preserve the exact original formatting, comments, or stylistic choices.

Therefore, AST is best considered one component of a larger code-analysis system.

AST and the Python Compiler

AST also plays an important role in Python's processing pipeline.

A simplified view is:

Python Source
      |
      v
Lexing and Parsing
      |
      v
Abstract Syntax Tree
      |
      v
Compilation
      |
      v
Bytecode
      |
      v
Python Virtual Machine

The AST represents the logical structure of the program before it is compiled into bytecode.

This explains why AST provides a powerful intermediate representation for understanding Python programs.

Important Safety Consideration

The ast.parse() function parses Python source code, but parsing is not the same as executing it.

For example:

tree = ast.parse(untrusted_code)

does not normally execute the parsed statements.

However, developers should still be careful when processing untrusted input, particularly if the workflow later compiles, evaluates, imports, or executes the resulting code.

AST should not be treated as a complete security sandbox.

Conclusion

Python Abstract Syntax Trees provide a structured way to understand Python programs without relying solely on text processing. The ast module allows developers to parse source code, inspect its structure, identify specific programming constructs, perform static analysis, and transform code programmatically.

AST is particularly valuable for building linters, code-quality analyzers, automated refactoring tools, dependency analyzers, educational applications, security scanners, and source-code transformation utilities.

The key idea is simple: instead of asking, "What characters are present in this Python file?", AST analysis allows a program to ask, "What programming structures are present in this Python code?" This structural understanding makes AST an important concept for advanced Python development and automated code analysis.

Python - Python Abstract Syntax Trees (AST) and Code Analysis

Introduction

What Is an Abstract Syntax Tree?

The Python ast Module

Parsing Python Source Code

Common AST Node Types

Examining an AST

Traversing an AST

Using NodeVisitor

Analyzing Function Definitions

Detecting Imports

Detecting Function Calls

Static Code Analysis

AST-Based Security Analysis

Modifying Python Code with AST

Using NodeTransformer

Converting an AST Back to Python Code

AST Versus Regular Text Processing

Practical Applications of AST

Code Quality Tools

Automated Refactoring

Linters

Dependency Analysis

Educational Tools

Code Metrics

Code Migration

Limitations of AST Analysis

AST and the Python Compiler

Important Safety Consideration

Conclusion

The Python `ast` Module

Using `NodeVisitor`

Using `NodeTransformer`