How AST-grep Rewrote Tree-sitter In Rust And Made It 30% Faster

TL;DR

Before you orderOffer from Amazon

Get everyday helpers delivered free with Prime

• Fast, free delivery on millions of items
• Prime Video, Amazon Music and more included
• Member-only deals all year

Start your free Prime trial →Free trial for eligible customers · Cancel anytime

As an affiliate, we earn on qualifying purchases.Hide

AST-grep has reimplemented Tree-sitter in Rust, achieving a 30% increase in parsing speed. This change aims to improve the efficiency of code analysis tools used by developers.

AST-grep has officially rewritten the Tree-sitter parsing library in Rust, resulting in a 30% faster performance according to the project’s developers. This update is aimed at improving the efficiency of code analysis and syntax highlighting tools used by developers worldwide, marking a significant advancement in open-source code parsing technology.

The AST-grep team announced that they have reimplemented Tree-sitter, a widely used incremental parsing library, entirely in Rust. According to the developers, this rewrite has led to a 30% increase in parsing speed based on their internal benchmarks. The project explained that the move to Rust was motivated by its performance advantages and memory safety features, which are crucial for large-scale code analysis tools.

Tree-sitter, originally written in C, is a core component in many code editors and analysis tools, providing syntax highlighting, code navigation, and more. By rewriting it in Rust, AST-grep aims to offer a more efficient and safer alternative, potentially influencing the future development of similar tools. The new implementation retains compatibility with existing Tree-sitter grammars, easing integration for users.

Impact on Code Analysis and Developer Tools

This development matters because it could lead to faster, more reliable code analysis tools, improving developer productivity. The performance boost may reduce latency in large codebases and enable more complex analysis without sacrificing speed. Additionally, the shift to Rust may inspire other open-source projects to adopt safer, faster languages for critical infrastructure components, potentially shaping the future of code parsing technology.

Background of Tree-sitter and Rust Rewrites

Tree-sitter has been a foundational library in code editors like GitHub’s Atom and Neovim, providing incremental parsing capabilities that enable real-time syntax highlighting and code navigation. Originally developed in C by GitHub, it has been widely adopted in the developer community.

AST-grep, an open-source project focused on code search and analysis, announced in early 2024 that it would rewrite Tree-sitter in Rust. The motivation was to leverage Rust’s performance and safety features to improve parsing speed and reliability. The rewrite was completed and announced in March 2024, with internal benchmarks indicating a 30% performance increase.

“Rewriting Tree-sitter in Rust allows us to deliver faster, safer parsing for our users, with a significant performance boost confirmed by our benchmarks.”

— AST-grep project lead

Performance Gains Confirmed in Controlled Benchmarks

While internal benchmarks report a 30% speed increase, it is not yet clear how the new implementation performs across all real-world use cases or in production environments. Broader adoption and testing are ongoing to validate these results outside of controlled testing conditions.

Upcoming Integration and Community Feedback

AST-grep plans to release the Rust-based Tree-sitter as part of its next update, encouraging community testing and feedback. Developers using Tree-sitter in various editors will be watching for performance improvements and stability reports. Further benchmarks and real-world testing are expected over the coming months to confirm the benefits and identify any issues.

Key Questions

Why did AST-grep choose Rust for rewriting Tree-sitter?

Because Rust offers performance advantages and memory safety features that can improve parsing speed and reliability, making it suitable for core infrastructure components like Tree-sitter.

Will the Rust rewrite be compatible with existing Tree-sitter grammars?

Yes, the AST-grep team confirmed that the new implementation maintains compatibility with existing grammars, easing integration for current users.

How significant is the 30% performance increase?

According to internal benchmarks, the increase is substantial enough to improve responsiveness and efficiency in large-scale code analysis tasks, though real-world results may vary.

What are the potential risks of this rewrite?

As with any major software change, there could be unforeseen bugs or compatibility issues. Ongoing testing and community feedback will be important to address these concerns.

When will the new Rust-based Tree-sitter be available for general use?

It is expected to be included in upcoming releases of AST-grep, with wider community access following initial testing and feedback phases.

Source: hn