Add Lua bindings for building stack graphs #351

dcreager · 2023-11-18T23:38:00Z

We're using the mlua crate, since it seems most actively maintained and performant.

github-actions · 2023-11-18T23:43:44Z

Performance Summary

Comparing base 3696992 with head bc424cd on typescript_benchmark benchmark. For details see workflow artifacts. Note that performance is tested on the last commits with changes in stack-graphs, not on every commit.

Before

--------------------------------------------------------------------------------
Command:            base/target/release/tree-sitter-stack-graphs-typescript index -D base.sqlite --max-file-time=30 --hide-error-details -- base/data/typescript_benchmark
Massif arguments:   --massif-out-file=perf.out
ms_print arguments: --x=72 --y=12 base-perf-results/perf.out
--------------------------------------------------------------------------------


    GB
3.276^                                                                     :# 
     |                                                                     :# 
     |                                                                     :# 
     |                                                                     :#:
     |                                                                     :#:
     |                                                                     :#:
     |                                             :::@                : :::#:
     |                                             :  @                :@: :#:
     |                                        ::@@@:  @               ::@: :#:
     |                                        : @  :  @               ::@: :#:
     |                                 @@   ::: @  :  @         :@   :::@: :#:
     |              : :::@     :     @:@ : @::: @  :  @       :::@::::::@: :#:
   0 +----------------------------------------------------------------------->Gi
     0                                                                   75.09

After

--------------------------------------------------------------------------------
Command:            head/target/release/tree-sitter-stack-graphs-typescript index -D head.sqlite --max-file-time=30 --hide-error-details -- head/data/typescript_benchmark
Massif arguments:   --massif-out-file=perf.out
ms_print arguments: --x=72 --y=12 head-perf-results/perf.out
--------------------------------------------------------------------------------


    GB
3.276^                                                                     :# 
     |                                                                     :# 
     |                                                                     :# 
     |                                                                     :#:
     |                                                                     :#:
     |                                                                     :#:
     |                                             :::@                 ::::#:
     |                                             :  @                 :: :#:
     |                                        ::@@@:  @               @ :: :#:
     |                                        : @  :  @               @::: :#:
     |                                @:@   ::: @  :  @         @:   :@::: :#:
     |              : :::@     :    ::@:@:::::: @  :  @       ::@:::::@::: :#:
   0 +----------------------------------------------------------------------->Gi
     0                                                                   74.99

github-actions · 2023-11-19T00:03:31Z

Performance Summary

Comparing base 3696992 with head 9a18451 on typescript_benchmark benchmark. For details see workflow artifacts. Note that performance is tested on the last commits with changes in stack-graphs, not on every commit.

Before

--------------------------------------------------------------------------------
Command:            base/target/release/tree-sitter-stack-graphs-typescript index -D base.sqlite --max-file-time=30 --hide-error-details -- base/data/typescript_benchmark
Massif arguments:   --massif-out-file=perf.out
ms_print arguments: --x=72 --y=12 base-perf-results/perf.out
--------------------------------------------------------------------------------


    GB
3.276^                                                                     :# 
     |                                                                     :# 
     |                                                                     :# 
     |                                                                     :#:
     |                                                                     :#:
     |                                                                     :#:
     |                                             :::@                : :::#:
     |                                             :  @                :@: :#:
     |                                        ::@@@:  @               ::@: :#:
     |                                        : @  :  @               ::@: :#:
     |                                 @@   ::: @  :  @         :@   :::@: :#:
     |              : :::@     :     @:@ : @::: @  :  @       :::@::::::@: :#:
   0 +----------------------------------------------------------------------->Gi
     0                                                                   75.09

After

--------------------------------------------------------------------------------
Command:            head/target/release/tree-sitter-stack-graphs-typescript index -D head.sqlite --max-file-time=30 --hide-error-details -- head/data/typescript_benchmark
Massif arguments:   --massif-out-file=perf.out
ms_print arguments: --x=72 --y=12 head-perf-results/perf.out
--------------------------------------------------------------------------------


    GB
3.276^                                                                     :# 
     |                                                                     :# 
     |                                                                     :# 
     |                                                                     :#:
     |                                                                     :#:
     |                                                                     :#:
     |                                             :::@                : :::#:
     |                                             :  @                :@: :#:
     |                                        ::@@@:  @               ::@: :#:
     |                                        : @  :  @               ::@: :#:
     |                                @@    ::: @  :  @         :@   :::@: :#:
     |              :::::@     :     :@@:::@::: @  :  @       :::@::::::@: :#:
   0 +----------------------------------------------------------------------->Gi
     0                                                                   75.10

github-actions · 2023-11-19T00:19:10Z

Performance Summary

Comparing base 3696992 with head 5a22bf7 on typescript_benchmark benchmark. For details see workflow artifacts. Note that performance is tested on the last commits with changes in stack-graphs, not on every commit.

Before

--------------------------------------------------------------------------------
Command:            base/target/release/tree-sitter-stack-graphs-typescript index -D base.sqlite --max-file-time=30 --hide-error-details -- base/data/typescript_benchmark
Massif arguments:   --massif-out-file=perf.out
ms_print arguments: --x=72 --y=12 base-perf-results/perf.out
--------------------------------------------------------------------------------


    GB
3.276^                                                                     :# 
     |                                                                     :# 
     |                                                                     :# 
     |                                                                     :#:
     |                                                                     :#:
     |                                                                     :#:
     |                                             :::@                : :::#:
     |                                             :  @                :@: :#:
     |                                        ::@@@:  @               ::@: :#:
     |                                        : @  :  @               ::@: :#:
     |                                 @@   ::: @  :  @         :@   :::@: :#:
     |              : :::@     :     @:@ : @::: @  :  @       :::@::::::@: :#:
   0 +----------------------------------------------------------------------->Gi
     0                                                                   75.09

After

--------------------------------------------------------------------------------
Command:            head/target/release/tree-sitter-stack-graphs-typescript index -D head.sqlite --max-file-time=30 --hide-error-details -- head/data/typescript_benchmark
Massif arguments:   --massif-out-file=perf.out
ms_print arguments: --x=72 --y=12 head-perf-results/perf.out
--------------------------------------------------------------------------------


    GB
3.276^                                                                   ###  
     |                                                                   #    
     |                                                                   #   :
     |                                                                   #  ::
     |                                                                   #  ::
     |                                                                   #  ::
     |                                           :::@                ::::#  ::
     |                                           :  @                ::  #  ::
     |                                      :::@@:  @              ::::  #  ::
     |                                      : :@ :  @              : ::  #  ::
     |                               @@   ::: :@ :  @        @::   : ::  #  ::
     |             :  ::@     :    ::@@:::::: :@ :  @       :@:::::: ::  #  ::
   0 +----------------------------------------------------------------------->Gi
     0                                                                   78.06

github-actions · 2023-11-19T01:24:02Z

Performance Summary

Comparing base 3696992 with head 58a0ca0 on typescript_benchmark benchmark. For details see workflow artifacts. Note that performance is tested on the last commits with changes in stack-graphs, not on every commit.

Before

--------------------------------------------------------------------------------
Command:            base/target/release/tree-sitter-stack-graphs-typescript index -D base.sqlite --max-file-time=30 --hide-error-details -- base/data/typescript_benchmark
Massif arguments:   --massif-out-file=perf.out
ms_print arguments: --x=72 --y=12 base-perf-results/perf.out
--------------------------------------------------------------------------------


    GB
3.276^                                                                     :# 
     |                                                                     :# 
     |                                                                     :# 
     |                                                                     :#:
     |                                                                     :#:
     |                                                                     :#:
     |                                             :::@                : :::#:
     |                                             :  @                :@: :#:
     |                                        ::@@@:  @               ::@: :#:
     |                                        : @  :  @               ::@: :#:
     |                                 @@   ::: @  :  @         :@   :::@: :#:
     |              : :::@     :     @:@ : @::: @  :  @       :::@::::::@: :#:
   0 +----------------------------------------------------------------------->Gi
     0                                                                   75.09

After

--------------------------------------------------------------------------------
Command:            head/target/release/tree-sitter-stack-graphs-typescript index -D head.sqlite --max-file-time=30 --hide-error-details -- head/data/typescript_benchmark
Massif arguments:   --massif-out-file=perf.out
ms_print arguments: --x=72 --y=12 head-perf-results/perf.out
--------------------------------------------------------------------------------


    GB
3.276^                                                                     :# 
     |                                                                     :# 
     |                                                                     :# 
     |                                                                     :#:
     |                                                                     :#:
     |                                                                     :#:
     |                                                                 :@:::#:
     |                                               @                 :@: :#:
     |                                         :::@@@@                 :@: :#:
     |                                         :  @  @                ::@: :#:
     |                                   @   :::  @  @         @::  :@::@: :#:
     |              : ::::@     :      @:@::@:::  @  @       ::@:::::@::@: :#:
   0 +----------------------------------------------------------------------->Gi
     0                                                                   71.62

github-actions · 2023-11-19T02:12:14Z

Performance Summary

Comparing base 3696992 with head 5fcd36b on typescript_benchmark benchmark. For details see workflow artifacts. Note that performance is tested on the last commits with changes in stack-graphs, not on every commit.

Before

--------------------------------------------------------------------------------
Command:            base/target/release/tree-sitter-stack-graphs-typescript index -D base.sqlite --max-file-time=30 --hide-error-details -- base/data/typescript_benchmark
Massif arguments:   --massif-out-file=perf.out
ms_print arguments: --x=72 --y=12 base-perf-results/perf.out
--------------------------------------------------------------------------------


    GB
3.276^                                                                     :# 
     |                                                                     :# 
     |                                                                     :# 
     |                                                                     :#:
     |                                                                     :#:
     |                                                                     :#:
     |                                             :::@                : :::#:
     |                                             :  @                :@: :#:
     |                                        ::@@@:  @               ::@: :#:
     |                                        : @  :  @               ::@: :#:
     |                                 @@   ::: @  :  @         :@   :::@: :#:
     |              : :::@     :     @:@ : @::: @  :  @       :::@::::::@: :#:
   0 +----------------------------------------------------------------------->Gi
     0                                                                   75.09

After

--------------------------------------------------------------------------------
Command:            head/target/release/tree-sitter-stack-graphs-typescript index -D head.sqlite --max-file-time=30 --hide-error-details -- head/data/typescript_benchmark
Massif arguments:   --massif-out-file=perf.out
ms_print arguments: --x=72 --y=12 head-perf-results/perf.out
--------------------------------------------------------------------------------


    GB
3.276^                                                                     :# 
     |                                                                     :# 
     |                                                                     :# 
     |                                                                     :#:
     |                                                                     :#:
     |                                                                     :#:
     |                                                                 @@:::#:
     |                                               @                 @@: :#:
     |                                         :::@@@@                :@@: :#:
     |                                         :  @  @                :@@: :#:
     |                                   @   :::  @  @         :@    ::@@: :#:
     |              @  :@:      :     :::@:::: :  @  @       :::@::::::@@: :#:
   0 +----------------------------------------------------------------------->Gi
     0                                                                   74.28

github-actions · 2023-11-20T19:56:40Z

Performance Summary

Comparing base 5a6744b with head c57360d on typescript_benchmark benchmark. For details see workflow artifacts. Note that performance is tested on the last commits with changes in stack-graphs, not on every commit.

Before

--------------------------------------------------------------------------------
Command:            base/target/release/tree-sitter-stack-graphs-typescript index -D base.sqlite --max-file-time=30 --hide-error-details -- base/data/typescript_benchmark
Massif arguments:   --massif-out-file=perf.out
ms_print arguments: --x=72 --y=12 base-perf-results/perf.out
--------------------------------------------------------------------------------


    GB
3.276^                                                                     :# 
     |                                                                     :# 
     |                                                                     :# 
     |                                                                     :#:
     |                                                                     :#:
     |                                                                     :#:
     |                                             :::@                 ::::#:
     |                                             :  @                 :: :#:
     |                                        ::@@@:  @                 :: :#:
     |                                        : @  :  @               @@:: :#:
     |                                @:@   ::: @  :  @         :@:  :@ :: :#:
     |             :: :::@     :    ::@:@:::::: @  :  @       :::@::::@ :: :#:
   0 +----------------------------------------------------------------------->Gi
     0                                                                   75.00

After

--------------------------------------------------------------------------------
Command:            head/target/release/tree-sitter-stack-graphs-typescript index -D head.sqlite --max-file-time=30 --hide-error-details -- head/data/typescript_benchmark
Massif arguments:   --massif-out-file=perf.out
ms_print arguments: --x=72 --y=12 head-perf-results/perf.out
--------------------------------------------------------------------------------


    GB
3.276^                                                                     :# 
     |                                                                     :# 
     |                                                                     :# 
     |                                                                     :#:
     |                                                                     :#:
     |                                                                     :#:
     |                                             :::@                 @:::#:
     |                                             :  @                 @: :#:
     |                                        ::@@@:  @                 @: :#:
     |                                        : @  :  @               ::@: :#:
     |                                 @@   ::: @  :  @         :@   :: @: :#:
     |              :@::::     :    ::@@ :::::: @  :  @       :::@::::: @: :#:
   0 +----------------------------------------------------------------------->Gi
     0                                                                   75.01

We're using the `mlua` crate, since it seems most actively maintained and performant.

github-actions · 2023-11-20T21:16:25Z

Performance Summary

Comparing base 5a6744b with head 82d0a0a on typescript_benchmark benchmark. For details see workflow artifacts. Note that performance is tested on the last commits with changes in stack-graphs, not on every commit.

Before

--------------------------------------------------------------------------------
Command:            base/target/release/tree-sitter-stack-graphs-typescript index -D base.sqlite --max-file-time=30 --hide-error-details -- base/data/typescript_benchmark
Massif arguments:   --massif-out-file=perf.out
ms_print arguments: --x=72 --y=12 base-perf-results/perf.out
--------------------------------------------------------------------------------


    GB
3.276^                                                                     :# 
     |                                                                     :# 
     |                                                                     :# 
     |                                                                     :#:
     |                                                                     :#:
     |                                                                     :#:
     |                                             :::@                 ::::#:
     |                                             :  @                 :: :#:
     |                                        ::@@@:  @                 :: :#:
     |                                        : @  :  @               @@:: :#:
     |                                @:@   ::: @  :  @         :@:  :@ :: :#:
     |             :: :::@     :    ::@:@:::::: @  :  @       :::@::::@ :: :#:
   0 +----------------------------------------------------------------------->Gi
     0                                                                   75.00

After

--------------------------------------------------------------------------------
Command:            head/target/release/tree-sitter-stack-graphs-typescript index -D head.sqlite --max-file-time=30 --hide-error-details -- head/data/typescript_benchmark
Massif arguments:   --massif-out-file=perf.out
ms_print arguments: --x=72 --y=12 head-perf-results/perf.out
--------------------------------------------------------------------------------


    GB
3.276^                                                                  :##   
     |                                                                  :#    
     |                                                                  :#  : 
     |                                                                  :# :: 
     |                                                                  :# :: 
     |                                                                  :# :: 
     |                                           :::@                @:::# :: 
     |                                           :  @                @: :# :: 
     |                                      :::@@:  @                @: :# :: 
     |                                      : :@ :  @              ::@: :# :: 
     |                               @:   ::: :@ :  @        :@:   : @: :# :: 
     |             @  ::@     :     :@::::::: :@ :  @       ::@::::: @: :# :: 
   0 +----------------------------------------------------------------------->Gi
     0                                                                   78.03

github-actions · 2023-11-20T21:51:13Z

Performance Summary

Comparing base 5a6744b with head 6e3285a on typescript_benchmark benchmark. For details see workflow artifacts. Note that performance is tested on the last commits with changes in stack-graphs, not on every commit.

Before

--------------------------------------------------------------------------------
Command:            base/target/release/tree-sitter-stack-graphs-typescript index -D base.sqlite --max-file-time=30 --hide-error-details -- base/data/typescript_benchmark
Massif arguments:   --massif-out-file=perf.out
ms_print arguments: --x=72 --y=12 base-perf-results/perf.out
--------------------------------------------------------------------------------


    GB
3.276^                                                                     :# 
     |                                                                     :# 
     |                                                                     :# 
     |                                                                     :#:
     |                                                                     :#:
     |                                                                     :#:
     |                                             :::@                 ::::#:
     |                                             :  @                 :: :#:
     |                                        ::@@@:  @                 :: :#:
     |                                        : @  :  @               @@:: :#:
     |                                @:@   ::: @  :  @         :@:  :@ :: :#:
     |             :: :::@     :    ::@:@:::::: @  :  @       :::@::::@ :: :#:
   0 +----------------------------------------------------------------------->Gi
     0                                                                   75.00

After

--------------------------------------------------------------------------------
Command:            head/target/release/tree-sitter-stack-graphs-typescript index -D head.sqlite --max-file-time=30 --hide-error-details -- head/data/typescript_benchmark
Massif arguments:   --massif-out-file=perf.out
ms_print arguments: --x=72 --y=12 head-perf-results/perf.out
--------------------------------------------------------------------------------


    GB
3.276^                                                                  :##   
     |                                                                  :#    
     |                                                                  :#  : 
     |                                                                  :# :: 
     |                                                                  :# :: 
     |                                                                  :# :: 
     |                                           :::@                @:::# :: 
     |                                           :  @                @: :# :: 
     |                                      ::@@@:  @              : @: :# :: 
     |                                      : @  :  @              ::@: :# :: 
     |                                :   ::: @  :  @        ::@  :::@: :# :: 
     |             @::::@     @    :@@::::::: @  :  @       :::@:::::@: :# :: 
   0 +----------------------------------------------------------------------->Gi
     0                                                                   78.07

You can't do anything with them other than call `tostring` on them, but even that is helpful in test cases!

This is the spackle that parses a source file using tree-sitter, and calls a Lua function with it and an empty stack graph. The Lua function can do whatever it wants to walk the parse tree and add nodes and edges to the graph.

This one is fun, because `SpanCalculator` holds a reference to the file's source code, while the `mlua::UserData` works best for Rust types that are 'static. To get around this, we make sure to only ever create `SpanCalculator` wrappers for source data that is owned by the Lua interpreter, and add that source data as a user value of the Lua wrapper that we create. That should cause Lua's garbage collector to ensure that the source code outlives the `SpanCalculator`, making it safe for us to transmute the source reference to a 'static lifetime.

The stack graph builder now imports the `lsp-position` module before handing control to your Lua script. That lets you create a span calculator, and use that to fill in spans and definiens for the stack graph nodes that you create.

github-actions · 2023-11-21T16:50:45Z

Performance Summary

Comparing base 1b01a8c with head ff1e99d on typescript_benchmark benchmark. For details see workflow artifacts. Note that performance is tested on the last commits with changes in stack-graphs, not on every commit.

Before

--------------------------------------------------------------------------------
Command:            base/target/release/tree-sitter-stack-graphs-typescript index -D base.sqlite --max-file-time=30 --hide-error-details -- base/data/typescript_benchmark
Massif arguments:   --massif-out-file=perf.out
ms_print arguments: --x=72 --y=12 base-perf-results/perf.out
--------------------------------------------------------------------------------


    GB
1.406^                                                #                       
     |                                                #                      :
     |                                             :::#                      :
     |                                          ::@:  #                     ::
     |                                          : @:  #                     ::
     |                                          : @:  #                 ::::::
     |                                         :: @:  #                 :   ::
     |                                  @::   ::: @:  #          :      :   ::
     |                    @             @::   ::: @:  #         ::    :::   ::
     |               :   :@            :@:: ::::: @:  #       @::: :::: :   ::
     |             @@:::::@     :     ::@:::::::: @:  #       @:::::::: :   ::
     |             @ :::::@ ::::: @ : ::@:::::::: @:  #   :  :@:::::@:: :   ::
   0 +----------------------------------------------------------------------->Gi
     0                                                                   70.80

After

--------------------------------------------------------------------------------
Command:            head/target/release/tree-sitter-stack-graphs-typescript index -D head.sqlite --max-file-time=30 --hide-error-details -- head/data/typescript_benchmark
Massif arguments:   --massif-out-file=perf.out
ms_print arguments: --x=72 --y=12 head-perf-results/perf.out
--------------------------------------------------------------------------------


    GB
3.276^                                                                  :##   
     |                                                                  :#    
     |                                                                  :#  : 
     |                                                                  :# :: 
     |                                                                  :# :: 
     |                                                                  :# :: 
     |                                           :::@                @:::# :: 
     |                                           :  @                @: :# :: 
     |                                      :::@@:  @                @: :# :: 
     |                                      : :@ :  @              ::@: :# :: 
     |                                :   ::: :@ :  @        :@:  :: @: :# :: 
     |             @ :::@     @    :@@::::::: :@ :  @       ::@::::: @: :# :: 
   0 +----------------------------------------------------------------------->Gi
     0                                                                   78.05

dcreager force-pushed the lua-bindings branch from bc424cd to 9a18451 Compare November 18, 2023 23:58

dcreager force-pushed the lua-bindings branch from 9a18451 to 5a22bf7 Compare November 19, 2023 00:13

dcreager force-pushed the lua-bindings branch from 58a0ca0 to 5fcd36b Compare November 19, 2023 02:06

dcreager force-pushed the lua-bindings branch from 5fcd36b to 470f7ab Compare November 20, 2023 19:51

dcreager added 3 commits November 20, 2023 15:29

Add Lua bindings for building stack graphs

f96b8d6

We're using the `mlua` crate, since it seems most actively maintained and performant.

Document Lua API

91e44c6

Access root and jump-to nodes from file's in Lua

b450a87

dcreager force-pushed the lua-bindings branch from 470f7ab to f9d69c3 Compare November 20, 2023 21:10

Iterate nodes belonging to file

9ae6a87

dcreager force-pushed the lua-bindings branch from f9d69c3 to 06e8340 Compare November 20, 2023 21:45

dcreager added 3 commits November 21, 2023 11:39

Add Lua wrappers for edges

9a30497

You can't do anything with them other than call `tostring` on them, but even that is helpful in test cases!

Move Lua test helpers out into separate (unpublished) package

ea2e203

Bring in (not-yet-released) tree-sitter updates

b82ec8b

dcreager force-pushed the lua-bindings branch from 06e8340 to f11afc7 Compare November 21, 2023 16:43

dcreager added 3 commits November 21, 2023 11:44

Add Lua+tree-sitter stack graph builder

ff1e99d

This is the spackle that parses a source file using tree-sitter, and calls a Lua function with it and an empty stack graph. The Lua function can do whatever it wants to walk the parse tree and add nodes and edges to the graph.

Calculate spans in Lua stack graph scripts

39ee686

The stack graph builder now imports the `lsp-position` module before handing control to your Lua script. That lets you create a span calculator, and use that to fill in spans and definiens for the stack graph nodes that you create.

dcreager force-pushed the lua-bindings branch from f11afc7 to 39ee686 Compare November 21, 2023 16:44

github deleted a comment from Behiiy Nov 29, 2023

Mrmahmoodi1376 mentioned this pull request Sep 27, 2024

## Performance Summary #463

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Lua bindings for building stack graphs #351

Add Lua bindings for building stack graphs #351

dcreager commented Nov 18, 2023

github-actions bot commented Nov 18, 2023

github-actions bot commented Nov 19, 2023

github-actions bot commented Nov 19, 2023

github-actions bot commented Nov 19, 2023

github-actions bot commented Nov 19, 2023

github-actions bot commented Nov 20, 2023

github-actions bot commented Nov 20, 2023

github-actions bot commented Nov 20, 2023

github-actions bot commented Nov 21, 2023

Add Lua bindings for building stack graphs #351

Are you sure you want to change the base?

Add Lua bindings for building stack graphs #351

Conversation

dcreager commented Nov 18, 2023

github-actions bot commented Nov 18, 2023

Performance Summary

github-actions bot commented Nov 19, 2023

Performance Summary

github-actions bot commented Nov 19, 2023

Performance Summary

github-actions bot commented Nov 19, 2023

Performance Summary

github-actions bot commented Nov 19, 2023

Performance Summary

github-actions bot commented Nov 20, 2023

Performance Summary

github-actions bot commented Nov 20, 2023

Performance Summary

github-actions bot commented Nov 20, 2023

Performance Summary

github-actions bot commented Nov 21, 2023

Performance Summary