Parse SQL Statements (by ;)

[psteinroe]

Hey,

fairly new to chumsky. I have been trying to get a simple SQL statement parser to work without success. For now, the only thing I want to achieve is to parse

select id from contact where company = 'supabase';

select id from employee where company = 'supabase';

into

["select id from contact where company = 'supabase';", "select id from employee where company = 'supabase';"]

Essentially, just a trimmed str.split(';').

I have been playing with any and the separated_by parser. The closest I could get is

    let statements = any::<_, extra::Err<Simple<char>>>().repeated().collect::<Vec<_>>().separated_by(just(';'));
    let results = statements.parse(src.trim());

    println!("{:?}", src.trim());
    println!("{:?}", results);

but it returns just an empty result.

This is a very beginner question, but I am thankful for a hint to get started.

Thanks
Philipp

[zesterer]

Hello! Interestingly enough, I'd actually suggest not starting with something like this: it's usually better to start with a far more minimal subset of a syntax and build up rather than starting with a whole syntax and building down. If you really feel like like you want to take this approach though:

any().repeated()

Here, any will consume any character (including semicolons) and repeated will result in this happening as many times as possible. In effect, these two parsers together will consume the whole input, regardless of what comes next. If you want to prevent any consuming semicolons, you can special-case semicolons away:

any().and_is(just(';').not()).repeated()

(or, more simply)

none_of(';').repeated()

As I say though, I recommend starting building up instead of down, so the best place to start would probably be parsing the most basic elements of SQL: integers and other bits of expressions, and then building on top of them.

You'll find that the tutorial is a helpful resource when building up a parser (note, however, that it's not been updated to 1.0 yet and so small details will differ).

[psteinroe]

thanks for the swift and elaborate answer! Much appreciated.

I am not trying to build a full-blown sql parser. Instead, I want to parse a (migration) file into separate sql statements and comments. In that sense, statements separated by ; already are the most minimal subset. Later, I want to add comments and create function statements, since those cannot be parsed by a simple string.split().

I tried your sample, but it returns a parsing error.

    let statements = any::<_, extra::Err<Simple<char>>>().and_is(just(';').not()).repeated().collect::<Vec<_>>();
    let results = statements.parse(src.trim());

    println!("{:?}", src.trim());
    // "select id from contact where company = 'supabase'; -- some comment inline\n\n-- some comment elsewhere\n\nselect id from employee where company = 'supabase';"
    println!("{:?}", results);
    // ParseResult { output: None, errs: [found '';'' at 49..50] }

Is chumsky the right tool for what I am trying to achieve? Or is it not well-suited for "higher level" parsing?

Anyways, I will walk through the tutorial first, and not try to jump ahead anymore.

[zesterer]

I tried your sample, but it returns a parsing error.

To be clear, the example I gave was a snippet that represents a single statement. You'll still want to add the .separated_by(just(';')) on the end.

I am not trying to build a full-blown sql parser. Instead, I want to parse a (migration) file into separate sql statements and comments.

Be careful with this approach. Correct parsers, even those that don't care for inner context, often need to understand the full syntax of a language. I don't know SQL that well, but many languages have constructs that allow nesting statements inside expressions, meaning that it's impossible to represent the source code as just a simple 'list of statements': and hence any parser that makes this assumption may just end up mangling the structure of the input in undesirable ways.

Is chumsky the right tool for what I am trying to achieve? Or is it not well-suited for "higher level" parsing?

Chumsky can parse pretty much whatever you want. But, like all parser combinators/generators, you need to be precise and honest about what the source language looks like. This isn't something specific to chumsky: playing fast & loose with syntax rules is generally a recipe for trouble for any parser, hand-written or otherwise. If "SQL is just a list of statements separated by semicolons" is the assumption that you code into the parser, then that's how it'll act: regardless of whether this assumption actually matches the real behaviour you're looking for.