Support predicates

[kaby76]

As noted here, the javadoc/ grammar contains actions. lab.antlr.org should output a warning message if the grammar contains action code since action code will not be executed. Even long-time users of Antlr will make an assumption that is wrong.

[kaby76]

There were a few comments about supporting other targets and even action code in the long discussion last year, e.g. interp simulator for Javascript and even antlr4ts. For now, it would be best for the interp itself to output a warming if an action is encountered.

[ericvergnaud]

The lab runs on server side in Java, running parsers in a lambda could be a solution to prevent damage

[kaby76]

Also to consider, there's only a syntaxError() as output from the parser. No such thing as a warning. Seems rather limited to what most compilers do. I suppose the interpreter not being able to execute an action is pretty much a failure though. What should happen if the parse doesn't actually need to execute the action from a particular start rule? No need to produce a warning even if the grammar does have actions.

[ericvergnaud]

I'd be comfortable not executing actions because the lab is about parsing, not about further steps. But predicates participate in parsing so it's certainly worth a discussion.

[KvanTTT]

There is no cruical difference between predicates and actions except the first ones may return boolean result. If predicates are supported, actions can be supported as well. Also, both predicates and actions are requred for parsing. For instance, take a look at PHP Lexer.

[kaby76]

Sorry, I use "actions" to be either { ... } or { ... }? target-specific code because the parse tree node is actionBlock for either when parsing .g4's. We may need to discuss both. I thought there were grammars that had both, where { ... } blocks were used to initialize some state variables for the { ... }? predicates. Checking now

for i in `find . -name desc.xml`; do echo $i; d="$(dirname $i)"; echo $d; trparse $d/*.g4 | trxgrep ' //element/actionBlock' | trtext; done

[ericvergnaud]

A solution worth exploring would be to generate the lexer and parser in JavaScript on the server, and run them in the browser. This would require devs to code predicates in JS (in the lab) but that sounds like a reasonable requirement

[kaby76]

Most if not all grammars in grammars-v4 that have a JS port are in "target agnostic format", which means there are addition source files in JS. It would be nice to have the tool have an extra tab for these files. Otherwise, we'd need to refactor the grammars on the fly to use @members and remove the superClass option.

[ericvergnaud]

Maybe we should convert this into a discussion?

[kaby76]

Maybe we should convert this into a discussion?

Yes, please do.

Also of note. There is no consistent "recognizer base class method reference" syntax across targets. So we have action code with:

• this.FooBar() in CSharp, Java
• l.FooBar() and p.FooBar()` in Go
• this.FooBar() in JavaScript, TypeScript
• this->FooBar() in Cpp
• $this->FooBar() in PHP
• self.FooBar() in Python3

(I haven't set up testing for Swift because of the lack of a Mac machine--currently a brick, can't install a new OS on a blank HDD.)

The grammars contain either self.FooBar() or this.FooBar(), and that is modified to the target-specific format.

[ericvergnaud]

I think this multi-target topic deserves a separate discussion in antlr4 repo. When I started using antrl3, I implemented a C-like macro system to overcome this limitation. With antlr4 I inherit the abstract parser and use $parser.someMethod() . It works for all targets I need, which fortunately don't include Cpp and PHP (which require ->).

Would it make sense to provide macros as part of the antlr 'language' ?
What problems could arise with the following construct ?

#define MY_PREDICATE
#if TARGET==Java
_input.LT(1) == '/'
#elif TARGET==JavaScript
this.input.LT(1)=="/".charCodeAt(0)
#elif
.../...
#endif
#enddef

.../...

my_rule:
{ MY_PREDICATE }? some_other_rule
;

Alternately we could provide a preprocessor that does the job and calls the tool...

[KvanTTT]

I think this multi-target topic deserves a separate discussion in antlr4 repo.

Such discussion already exists: antlr/antlr4#4067

Would it make sense to provide macros as part of the antlr 'language' ?
What problems could arise with the following construct ?

The problem that real code after preprocessing differs from the original one. It compilcates error reporting a lot. Also, such syntax looks confusing.

Moreover, modern programming languages don't have or have very limited directives support. They cause a lot of pain in C/C++ languages.

That's why I vote against such directives-like syntax.

[ericvergnaud]

So what would you propose ?

[kaby76]

BTW, it's not necessary to have a Mac machine for Swift testing. You can use Linux or even Windows. At least our test infrastructure works on these platforms.

Thanks, that helped. I installed it on Windows. I needed to add export a few environmental variables and not use "swift" but "swiftc" instead. I can compile and run a Swift "Hello World" program. Working on Swift testing for grammars-v4.

[KvanTTT]

I suggest using universal function calls (use sites) and target-specific function declarations. Something like that:

stat
    : { java5() }? 'goto' ID ';'

@parser::members::java {
  boolean java5() {
    ...
  }
}
@parser::members::cpp {
  bool java5() {
    ...
  }
}

The idea was originally suggested by @udif.

[parrt]

I think the server runs the grammars via the Java-based interpreter so it's not compiling/executing random code from the internet. One infinite loop and server is dead ;)

[KvanTTT]

It can be run within isolated process that is being terminated by timeout.

[parrt]

something could be done but they could always fill up my google storage and cost me $100k overnight haha

[KvanTTT]

I suspect infinity loop might be implemented via grammar itself without any line of Java code. We've encountered StackOverflow some time ago: antlr/antlr4#2860 (that also quite dangerous vulnerability) but infinity loop is also possible. I haven't try to write a 'grammar-bomb' but almost sure it's possible. I think timeout and memory protection should be implemented anyway.

[KvanTTT]

Also, there are a lot of online compilers in the Internet. They solve the problem of time/memory somehow.

[ericvergnaud]

@parrt running it in a lambda would eliminate that risk entirely, since a lambda cannot run for more than 15 mns (AWS, not sure about GCP).

[parrt]

I have a killer thread that won't let the requests run very long already :)

[ericvergnaud]

Yup I saw that

[ericvergnaud]

But it could still trigger an OOM 😉

[parrt]

indeed!

[kaby76]

Macros would be confusing as C++ already has a preprocessor in the language, and presumably one can embed macros in @member code in a grammar.