[WIP] Add tests and started implementation of an expression parser

[MarcelloDuarte]

TODO

This is the todo of what the expression parser should cover based this grammar available on the PHP repo.

• variable
• T_LNUMBER
• T_DNUMBER
• other_scalar_expr
• T_LIST ( array_pair_list )
• array_pair_list
• variable phpOperator expr
• expr phpOperator expr
• variable = & variable
• T_CLONE expr
• variable instanceof ns()
• expr instanceof ns()
• ( expr )
• new_expr | new anonumous class
• ternary_expr
• expr '?' ':' expr
• expr T_COALESCE expr
• casting expr
• '@' expr
• '' backticks_expr ''
• T_PRINT expr
• T_YIELD | T_YIELD expr | T_YIELD expr T_DOUBLE_ARROW expr | T_YIELD_FROM expr
• anonymous function

[marcioAlmada]

Hi @MarcelloDuarte,

Perhaps if we keep the idea of ·expr() match as "greedy" and avoid internal language knowledge like operator precedence, at least for now, the feature will cover 90% of use cases while maintaining a high KISS factor.

This looks like an awesome start!

[marcioAlmada]

I'm sure you already know it, this is just an obvious reminder: variable has some other branches like https://3v4l.org/10bFt and the not so popular var var $$$x syntax.

[MarcelloDuarte]

I seem to be missing something about rtoken. I would think the following would work:

    return rtoken('/\$[\$]+[a-zA-Z_\x7f-\xff][a-zA-Z0-9_\x7f-\xff]*/');

but it doesn't. $$a is matched as $a.

[marcioAlmada]

A single \$+ works nicely according to https://regex101.com/r/WhuVPY/1

[MarcelloDuarte]

Yeah, it should work, but phpunit disagrees:

There was 1 failure:

1) Yay\SpecsTest::testSpecs with data set "/Users/mduarte/Developer/code/MarcelloDuarte/yay/tests/phpt/expr/variable.phpt" (Yay\Test Object (...))
Failed asserting that format description matches text.
--- Expected
+++ Actual
@@ @@
 <?php

-expression {
-    $$a
+$expression {
+    $a
 }

 ?>

[marcioAlmada]

Just double checking, did you really use /\$+[a-zA-Z_\x7f-\xff][a-zA-Z0-9_\x7f-\xff]*/ as the regexp? This seems to work as specified:

use \Yay\{TokenStream, function rtoken};

$ts = TokenStream::FromSourceWithoutOpenTag('$a $bb $ccc');

token(T_OPEN_TAG))->parse($ts); // jump over <?php

var_dump(rtoken('/\$+[a-zA-Z_\x7f-\xff][a-zA-Z0-9_\x7f-\xff]*/')->parse($ts));
# T_VARIABLE($a)

var_dump(rtoken('/\$+[a-zA-Z_\x7f-\xff][a-zA-Z0-9_\x7f-\xff]*/')->parse($ts));
# T_VARIABLE($bb)

var_dump(rtoken('/\$+[a-zA-Z_\x7f-\xff][a-zA-Z0-9_\x7f-\xff]*/')->parse($ts));
# T_VARIABLE($ccc)

var_dump(rtoken('/\$+[a-zA-Z_\x7f-\xff][a-zA-Z0-9_\x7f-\xff]*/')->parse($ts));
# null

If so, that would be scary, perhaps PCRE engine is behaving differently on our builds 🤔 Are you using Windows?

[MarcelloDuarte]

Yeah, I did use it. It works fine with normal variables, but not with variable variable:

var_dump(\Yay\rtoken('/\$+[a-zA-Z_\x7f-\xff][a-zA-Z0-9_\x7f-\xff]*/')->parse(\Yay\TokenStream::fromSourceWithoutOpenTag('$$a')));
# null

[marcioAlmada]

Ok, so there was one issue fixed with a2329f8.

But the real culprit is that PHP tokenizes variable variables as [token('$'), token('$'), token('$a')] instead of [token('$$a')], hence why rtoken would never work.

The following is guaranteed to work:

use \Yay\{TokenStream};

use function \Yay\{chain, optional, repeat, token};

$variable = chain(optional(repeat(token('$')))->as('varvar'), token(T_VARIABLE)->as('var'));

$ts = TokenStream::FromSourceWithoutOpenTag('$a $$b $$$c');

var_dump($variable->parse($ts)); // Ast for $a
var_dump($variable->parse($ts)); // Ast for $$b
var_dump($variable->parse($ts)); // Ast for $$$c
var_dump($variable->parse($ts)); // null

And sorry, my previous test was wrong.

👍

[marcioAlmada]

It seems that whitespace should be preserved in the Ast every time we get an operator that is also a valid identifier, so expansion won't break like in this case $ainstanceof\Some\Large\FQCN. A self contained example.:

$instanceof = chain(
    chain(variable(), indentation())->as('left'),
    token(T_INSTANCEOF)->as('operator'),
    chain(indentation(), ns())->as('right')
)
->as('expr');

$ts = \Yay\TokenStream::fromSource('<?php $a  instanceof     \Some\Large\FQCN');

\Yay\token(T_OPEN_TAG)->parse($ts); // get rid of <?php

$ast = $instanceof->parse($ts);

$src = '';
$nodes = $ast->array();
array_walk_recursive($nodes, function($n) use(&$src){ $src .= $n; });
var_dump($src); #>>  "$a  instanceof     \Some\Large\FQCN"

[MarcelloDuarte]

fixed in f60fe4f

[marcioAlmada]

I really appreciate the efforts here, but turns out the task was quite complex. First, it was necessary to add indirect and direct left recursion to the parser, then it became necessary to add knowledge about operators and operator precedence and, later, be able to self-optimize the resulting parser stacks - so the resulting grammar was at least efficient.

Refer to ExpressionParser. The resulting grammar was converted from zend_language_parser.y and may certainly contain a few bugs right now.

Contributions are still very welcome 😄