handling memory allocation limit for query eval

[r3v4s]

Description

In pr #267, max memory cap has been set to 1.5g for qeval with comment // higher limit for queries

I'm aware of increasing this limit can break rpc service (similar research against ethereum => paper & video

One of suggested countermeasure by author is, Performance anomaly detection plus security deposit which is for client(or d-app) to use rpc, it has to deposit certain amount money to rpc provider and if abnormal behavior has been detected deposit going to be confiscated.

However current limit may not be enough for certain use case.

For example, to get best result in dex(defi) it needs to search all over existing position. And to give user a estimated result, interface calls DrySwap over RPC using qeval to get it. This is where current limit can be insufficient.
( if positions are spread sparsely in all over range, bunch of iteration can be happen in qeval requests which can result panic, allocation limit reached )

Question

1. Is current limit(1.5g) is well-known number(or calculated from certain formula)
2. Does it have to be static value? Can't it be dynamic?

cc @mconcat @notJoon @dongwon8247 @zivkovicmilos

[zivkovicmilos]

Based on discussions from our call today, the suggestions were:

• optimize the SC itself so it minimizes gas usage
• run a separate node that you can query, where the limit on this is much higher (since qevals are temporary, not committed to a transaction, and local to the node executing them)

I'm curious if you can provide us some more information:

• The gas usage limits you're hitting, even when increasing this
• The types of operations
• The contracts (if this is public), and what calls you're doing -- this one is primarily for tracing and seeing what operations are eating gas usage the most for your use-case

[thehowl]

Aside from what Milos said, I should point out that the real solution is necessarily not increasing the allocation limit, but rather addressing synchronous GC: #266.

This way, the "allocation" of a contract reflects how much memory it has actually stored, rather than a sum of all of the times it has attempted to store memory.

[notJoon]

I should point out that the real solution is necessarily not increasing the allocation limit, but rather addressing synchronous GC

I also believe it's more generalizable in the long term to address this problem directly at the language level.

While 'external solutions' (I don't know if this is the right expression) such as optimization in SC, are still practical and valuable and should have considered in development process. But some approaches may to be cumbersome or necessitate distinct strategies for each project, which can be restrictive.

BTW has there any further discussion about GC related things?

[zivkovicmilos]

BTW has there any further discussion about GC related things?

@petar-dambovaliev
Do you know what the latest status on the GC efforts is? I vaguely remember us discussing it, that it was temporarily tabled

[moul]

This should be addressed by sync GC, which should have a reasonable default. We can also expect some users to run special RPC nodes with higher values. In other words, it makes sense to make it configurable while providing a reasonable default.