forth: evaluateBoth property

[NobbZ]

In the forth specification there is a single test for the evaluateBoth property.

The remaining tests are for the evaluate property.

https://github.com/exercism/problem-specifications/blob/main/exercises/forth/canonical-data.json#L510

This confused the erlang test generator recently, as it assumes the property key points to the function under test.

Is this a missunderstanding on my side or do we need to find another way to specify a sequence of operations/function calls in the spec?

[glennj]

Nobody's commented on this so far: my take:

The README describes it like

   " The 'property' is a string in lowerCamelCase identifying ",
   " the type of test, but most of the times it is just the   ",
   " name of a function being tested.                         ",

I don't see it documented anywhere else as meaning anything more specific.

Clearly, it can't be absolutely relied upon to be anything more than a label describing some action to impose upon the system under test.

For forth specifically, the point is to instantiate two forth evaluators and ensure that inputs to one do not bleed into the other -- that can't be a single function on a single instance. (The bash track excludes this test: clearly two separate scripts running in different processes cannot accidentally interact).

Sadly, the generator will need some specific instructions for some exercises.

[ErikSchierboom]

I don't see it documented anywhere else as meaning anything more specific.

Clearly, it can't be absolutely relied upon to be anything more than a label describing some action to impose upon the system under test.

This is true. In 99% of the test cases, the property can be used as the function name, but there are test cases where this can't work, either because the thing that is being tested is impossible to map to a JSON object with inputs/outputs (e.g. something being random, see https://github.com/exercism/problem-specifications/blob/main/exercises/diffie-hellman/canonical-data.json#L33), or because it is describing multiple actions (like the forth test case you mentioned).

Sadly, the generator will need some specific instructions for some exercises.

It should indeed. I wonder if it would be a good idea to add a scenario for these exceptional test cases, which could then be detected by a generator.

[kytrinyx]

I think a scenario is a very good suggestion.
How about local-scope? We already have that scenario, and it seems to fit the bill.

[ErikSchierboom]

How about local-scope? We already have that scenario, and it seems to fit the bill.

Yes, that does indeed fit the bill well and should indeed be added!

What I was originally suggesting was to add another scenario, namely one that indicates that the property is different from a regular property and requires special treatment by a test generator. But maybe we don't need it as certain scopes will always require this special treatment, like local-scope and immutable. Maybe it is enough to identify all test cases that have special behavior and a add an appropriate scenario?

Here is another weird test case in dnd-character:

{
  "uuid": "2ca77b9b-c099-46c3-a02c-0d0f68ffa0fe",
  "description": "each ability is only calculated once",
  "property": "strength",
  "input": {},
  "expected": "strength == strength"
}

Should this have a memoized or memoization scenario?

[kytrinyx]

Ah, yeah, memoization is probably a good idea there.