Problems using EnumerationFields in queries

[AndreaPiovanelli]

Since the format values are now saved, the Equals clause in queries doesn't work anymore.

This is a side effect of #8789 , which saves the EnumerationField value in a different way then before, as shown by the following screenshot from Orchard_Projections_StringFieldIndexRecord table:

As you can see, the new version of the Value column contains now the separator character ";" at the start and at the end of the actual value, even when only one value is selected.
There are two topics to discuss about:

1. Values inside the table are now ambiguous and should be coherent, perhaps with a migration step replacing old values with new ones. As a side note: saving a content item is enough to replace the "old" value format with the new one and using the "Value" property of the field works as expected.
2. Queries filtering on the Value of an EnumerationField do not work anymore, because the value is now saved with ";" characters even if a single option is selected. For this reason, the Equals clause cannot be used and currently needs to be replaced by the Contains clause. Creating a migration step to change queries isn't a viable solution: any suggestions on how to make EnumerationFields compatible with the old query usage?

I add to the discussion @MatteoPiovanelli-Laser and @BenedekFarkas because they worked on the referenced pull request and @sebastienros who approved it.

[AndreaPiovanelli]

Orchard/src/Orchard.Web/Modules/Orchard.Fields/Fields/EnumerationField.cs

Line 14 in 6e4fefd

: Separator + value.Trim(Separator) + Separator);

If we do not add separators to the Value property of the field, queries seems to work as expected (using both Equals and Contains operators) and I haven't noticed any side effects.

[BenedekFarkas]

It seems the changes weren't completely backwards-compatible. :(

Updating the StringFieldIndexRecord.Value and/or StringFieldIndexRecord.LatestValue entries in a migration is fairly simple, but we can also change the filter to use an additional condition to match the value both with and without the separators at the beginning/end.

[AndreaPiovanelli]

I used Upgrade module to create a scheduled task to update EnumerationField values, because content items needed to be re-published to ensure infosets to be updated too. This may be moved to a migration to ensure the automatic upgrade of values and avoid the manual operation of a back office user, but this may be a little tricky because of circular dependencies between Orchard.Projections and Orchard.Fields. This is also a very long task, potentially.

I am still working on a viable solution to ensure query compatibility with the newer value format, because creating a new condition isn't really an acceptable solution, since most back office users usually lack the access to the query configuration options and, ideally, the "Equals" condition should still be the right one for single selection EnumerationFields.

If interested, you can find current work in progress on the branch https://github.com/LaserSrl/Orchard/tree/fix/enumerationfieldvaluesmigration, which is based on Laser's fork of Orchard but should give you an idea of what I am working on.

[AndreaPiovanelli]

@BenedekFarkas in the same branch, I found a working solution to the "query" problem. I added a specific IFilterProvider that is almost identical to current ContentFieldsFilter, filtering the fields inside the "Describe" method to only work on EnumerationFields. Consequentially, I modified the original ContentFieldsFilter to exclude from its Describe method the EnumerationFields (https://github.com/LaserSrl/Orchard/blob/acda3e32b37370423cb8cac17c5f29adfe46cc7d/src/Orchard.Web/Modules/Orchard.Projections/Providers/Filters/ContentFieldsFilter.cs#L39).

The ApplyFilter function for the EnumerationFieldsFilter modifies the Value parameter of the context object to properly add ';' separator characters when needed (see https://github.com/LaserSrl/Orchard/blob/acda3e32b37370423cb8cac17c5f29adfe46cc7d/src/Orchard.Web/Modules/Orchard.Projections/Providers/Filters/EnumerationFieldsFilter.cs#L77).
Since Category and Name parameters for the filter are the same as the original content fields filter, this solution works without a migration (old filters for EnumerationFields are now applied using the new provider).

The Upgrade procedure for the fields is needed to make everything work properly and has to be executed separately for each tenant. I excluded the upgrade of the values through a migration because the operation potentially involves thousands of content items for each tenant and may lock all tenants for a long time.

I released the branch on a test environment and will post a pr for the community as soon as possible.

[BenedekFarkas]

Thanks for the update, I'll check get back to this towards the end of September (after Harvest).

[BenedekFarkas]

Let's take a step back to have a clearer picture:

1. The original issue (Inconsistent use of EnumerationField #8460) was raised, because changing the ListMode of an EnumerationField can cause the editor not to pick up the existing value (and lose it upon saving the editor) when ListMode changes from one of the multi-select flavors (ListBox, Checkbox) to a single-select one (Dropdown, RadioButton). That is because a multi-select flavored editor wrote to SelectedValues, which surrounds the value with the separator (;) even if a single value is selected, while Value remains empty. On the other hand, the single-select flavored editors operate with Value directly, so the value won't be surrounded with separator characters.
2. The applied fix (#8640: Fixing consistency between different Enumeration Field flavors' data storage #8789) updated EnumerationField property setters so that when a single value is saved, it's surrounded by separator characters.
3. This issue is raised, because the changes in #8640: Fixing consistency between different Enumeration Field flavors' data storage #8789 caused projection filters not to match the exact value with the Equals operator for a single value, because it's saved with separator characters around it.
4. Upgrades to enumeration fields and queries related to them #8801 is opened to fix this by adding a new filter specific to EnumerationField and some other changes, but it feels like we're getting into a complexity spiral.

What are our options?

1. We can revert the first PR and change EnumerationFieldDriver or the associated template to make sure that the editor is correctly populated with a value after the editor flavor is changed from a multi-select to a single-select one (e.g., use the first entry from SelectedValue). This is a super simple change that was in the original PR already (just initialize the editor fields with Model.SelectedValues.FirstOrDefault() instead of Model.Value), I just tested it again, works fine and it addresses the original issue without changing anything about how data is stored. See #8800: Reverting breaking changes to Enumeration Fields caused by #8789 #8824.
   However, regarding consistency, I'd consider changing the flavor of the editor from a single-select to multi-select one (or vice versa) a rare/edge case, so you also need to care of your data consistency too anyway. IMO that should be up to custom code, not Orchard.
2. Can you please confirm that the current code works if you use the Contains operator (instead of Equals) for every EnumerationField? Looking at the code, I think it does, so that would be an easy win.
3. Depending on your business logic, you could surround your single-select values with ; before they are passed to the projection filter (the filter editor field is tokenized), so that the Equals operator continues to work. However, that would require updating the all the content items that use an EnumerationField.

What do you think?

BTW I'd encourage you all from Laser to join the Orchard Discord channel to discuss things not specific to issues/PRs and we can also get on a call to discuss your PRs.

[BenedekFarkas]

@AndreaPiovanelli can you please take a look at #8824?