Automapping: Ignore empty outputs per-rule (#3896)

This change introduces a "compileOutputSet" step which is executed as
part of collecting the rules. In this step, each rule only receives as
possible outputs the output indexes that are non-empty for the region of
that rule.

This makes it much easier to add output variation to only some of the
rules. Previously, due to the possibility of empty outputs being chosen,
having rules with various amounts of variation required setting up
separate rule maps.

This new behavior not kick in for legacy rule maps that still define the
regions manually.

Also fixed a failing Automapping test (unrelated to this change).

Closes #3523

NEWS.md

@@ -12,6 +12,7 @@
 * tmxrasterizer: Added --hide-object and --show-object arguments (by Lars Luz, #3819)
 * tmxrasterizer: Added --frames and --frame-duration arguments to export animated maps as multiple images (#3868)
 * tmxviewer: Added support for viewing JSON maps (#3866)
+* AutoMapping: Ignore empty outputs per-rule (#3523)
 * Windows: Fixed the support for WebP images (updated to Qt 6.6.1, #3661)
 * Fixed mouse handling issue when zooming while painting (#3863)
 * Fixed possible crash after a scripted tool disappears while active

docs/manual/automapping.md

@@ -109,7 +109,7 @@ output[index]_name
 
 Everything after the first underscore is the **name**, which determines which layer in the working map the tiles or objects will be placed on. If the working map includes multiple layers by this name, the bottom-most one will be used. If the rule matches and the working map does not already contain the named output layer, Automapping will create the layer.
 
-The **index** is optional, and is not related to the input indices. Instead, output indices are used to randomize the output: every time the rule finds a match, a random output index is chosen and only the output layers with that index will have their contents placed into the working map.
+The **index** is optional, and is not related to the input indices. Instead, output indices are used to randomize the output: every time the rule finds a match, a random output index is chosen and only the output layers with that index will have their contents placed into the working map. If an output index is completely empty for a given rule, it will never be chosen for that rule (since Tiled 1.10.3).
 
 #### Random Output Example
 
@@ -218,6 +218,7 @@ Probability {bdg-primary}`New in Tiled 1.10`
 
 {bdg-secondary-line}`Since Tiled 1.9`
 
+(object-properties)=
 ### Object Properties
 
 A number of options can be set on individual rules, even within the same rule map. To do this, add an Object Layer to your rule map called `rule_options`. On this layer, you can create plain rectangle objects and any options you set on these objects will apply to all rules they contain.
@@ -384,7 +385,7 @@ A result from the two rules above.
 (updating-rules)=
 ## Updating Legacy Rules
 
-If you have some Automapping rules from before Tiled 1.9, they should still work much as they always did in most cases. When Tiled sees that a rule map contains `regions` layers, it will automatically bring back the old behavior - rules will be matched in order by default, and cells within input regions that are empty in all the input layers for a given layer and index will be treated as "Other".
+If you have some Automapping rules from before Tiled 1.9, they should still work much as they always did in most cases. When Tiled sees that a rule map contains `regions` layers, it will automatically bring back the old behavior - rules will be matched in order by default, cells within input regions that are empty in all the input layers for a given layer and index will be treated as "Other", and completely empty output indices will still be selected as valid outputs.
 
 :::{warning}
 In Tiled 1.9.x, the presence of `regions` layers did not imply **MatchInOrder**. If you're using 1.9.x rather than 1.10+ and want to use legacy rules, you'll need to set the **MatchInOrder** map property to `true`.
@@ -394,13 +395,14 @@ If you'd like to instead update your rules to not rely on any legacy behavior, t
 
 * If your rules rely on being applied in a set order, set the [**MatchInOrder**](#MatchInOrder) map property to `true`.
 * When deleting your `regions` layers, make sure you weren't relying on them to connect otherwise disconnected areas of tiles. If you were, use the [Ignore]{.tile .ignore} [special tile](#specialtiles) to connect them on one of the `input` layers, so that Tiled knows they're part of the same rule. To make sure the rules behave exactly the same, fill in any part that was previously part of the input region.
-    
+
 * If were using the [**DeleteTiles**](#DeleteTiles) map property to erase tiles from the output layer, you can keep using this property. If you want to make your rule more visually clear, however, you should unset the **DeleteTiles** property, and instead use the [Empty]{.tile .empty} [special tile](#specialtiles) in all the output cells you want to delete from.
-    
+
 * If were using the [**StrictEmpty**](#AutoEmpty) map property to look for empty input tiles, you should now use the Empty special tile instead in the cells you want to check for being empty. You can also continue use the **StrictEmpty** property (or its newer alias, **AutoEmpty**), as long as at least one other input layer is not empty at those locations.
-    
+
 * If were relying on the behavior that any tile which is left empty on all of the input layers for a given index is treated as “any tile not in this rule”, you should instead use the [Other]{.tile .other} [special tile](#specialtiles) at those locations, and also the [Empty]{.tile .empty} [special tile](#specialtiles) on an inputnot layer at those same locations. The Empty tile is needed because old-style Other never matched Empty, but the MatchType Other tile does match Empty.
-
+
+* If you have rules that rely on some output indices being empty to randomly not make any changes, you will need to place [**Ignore** special tiles](#specialtiles) in at least one layer of each empty output index so that those indices aren't ignored. Alternatively, you can use [`rule_options`](#object-properties) to give those rules a chance to not run at all.
 
 ## Credits
 

src/libtiled/tilelayer.h

@@ -251,16 +251,16 @@ class TILEDSHARED_EXPORT TileLayer : public Layer
         {
             if (lhs.mChunkPointer == lhs.mChunkEndPointer || rhs.mChunkPointer == rhs.mChunkEndPointer)
                 return lhs.mChunkPointer == rhs.mChunkPointer;
-            else
-                return lhs.mCellPointer == rhs.mCellPointer;
+
+            return lhs.mCellPointer == rhs.mCellPointer;
         }
 
         friend bool operator!=(const iterator& lhs, const iterator& rhs)
         {
             if (lhs.mChunkPointer == lhs.mChunkEndPointer || rhs.mChunkPointer == rhs.mChunkEndPointer)
                 return lhs.mChunkPointer != rhs.mChunkPointer;
-            else
-                return lhs.mCellPointer != rhs.mCellPointer;
+
+            return lhs.mCellPointer != rhs.mCellPointer;
         }
 
         Cell &value() const { return *mCellPointer; }
@@ -307,16 +307,16 @@ class TILEDSHARED_EXPORT TileLayer : public Layer
         {
             if (lhs.mChunkPointer == lhs.mChunkEndPointer || rhs.mChunkPointer == rhs.mChunkEndPointer)
                 return lhs.mChunkPointer == rhs.mChunkPointer;
-            else
-                return lhs.mCellPointer == rhs.mCellPointer;
+
+            return lhs.mCellPointer == rhs.mCellPointer;
         }
 
         friend bool operator!=(const const_iterator& lhs, const const_iterator& rhs)
         {
             if (lhs.mChunkPointer == lhs.mChunkEndPointer || rhs.mChunkPointer == rhs.mChunkEndPointer)
                 return lhs.mChunkPointer != rhs.mChunkPointer;
-            else
-                return lhs.mCellPointer != rhs.mCellPointer;
+
+            return lhs.mCellPointer != rhs.mCellPointer;
         }
 
         const Cell &value() const { return *mCellPointer; }
@@ -646,8 +646,8 @@ inline const Cell &TileLayer::cellAt(int x, int y) const
 {
     if (const Chunk *chunk = findChunk(x, y))
         return chunk->cellAt(x & CHUNK_MASK, y & CHUNK_MASK);
-    else
-        return Cell::empty;
+
+    return Cell::empty;
 }
 
 inline const Cell &TileLayer::cellAt(QPoint point) const