Implement support for subtractive shadowmapping or shadowmasking in DirectionalLight (for use with BakedLightmap)

[Calinou]

See also discussion in godotengine/godot#46332.

Edit: Going for shadowmasking instead of subtractive shadowmapping may be a better idea in the end, as it's easier to use and more flexible. See #2354 (comment).

Describe the project you are working on

The Godot editor 🙂

Describe the problem or limitation you are having in your project

When targeting low-end desktop hardware or mobile hardware, baked lightmaps are often used but combining them with real-time shadows is difficult.

While it's possible to use a DirectionalLight with the bake mode set to Indirect and point lights' bake mode set to All, there are several downsides to this:

• Distant objects won't cast any shadows, which can look bad with large structures. You can increase the DirectionalLight shadow Max Distance property to help with this, but this will make nearby shadows blurrier (including those from dynamic objects).
• Performance will be lower than the proposed subtractive mode, since the whole lightmapped level will cast shadows. This increases the number of vertices and draw calls in the DirectionalLight shadow cascades.

The indirect DirectionalLight approach described above is suited for many types of games, but when you want to display relatively large worlds on mid-range hardware, you may need a more oldschool solution as proposed below.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

Mixing baked and real-time lighting is fundamentally a difficult problem to solve with no perfect solutions. One way to handle this is to use subtractive shadow maps that darken the underlying surfaces to a predefined color.

Subtractive shadowmapping isn't very realistic, but it's cheap enough to be usable on mobile platforms nowadays. On desktop platforms, its low cost will also allow players to enable more effects on low-end/mid-range hardware compared to other lighting solutions. On top of that, since subtractive shadowmapping still uses fully-baked shadows for lightmapped meshes, shadows in the distance will never fade away or exhibit seams between splits. Only dynamic objects will lose their shadow in the distance, but this is far less problematic than having both static and dynamic objects lose their shadows.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

A subtractive shadowmapping option means that the DirectionalLight won't emit any light and will only cast a shadow. As you might expect, this isn't going to work with standard light equations 🙂

The blending formula needs to be made toggleable with a DirectionalLight property, and will require modifying the shadow blending equations in drivers/gles3/shaders/scene.glsl. The shadow color can be defined as a project setting, since we don't want to introduce per-light shadow colors in 4.0 for performance reasons. It should be configured by the user to roughly match the color of baked shadows.

For proper appearance, lightmapped meshes will also have to have their GeometryInstance shadow mode set to Off after baking the lightmap. When using subtractive shadowmapping, only dynamic objects should cast shadows since static shadows are already handled by the lightmap itself. Currently, this can be done after baking lightmaps but you need to revert it to On every time you wish the bake the lightmap, which is inconvenient. We'll have to figure out how to do this automatically.

For reference, Unity provides a subtractive lighting mode. Here's a video comparing all of Unity's baked lighting modes. Unity also provides a shadowmask mode in its HDRP rendering pipeline, but it's more demanding on the hardware and significantly more complex to implement since it requires two sets of lightmaps and light probes. Therefore, I consider implementing shadowmask to be outside the scope of this proposal.
Edit: There's a tutorial on implementing shadow masking in Unity SRP which is a good read: https://catlikecoding.com/unity/tutorials/custom-srp/shadow-masks/

I started looking at implementing subtractive lighting in DirectionalLight but haven't succeeded yet: https://github.com/Calinou/godot/tree/add-directionallight-subtractive-lighting

In practice, subtractive shadowmapping could look something like this: https://www.youtube.com/watch?v=lQMCy328avE
Notice how the character's real-time shadow is configured to roughly match the color of the baked shadows, yet it never darkens surfaces that are already darkened by baked shadows. To improve the appearance of dynamic objects in interiors, a subtle blob shadow can be used on top of the dynamic shadow (since the dynamic shadow only appears in exteriors).

Note that this shadowmapping mode only really makes sense for DirectionalLight, since its behavior is most likely not very suited to OmniLights and SpotLights. Also, this proposal refers to GLES3 but it's likely implementable in GLES2 and Vulkan as well.

If this enhancement will not be used often, can it be worked around with a few lines of script?

No, as shadowmapping appearance is handled by the core renderer.

Is there a reason why this should be core and not an add-on in the asset library?

See above.

[Janders1800]

Why OmniLights and SpotLights can't still be real time casting shadows using the same technique?

I mean, from the bug report I get that will cause problems if they overlap, but I feel that's more a level design problem, at least I prefer to have the option.

It can be documented so users understand the limitations of the technique.

[Calinou]

Why OmniLights and SpotLights can't still be real time casting shadows using the same technique?

This is technically possible too, it's just way lower-priority due to the aforementioned potential issues with overlapping shadows. Also, determining the correct shadow color to use is more difficult in this scenario.

I was about to link a video that showed this in action with fully baked OmniLights in the game Quetoo, but it seems they recently replaced the previous per-light stencil shadows with blob shadows…

[Calinou]

Coming back to this after a few months, I wonder if Shadowmask would be a better alternative in the end. Quoting from the Shadow Masks article I recently added to OP:

What about the Subtractive mixed lighting mode?

Subtractive lighting is an alternative way to combine baked lighting and shadows, using only a single light map. The idea is that you fully bake a light but also use it for realtime lighting. You then calculate realtime diffuse lighting for that light, sample realtime shadows, and use that to determine how much diffuse light was shadowed, which you subtract from the diffuse GI.

So you end up with static objects that use baked lighting—even though diffuse realtime lighting is calculated for them—that can receive realtime shadows. Dynamic objects have to rely on occlusion probes to receive static shadows.

It is a budget approach that is severely limited. It only works for a single directional light that cannot change. All indirect lighting or any other baked light produces incorrect results, which is mitigated by constraining the darkening via a configurable shadow color, which should match the average indirect GI color of the scene.

I won't include support for the subtractive mode in this series. If you have space for a shadow mask map then always-shadow-mask mode is superior to subtractive. If not then consider going fully-baked, which allows a more complex lighting setup.

It is possible to use a more low-end-friendly form of shadowmask known as "always shadowmask", where objects that are present in the shadow mask don't cast real-time shadows (but still affect light probes for dynamic objects). This form of baked + real-time lighting combination has less limitations and doesn't require manually setting a dynamic shadow color that looks good for the given scene.

---

I have an early WIP here, with many limitations that need to be resolved (see the commit message): https://github.com/Calinou/godot/tree/bakedlightmap-add-shadowmask-3.x

Notice how the last DirectionalLight split now fades with baked shadows, as opposed to fading with nothing. (I haven't figured sampling the directional shadow purely from the baked shadowmask yet, so it will suddenly disappear after the shadow max distance was reached.)
The cube in the middle is a dynamic object, so its shadow will fade completely as expected.

shadowmask.mp4

If we do it right, the end result should be pretty similar to something like this:

csgo-shadowmask.mp4

[Calinou]

@KiwwiPity Please don't bump issues without contributing significant new information. Use the 👍 reaction button on the first post instead.