New render function with support for material properties and multiple light sources

[p-e-w]

This is a complete rewrite of the existing render code, using only this Wikipedia article as a reference.

Compared to the previous implementation, it has:

• No more black magic! Everything is clearly named and easy to follow. The algorithm itself contains no arbitrary constants. I really did not like that such cryptic code lies at the heart of Curv and this was my number one motivation for this rewrite.
• Support for per-point material reflectivity properties, retrieved via the material function (which for now is hardcoded).
• Support for arbitrarily many point light sources (which are also hardcoded in the current implementation).
• Real shadows instead of ambient occlusion. This leads to a noticeable improvement in the rendering quality; see below. While shadows are not normally part of the Phong reflection model, I realized that they would be trivial to add using ray casting, so I did.
• Everything can be controlled on a per-color-channel basis. While most physical materials don't really have different reflectivity behavior for different colors, this can be used for some interesting artistic effects.

And here is how it looks:

Old render function

New render function

I think it is clear from this comparison that the new implementation generates a superior illusion of depth, with much more complex shadows made possible by multiple independent light sources casting overlapping shadow regions. The color is also noticeably different and while I prefer the more "lively" colors I realize this might not be everyone's preference. If you want to see muted colors more similar to the old renderer you can e.g. reduce the material's ambient_reflectivity.

This PR paves the way for #73 and #74. I did see that you and @sebastien are currently discussing a much more far-reaching overhaul of the rendering system but whatever additional improvements there are going to be in the future, having rendering code that we can actually understand is surely the first step.

[doug-moen]

The rendering of shreks_donut looks really nice! Great work.

But man is it ever slow. On my 2010 Macbook Air, shreks_donut renders at 60FPS using the old algorithm, but at only 3FPS using the new algorithm. The mandelbulb renders at some small fraction of 1 FPS -- I need to fix a bug in the FPS calculator to report fractional FPS numbers to quantify how slow it is.

The original render code by Inigo Quilez is fast because it is straight line code with no loops or if statements. If statements are a lot more expensive on GPUs than on CPUs. I'm sure the biggest performance hit is from calling castRay() multiple times within the render() function. Ambient occlusion is a much cheaper way of computing shadows.

[sebastien]

As much I like the new rendering output, speed is paramount for interactive editing and parameter exploring, anything below 20FPS is going to be problematic. To me, this shows how important it is to have "pluggable renderers" in Curv. The default one could favour speed over quality, and then we might choose alternative if we want to have a better quality output, like the one by @p-e-w.

Also, I think it would be much better to have the improved phong shading implemented in Curv, so that we could tweak it on a per-sketch basis.

[p-e-w]

@doug-moen It seems that there is once again a GPU/driver issue here as I did test this and there is no way it should be anywhere near as slow as it is for you.

On my 4 year old laptop with only an integrated GPU and open source drivers, I am seeing 60 FPS with both the old and the new implementation for the default viewer window size. If I make the viewer window full screen and zoom in a lot, the framerate drops to 9 FPS for the new render function and 27 FPS for the old one. So for me, the new code is at most 3 times slower than the old one, not 20+ times like it is for you. And with my regular window size, it makes essentially no difference.

Having used the new renderer for a few days, I find it hard to go back to the old one as it looks so much better. I really want something like it to become the default, so I will try and see if I can achieve a similar effect while maintaining higher framerates. However, given the huge performance difference between your setup and mine, it's plausible that some of it is not due to the code itself but due to driver limitations or bugs. We will see.

[sebastien]

I just tested your branch today and the rendering is better for some examples, but worse for some others. For instance, the mandlebulb would require tweaking:

here we see that it seems like the ambient is too high/colors are overexposed

The performance is adequate at preview size (my machine is from 2018 and has an Intel GPU), but degrades quickly once it's more than half the screen, but do not seem as bad as what @doug-moen experienced.

Personally I think it's best to have a default rendering that looks good and is reasonablly fast and then let authors choose specific parameters tweaking and configuration to get the best results. For people who want to use Curv only for modeling, I think speed and the ability to show surface details would be key factors of the rendering algorithm. That's something I like about Thingiverse's rendering: it removes styling from the equation and displays the models on an equal basis.

Maybe your implementation can be tweaked to have more consistent quality across the examples? The ability to have more light sources and dynamic materials seems definitely nice.

[doug-moen]

The current renderer has been tweaked over time to produce reasonable results for a wide range of models. It's a compromise: not ideal for any model, but not terrible either.

The current design attempts to render colours accurately. The colour of shreks_donut is a rather drab green, sRGB.HSV (1/3, 1, .5), and that colour is what is shown in the render. The colour shown in the new renderer is brighter, more like sRGB.HSV (1/3, 1, .65).

The current design has a 'contrast' setting, which is set to 0.5 as a compromise for getting reasonable results for a wide range of models. Too much contrast changes the colours too much, not enough and you can't see the geometry. The new renderer has deeper contrast, which is a factor in why it looks better on this particular model.

I tried turning off ambient occlusion in shreks_donut, and it looks the same. The fake shadow effect must be computed from the surface normal. Turning off ambient occlusion is a performance gain, maybe I'll see if this causes a negative effect for any of the other models.

If I turn off ambient occlusion (for speed), adjust the HSV brightness to .65 in the model's colour, and increase the contrast of the lighting model to 0.7, then I get something closer to the new renderer:

The new renderer has nicer shadows, but 3x slower rendering is not acceptable as a default.

So just exposing the contrast as a tweakable parameter in the old lighting model would be a noticeable improvement in usability. Another improvement would be to change the renderer so that the light source stays at a fixed position while you rotate the object.

[doug-moen]

In the current renderer, there is an interaction between ambient occlusion and the lipschitz operator. High Lipschitz values causes the model to be rendered darker, which is bad since I want accurate colours in the default render. Disabling ambient occlusion makes this bug go away. That's another reason (other than performance) for disabling ambient occlusion in the default lighting model.

At this point, I think that the default lighting model should have 1 light source and fake shadows, based on however the current code works. To get better shadows (at a performance cost) you should change the lighting model or change its parameters.

The current lighting model is based on code written by Inigo Quilez. He has written many blog posts about lighting. Check out his web site. For example,

• https://iquilezles.org/www/articles/simplepathtracing/simplepathtracing.htm
• https://www.iquilezles.org/www/articles/outdoorslighting/outdoorslighting.htm
• https://iquilezles.org/www/articles/rmshadows/rmshadows.htm

There is a lot of diversity in Quilez's rendering techniques, too much to be captured in a simple set of parameters. It would be interesting to build a render library containing high level modules that can be composed and parameterized to reproduce all of his techniques. This library would include multiple lighting models, each with its own parameters. Some of Quilez's rendering effects are done outside of the lighting model (eg, motion blur, depth of field).

I haven't studied Quilez's rendering techniques enough to be able to propose what a high level rendering library would look like, so I don't know how to design the API. The low level 'shadertoy' API that I discussed, the 'view' function, would allow the rendering library to be prototyped in Curv. As I mentioned elsewhere, this low level API would eventually be removed once the 'new renderer' is designed, and the render library would be ported to the new renderer.

[p-e-w]

Thank you for your feedback, @doug-moen and @sebastien!

Just a quick update, I'm a bit short on time at the moment but I'm definitely still working on this. Already, I have improved the algorithm to the point of being just 10-15% slower than the existing one, while looking even better than the "New render function" screenshot above. Hopefully, I'll be able to refine and push the changes next week.

[p-e-w]

I regret to say that I won't have time to work on this PR for the foreseeable future, so I'm closing it to not be in the way or create a false impression of ongoing activity.

As stated before, I did make some more progress, but every attempt I made that looked even halfway decent was slower than the existing code.

Along the way, I managed to explore some of the Shadertoy community. While I am absolutely in awe of the incredible stuff that people create there, I found it rather difficult to learn from it because the overwhelming majority of the code is undocumented if not outright obfuscated. Many authors seem to be (re)using code they do not understand completely. A prime example is the RNG one-liner you mentioned in #59:

frac(sin(dot(xy, (12.9898,78.233)))*43758.5453123)

At least half of all Shadertoy shaders seem to contain this line, so I was eager to find out where it originates from and what the constants mean. Turns out that apparently nobody knows. People just copy and paste mystery code from one shader to another.

In that world of secrets, Inigo Quilez' articles are a ray of light (no pun intended). The shadow rendering post you linked to above was extremely helpful and I did implement the penumbra techniques described therein. The result looks amazing, though as mentioned, performance is worse than the existing code.

But those articles didn't bring me any closer to understanding what exactly the existing code does. It's quite unfortunate that the naming is so unclear (What are dom and fre? Is there some rule that states GLSL identifiers must be three characters or less?). Personally, I would consider having clear code even more important than rendering quality, because clear code can presumably be modified to give better results whereas cryptic code can only be randomly tweaked in the hope of getting some improvement by chance.

[doug-moen]

Thanks for the contribution, even if you aren't able to carry the work forward.

I agree with your comments about the code found on shadertoy. I would like to get rid of the "magic" code in Curv. It's possible that "fre" stands for "Fresnel" and "dom" might be "Discrete Ordinate Methods". I've recently been reading about the BRDF material model, and "physically based rendering". For example, this code is documented and has lots of useful links: https://github.com/McNopper/OpenGL/blob/master/Example32/shader/brdf.frag.glsl

I'm keeping this PR open until I have a chance to make some use of the code in an updated Curv shader.

[p-e-w]

@doug-moen Since you indicated that you might be interested in using some of this code in the future, here are some more observations from my now abandoned investigations:

• Ambient illumination is independent of any direction vectors. It therefore makes no sense to keep ambient coefficients on a per-light basis. The ambient_intensity field should be removed from the Light struct and kept instead in a global constant.
• The main reason why the code in this PR performs poorly is because there are multiple light sources. Huge gains can be made by reducing these to a single light source. To make the model still look good, the light position needs to be tied to the camera position so the visible part is illuminated for every viewing direction. My plan (unrealized) was to express the light direction using Euler-type angles with the camera coordinate system as a reference. These angles would then be exposed as parameters to the user. This is much easier to visualize mentally than matrix transformations.
• In the Phong model, diffuse and specular reflection are only taken into account when their direction-dependent terms are positive. As a result, the total illumination for any given point is always positive. This leads to a brightness/saturation bias with the logic from this PR, as you noted above. One solution is to subtract a "baseline illumination" constant from illumination before multiplying it with the material color, which yields a similar appearance as the original rendering code.
• The code in this PR casts a ray from the surface point in the direction of the light in order to find shadows. The better solution, however, is to instead cast a ray from outwards towards the surface point, along the light's direction vector. The difference is that with the first approach, ray marching proceeds in minuscule steps because the distance to the surface is already so small, thus even after dozens of steps the ray barely gets away from the surface point. Reversing the direction often hits either the surface point or an obstacle in a single step, which gives faster and more accurate results. To make this possible when lights are defined as directional rather than positional vectors, move along the light direction by an amount equal to the total diameter of the bounding box. This is guaranteed to be completely outside of the rendered model and thus produce accurate shadows.

[doug-moen]

The new rendering code has been added to the master branch. There is a new rendering parameter called shader, which defaults to #standard, but if you set it to #pew then you get the renderer in this PR. This parameterization is a temporary kludge to enable experimentation while I figure out what the new design will look like.