GLTF KHR_materials_anisotropy support

[elalish]

Related issue: #9250

Description

The PBR Next working group at Khronos is actively working on an anisotropy extension now, so I'm starting work to implement it in order to find any flaws early. The spec has not been updated recently, so don't read it closely just yet. The current idea is to store anisotropy as a two-channel vector texture, with a scalar strength multiplier. This means the shader will have to decompose it into polar coordinates (strength and angle), but it will allow better texture compression and better interpolation.

We want to wait until the extension is settled before merging this, but this will allow our progress to be followed more openly. FYI @emackey.

Results so far for an IBL that represents a directional light (left) vs the actual directional light (right):

See for yourself in this example.

[LeviPesin]

Not sure what happened with the job https://github.com/mrdoob/three.js/actions/runs/4288360805/jobs/7470277469 on this PR... Seems like a GitHub issue because the job itself completed just in a few minutes.

[elalish]

@WestLangley I still haven't actually implemented anisotropy here yet, but I realized I'll need the tangent space for it. It wasn't very accessible, so I did this refactor of the normal map shader chunks. There should be no change in behavior here so far, and I think this both simplifies the code and reduces some duplicate calculations in the clearcoat normal map. But, it would be great to get some early feedback here since I want to base my anisotropy calculations on this.

[elalish]

Okay, I have a draft implementation of anisotropy for both direct and indirect lighting. This is largely following Filament's implementation (I need to add links), except for the parameterization and mapping of the inputs.

[elalish]

This is the key mapping we're trying to figure out for the spec - every renderer seems to use a different mapping, so we're looking to ensure the encoded values are perceptually linear-ish, and that the direct and IBL approximations agree decently well.

[elalish]

Another refactor here is to make the Clearcoat BRDF the special function instead of the iridescence BRDF. This is nice because clearcoat always uses the simple version, whereas the base BRDF get adjusted by several things, so far iridescence and anisotropy. This feels cleaner to me, hopefully others agree, especially since I removed a lot of the function inputs.

[elalish]

However, this did also necessitate moving the PhysicalMaterial definition into its own chunk so it could be included earlier.

[elalish]

Oh, I see the problem - okay, coming back tomorrow.

[elalish]

Okay, I did a different refactor instead due to troubles with include order. Basically bsdfs.glsl.js had mostly Physical BRDFs that are only used by lights_physical_pars_fragment.glsl.js, so I just moved those functions into that file (a little big now, but self-contained). The only other function used broadly was BRDF_Lambert, so I moved that to common.glsl.js. Now bsdfs.glsl.js is just BlinnPhong functions, and no longer imported by meshphysical.

Sorry this makes the PR a bit hard to read - probably best to look at commits individually.

[elalish]

I just switched this from Draft because for some reason the CI stopped running on my PR and I thought that might fix it, but no luck. Is there some quota I exceeded? Oh well, I guess I'll just run it locally for now...

[mrdoob]

I just switched this from Draft because for some reason the CI stopped running on my PR and I thought that might fix it, but no luck. Is there some quota I exceeded? Oh well, I guess I'll just run it locally for now...

/ping @LeviPesin

[LeviPesin]

No idea 🤷‍♂️ You can enable actions in your fork (i.e. https://github.com/elalish/three.js) to at least run them there, but no idea why they no longer work in this PR...

[LeviPesin]

Can you merge dev into your branch? Maybe it's something related to not updated actions.
But this is also strange -- deprecated versions were planned to be removed only on 1 June and 22 September...

[LeviPesin]

Ah, it seems it was just a GitHub Actions' outage.

[elalish]

Okay, shaders are all working, but still need to test the direct lighting path.

[elalish]

Anyway, there's a fair bit of disagreement on this one, even between the path tracers, and ours looks decent to me, so I think it's fine for now: https://modelviewer.dev/fidelity/#khronos-LightsPunctualLamp

[elalish]

@mrdoob Okay, so the one problem I've found with your UVs update is that IBLs now fail to compile when we have anisotropy, no map, and no tangents. The reason is that even if no map is using the UVs, anisotropy still needs them to calculate tangents if they're not supplied. I also noticed you duplicated some code in normal_fragment_begin.glsl, which is kind of related. There's no real difference between tbn and tbn2, we're just unsure which UV coordinate name is available (they'll all give the same results). It might be nice to have some "master uv" that just grabs the first set that's defined for cases like this.

[donmccurdy]

... there's a fair bit of disagreement on this one, even between the path tracers ...

I'm been beginning to wonder whether it's really possible to transport punctual lights effectively between engines, without also controlling more of the image formation chain. At minimum, setting up equivalent exposure in each engine, where exposure = 1 in one is not necessarily the same thing as exposure = 1 in another. Blender's new "Lighting Mode: Standard" export option makes a good case study here.

... we're just unsure which UV coordinate name is available (they'll all give the same results)

Are we sure TEXCOORD_0 and TEXCOORD_1 are going to give the same results? This seems to be an open question under KhronosGroup/glTF#1798 (comment), with the current proposal being that vertex tangents are required if anisotropyTexture is not defined.

[emackey]

@donmccurdy With yesterday's draft anisotropy update, I've added this proposed wording. What do you think?

All meshes that use nonzero anisotropy strength SHOULD supply TANGENT vectors as a mesh attribute. If TANGENT vectors are not supplied for such a mesh, TEXCOORD_0 MUST be supplied instead, such that tangent vectors can be automatically calculated.

I'm honestly not certain about hard-coding the zero in the spec like that, but the anisotropyTexture is optional where the tangents are required, and it feels perhaps too cumbersome to specify a bunch of logic for looking up the TEXCOORD number based on this map if it happens to exist, and what to do if it does not exist.

Sorry, this is off-topic here though, feel free to post a reply on KhronosGroup/glTF#1798.

To directly answer the question:

Are we sure TEXCOORD_0 and TEXCOORD_1 are going to give the same results?

No, the "U" direction can be completely different for alternate maps, so the tangents can flow different ways based on the TEXCOORD index number. Makes me wonder why glTF has only the one TANGENT attribute, actually, but at least that singularity may be of use to anisotropy.

[donmccurdy]

Potential conflicts here:

#	anisotropyTexture	normalTexture	TANGENTS	outcome
A	TEXCOORD_0	TEXCOORD_0	✓	✅ use existing tangents
B	TEXCOORD_0	TEXCOORD_0		✅ generate tangents with TEXCOORD_0
C	TEXCOORD_0	TEXCOORD_1	✓	⚠️ tangents valid for which UVs?
D	⨉	TEXCOORD_1		⚠️ how to generate tangents?
E	TEXCOORD_1	TEXCOORD_0	✓	⚠️ tangents valid for which UVs?
F	TEXCOORD_1	TEXCOORD_0		⚠️ how to generate tangents?

In cases D+F, the implementation would need to calculate TANGENTS for the normal map and anisotropy with competing sources of texture coordinates. This is possible (if unwieldy) for a runtime, but impossible for a processing tool like glTF Transform or gltfpack trying to output valid glTF files.

I'm not sure what it'd mean if TANGENTS are present, but normalTexture and anisotropyTexture use different texture coordinates — one or the other must be incorrect?

I don't love the idea of having multiple sets of tangents in the file — it's really rare that we need to generate MikkTSpace tangents at all, computing tangents in the shader works well for a large majority of normal maps we see in three.js. I generally recommend users bake vertex tangents with MikkTSpace only when the alternatives fail; MikkTSpace is too expensive for us to do by default. But perhaps something like TANGENTS_n should be considered in a hypothetical glTF 3.0.

In the meantime, would it make sense to say anisotropyTexture is restricted to use the same UV layout as normalTexture, falling back to TEXCOORD_0 if neither is defined?

[elalish]

In the meantime, would it make sense to say anisotropyTexture is restricted to use the same UV layout as normalTexture, falling back to TEXCOORD_0 if neither is defined?

I like this idea. I think that means in three we could combine normalMapUv and anisotropyMapUv. Looking at the code, it would also be great to combine clearCoatNormalMapUv, but not sure if we can get away with changing the spec at this point. I'd love to understand better the multi-UV channel thing in glTF - individual texture transforms I get (for tiling) but I don't quite understand having whole separate mappings.

[elalish]

I believe this is ready for final review and merging. I've added an example and the spec has been updated such that our handling of UVs is now compliant, as they are not allowed to mix and match.

[emackey]

@elalish Congrats on the merge!

One wrinkle, looks like the AnisotropyBarnLamp.glb sample model merged here contains one incorrect (old) name, anisotropyDirectionTexture instead of anisotropyTexture in the material's extension JSON. I just pushed an updated GLB to KhronosGroup/glTF-Sample-Models#366.

The loader code looks good to me as far as parameter names in the glTF. Awesome work here.

[elalish]

@emackey Ah, that makes sense why it was looking a little odd then. Eric said he's almost done updating the model (from an artistic standpoint) anyway, so I'll probably wait for that merge and then update it.

[Mugen87]

@elalish It looks like there is an issue with this particular line of code. Users report about assets that throw a shader error now:

Program Info Log: Fragment shader is not compiled.


FRAGMENT

ERROR: 0:1384: 'vNormalMapUv' : undeclared identifier
ERROR: 0:1384: 'getTangentFrame' : no matching overloaded function found
ERROR: 0:1384: '=' : dimension mismatch
ERROR: 0:1384: '=' : cannot convert from 'const mediump float' to 'highp 3X3 matrix of float'


  1379: #endif
  1380: #if defined( USE_NORMALMAP_TANGENTSPACE ) || defined( USE_CLEARCOAT_NORMALMAP ) || defined( USE_ANISOTROPY )
  1381: 	#ifdef USE_TANGENT
  1382: 		mat3 tbn = mat3( normalize( vTangent ), normalize( vBitangent ), normal );
  1383: 	#else
> 1384: 		mat3 tbn = getTangentFrame( - vViewPosition, normal, vNormalMapUv );
  1385: 	#endif
  1386: 	#if defined( DOUBLE_SIDED ) && ! defined( FLAT_SHADED )
  1387: 		tbn[0] *= faceDirection;
  1388: 		tbn[1] *= faceDirection;
  1389: 	#endif
  1390: #endif

It seems not right to use USE_CLEARCOAT_NORMALMAP and USE_ANISOTROPY at this point since vNormalMapUv is only defined when a normal map is present (meaning when USE_NORMALMAP is defined).

What was the reason for adding both defines to this check?

An asset for reproduction is available here: https://discourse.threejs.org/t/shader-error-0-validate-status-false/53195

[elalish]

Ah, I may have missed that case. I did add this to ensure the tangents were defined, but perhaps a similar thing is necessary for the UVs? Anisotropy requires UV coordinates (and ideally tangents too), even if no normal map is present. I think it's perfectly reasonable to throw an error if anisotropy is requested without any UV channels being present (that's in the glTF extension spec too).

[repalash]

Hey sorry to jump in, I was debugging this bug and got here.
@elalish It's possible to have UVs without a normal map being present, which won't create the vNormalMapUv in the shader.
I solved it by setting USE_UV in material.defines that will make vUv available and use that when normal map is not available.

[elalish]

Yes, I think that's the solution - to set USE_UV in the WebGLRenderer when anisotropy is present even if there's no normal map. Want to try a PR? I'll be happy to review.

[Mugen87]

It's also necessary to support the clear coat case. Otherwise the car asset will still fail (it does not use anisotropy).

[Mugen87]

Besides, I don't think USE_UV solves the above issue. USE_NORMALMAP defines the vNormalMapUv varying which is required to make the shader code work.

[elalish]

Yeah, it would be nice if the UVs were a little less closely-tied to particular maps, but I'm not sure how much surgery you want to do, @Mugen87. Basically we just need a UV coordinate when anisotropy is used, even if maps aren't present. I think there is some tension between the smallest change to make that work, and the most readable.

[Mugen87]

I'm okay with a hot fix (smallest change) for now. Things can be refactored later.

[repalash]

I created a small PR #26334 With these changes the model from sketchfab loads without throwing an error.