Weird texture .exr .hdr artifacts on IOS device

[TrongHieu90]

The IOS test device is Iphone 8.

Description

The texture shows some errornous banding artifacts on .exr lightmap texture. It is normal on Android device (Samsung s9) as well as desktop.

Tested on all popular browser (Firefox, Chrome, Safari on Iphone8), (Firefox, Chrome on Samsung S9)

It is normal on MAC pc.

It also happens with the official Orange Room project, which uses .hdr file for lightmap texture.

Topic on the forum with other user confirmation:
https://forum.playcanvas.com/t/weird-texture-artifacts-when-open-with-ios-iphone-8/12374

Steps to Reproduce

1. Use .exr, hdr file for lightmap texture
2. View the texture in IOS device

[slimbuck]

I haven’t been able to repro on iPhone 7:

(Using the link in the forum post)

[slimbuck]

(Though the orange room example does have lightmap banding on the same device)

[willeastcott]

It's fine on iPhone XS with iOS 13.4.1 too:

[TrongHieu90]

@slimbuck , @willeastcott
Its fine now because I fixed it (by using a HDR plugin of Photoshop to compress the 32 bit.exr to 8 bit png file)

I found out that Iphone does not support WebGL 2 (on all of the browsers), which will lead to this issue. Even with dithering setting when baking the lightmap does not fix it.

The orange room sample by playcanvas will display this issue the clearest.

[slimbuck]

Turns out the banding is due to the way safari handles alpha when loading png images. Safari on iOS applies 'premultiply alpha' to the texture at load time, which means multiplying the R, G and B channel by A. When the image is subsequently used as a texture, safari then appears to reverse this operation (dividing RGB by A) in order to return the image to roughly it's original state. However in the process the R, G, B channels loose bits compared to the original. Thus the banding.

I've created the following test to demonstrate the banding https://launch.playcanvas.com/936882.

[slimbuck]

The reason this is an issue for lightmaps is that we convert 'high dynamic range' pixels (which can contain values > 1.0) to RGBM format. This format stores a multiplication factor in the PNG's alpha channel.

In some cases, another option is to convert lightmaps directly to RGB without storing a multiplication factor and effectively clamping lighting values to a maximum of 1.0. (This is what @TrongHieu90 has done in the example scene). In this case there will be no banding on iOS, because premultiplying with an alpha value of 1.0 leaves the data unchanged. Note however that throwing away high dynamic range lighting will result in flatter and much less realistic lighting in scenes with bright lights, sunlight etc.

[slimbuck]

This bug has been logged on webkit already https://bugs.webkit.org/show_bug.cgi?id=138477.

[Maksims]

Bug has not been updated since 2014. This affects many of my clients, and it would be great to find either alternative to PNG for RGBM or other way to store HDR lightmap data, or provide more testing and details to bug report, so that they can resurrect efforts on fixing it.

[willeastcott]

Basis?

[Maksims]

Basis?

Currently it is not available for RGBM texture, nor there is a way in Editor to upload HDR texture and use basis only path.

Also ThreeJS had to solve this:
mrdoob/three.js#21145

[mvaligursky]

maybe we could use
#2418
or
#1408

[Maksims]

Just as a reminder, here is an example of what is being talked, and how severe it is (left iOS, right others):

We're doing a workaround on one project: have two assets, one with RGB and another greyscale with Alpha in it, and then create RGBA8 texture and upload it. But it is done on load times, and not great for bandwidth and cpu parsing.

[marklundin]

Has anyone tried compressing the lightmaps with Basis? Theoretically, this should work, right?

[Maksims]

Has anyone tried compressing the lightmaps with Basis? Theoretically, this should work, right?

It won't work.
RGBM is very specific of the relationship between RGB and A (M), channels, and if there is any compression that shifts colors, it will create bandings and other artifacts straight away. So compressing RGBM as is - won't work.
An alternative could be looking for alternative compressed formats that allow storing HDR data, if such exist, then this could be an alternative path.

What many have to do, to get around this - is to store RGB and A (as RGB) in two separate PNG files. And then combine in runtime.

[slimbuck]

@marklundin ASTC basis files should work fine, but will likely be quite big compared to equivalent PNG. Whereas the ETC basis files will be small but suffer from the artifacts @Maksims mentions.

[marklundin]

Hmm fair point. Does anyone know the status of AVIF for safari? I can't find the ticket, but my understanding was that it was there but not enabled as yet. This is HDR correct?

[mvaligursky]

AVIF ticket. #2418
it's HDR but only 10 bites I think ..

[mvaligursky]

In the past I had issue with banding then storing larger range lightmaps, like we do here: 0..8. I ended up storing range 0..3 or 0..4 (not sure, but it was acceptable range) and that pretty much removed the banding. Maybe we could try something like that here as well. Don't store anything in the alpha channel, but allow 0..3 range to be stored as 0..255 in the texture, and multiply it by 3 in the shader.

[marklundin]

We're trying to figure out a workaround for IOS in a current project as the lightmaps look truly terrible at the moment. I think for the time being, a flatter LDR version is preferable to the banding, so we're running through all the lightmaps and converting them so we have a set for IOS and a set for everything else.

I'm not totally sure how the range is encoded, but what would be the conversion function that would translate one of the existing lightmaps into a one that we could load for IOS so that HDR(r, g, b, a) => LDR(r, g, b, 1) if that makes any sene 😆

[mvaligursky]

HDR lightmaps are stored in the RGBM format, and this is the function we use to decode it back to linear space:

engine/src/graphics/program-lib/chunks/reproject.frag

Lines 55 to 58 in 957f717

	vec3 decodeRGBM(vec4 rgbm) {
	vec3 color = (8.0 * rgbm.a) * rgbm.rgb;
	return color * color;
	}

Note that this give you values in larger range than 0..1 (should be 0..8), so you'd need to handle it .. maybe by clamping, or scaling the range? In the past I stored 0..3 or similar range in the LDR texture, and had a fixed multiplier in the shader to bring it to full 0..8 range.

[mvaligursky]

@marklundin - see the linked PR above for a possible workaround solution by @slimbuck

[marklundin]

Thanks chaps! Super helpful as always! So this basically adds a JS decoder that avoid the native Safari version? Do you know if there's any performance overhead with this?

[slimbuck]

This version should be worse in a number of ways:

• it's an additional script adding to download size
• probably decodes slower than the built-in browser png decode
• runs on the main thread

I actually tried to reproduce banding on my iphone 7 device but didn't manage. Which version of iOS/Safari are you seeing banding?

[Maksims]

iPhone 7 Plus, in Safari has this issue.
Tons of iOS devices had this issue for years now.

[marklundin]

Latest IPad Pro/iPad/IPhone. I've not done a full test across iOS devices as I saw this was a long standing WebKit issues so assumed it was across most devices.

The additional script is ok in our case as we have a high load anyway. the decode perf might be an issue as we're already doing mesh decompression, but I'd need to test this

[slimbuck]

iPhone 7 Plus, in Safari has this issue.
Tons of iOS devices had this issue for years now.

Seems I was using the wrong lightmap. I've updated #3419 to clearly show the banding.

[willeastcott]

The original WebKit bug has now been resolved. Here's 'Orange Room' running on my iPhone XS:

Note that older devices (e.g. iPhone 6 and lower) will not get this fix.

However, I am closing this issue as this is now greatly mitigated.