RGBM texture banding on iOS

[vjevremovic]

Describe the bug
RGBM PNG textures on iOS are showing extreme banding.

To Reproduce
Run these examples on any iPhone/iPad
https://threejs.org/examples/#webgl_loader_texture_rgbm

https://threejs.org/examples/#webgl_materials_envmaps_hdr
(select RGBM map)

Platform
iOS iPhone/iPad

[Mugen87]

Please fill out the bug report template properly. Especially you should share a screenshot that shows the rendering issues.

[vjevremovic]

Will do that. By studying the forums it looks that it is related to the way iOS loads PNGs. Other formats like BASIS work ok, but then again they suffer from artifacts, especially in the alpha channel. I have no idea what would be the right solution.

[Mugen87]

I've tested this issue today on a iPad (8th generation) with iPadOS 14.3 and Safari. I can see the same artifacts like in the screenshot.

The RGBM example does look as expected on a Pixel 4a though. Besides, the RGBE/HDR example looks good on the iPad, too.

[mrdoob]

Do we still need RGBM? 🤔

[mrdoob]

/cc @elalish

[Mugen87]

AFAIK, encoding/decoding with RGBM is faster than RGBE. Besides, you can always use linear filtering which does only work with RGBE when using floating point textures.

[vjevremovic]

RGBM is essential for working with 8bit encoded HDR images. Alpha channel usage for holding noncolor data is also mega important.

[mrdoob]

But, can't those 8bit HDR images be converted into a RGBE DataTexture at load time?
That way we'll have less shader code paths.

[vjevremovic]

No.
If it is premultiplied after load you lose precision and get that banding from the screenshot.

[elalish]

It's true that RGBE is a bit more expensive than RGBM (since you have to do custom interpolation instead of the normal hardware version), but it also works much better, especially for very high dynamic ranges. RGBE does great with 8bit HDR images, but converting between RGBM and RGBE could easily have bugs. However, using hardware interpolation on RGBM is also wrong (just not quite as wrong as RGBE) and could easily be responsible for the banding you see. I would recommend removing RGBM support entirely, as the whole format feels like a hack to me.

[vjevremovic]

RGBM format is not a hack. It is a legitimate algorithm to pack high dynamic range data in 8bit per channel image. Banding occurs due to iOS natively premulitplying alpha on PNG loading. Premultiplying alpha channel is wrong in so many different ways, especially in 3D engines. I would suggest implementing a custom loader for PNG on iOS (based on libpng), as this would solve many other issues too.

[elalish]

Okay, but have you looked at how much error is introduced by doing naive 4-channel linear interpolation on correct RGBM data? It can get significant if the gradients are steep. Also, for any given float triple there are multiple ways to encode the value to RGBM, which makes consistency tricky.

[vjevremovic]

Of course limitations are expected when you are packing float point data in fixed point. No matter how you pack/unpack them what is important is that texture data that has alpha channel is tempered natively by iOS on loading PNGs, and that is not ok. Let's focus on that problem. RGBM or any other format (or algorithm) that is utilizing alpha channel in textures is in the same situation.

[Mugen87]

Is there a JavaScript port of libpng? As a start, we could use it to implement a RGBMLoader that resolves this issue.

[Mugen87]

@vjevremovic I've quickly hacked a RGBMLoader in based on UPNG.js. It seems to solve the issue on my iPad. How does it look on your devices?

https://rawcdn.githack.com/Mugen87/three.js/26e6e34b563df802ab2aa0058160e2a04cbed01f/examples/webgl_loader_texture_rgbm.html

[Mugen87]

There is a refactored version of RGBMLoader which also supports loading cube maps. It is now derived from DataTextureLoader.

@mrdoob If you like I can make a PR with this code so RGBM also works on iOS. If so, I'd like to add a small enhancement to DataTextureLoader so it's not necessary to overwrite load() in RGBMLoader.

Otherwise I leave my branch as it is so user can copy/paste the implementation into their project if necessary.

[vjevremovic]

@Mugen87 it works! well done!

[WestLangley]

Yes, this is much better for me, too.

Two issues, though...

1. The color profile of memorial.png is sRGB, which clearly is not correct. (Similarly for two of the six pisaRGBM16 textures.) I assume the new loader reads the color profile but does nothing with it. Is that correct?

2. Separately, the pisaHDR textures have significant banding for some reason. Look at the skybox, and compare with pisaRGBM16, which is improved.

[Mugen87]

The color profile of memorial.png is sRGB, which clearly is not correct. (Similarly for two of the six pisaRGBM16 textures.) I assume the new loader reads the color profile but does nothing with it. Is that correct?

I'm not familiar with the internals of PNG but according to the UPNG.js code the parser does honor the sRGB chunk which means the color data conforms to the sRGB specification.

[WestLangley]

I think it is the application's responsibility to honor the sRGB flag -- not UPNG.js. It just reads it. I assume the flag is ignored by the new loader, as it should be. The flag is incorrect anyway.

Note that the posY and negY textures in pisaRGBM16 have sRGB set and the other four textures do not.

So the file headers are messed up.

[Mugen87]

Note that the posY and negY textures in pisaRGBM16 have sRGB set and the other four textures do not. So the file headers are messed up.

We probably should regenerate the RGBM files to have consistency. As far as I can see here, the original tool is not available.

@vjevremovic Can you recommend a tool that converts HDR/EXR to RGBM? Otherwise the conversion has to be done with three.js.

BTW: Couldn't we store RGBM textures in TGA files instead of PNG? In this way, we could load the textures with the existing TGALoader.

[mrdoob]

@Mugen87

@mrdoob If you like I can make a PR with this code so RGBM also works on iOS. If so, I'd like to add a small enhancement to DataTextureLoader so it's not necessary to overwrite load() in RGBMLoader.

Sounds good to me!

BTW: Couldn't we store RGBM textures in TGA files instead of PNG? In this way, we could load the textures with the existing TGALoader.

Are file sizes comparable?

[Mugen87]

Are file sizes comparable?

AFAIK, TGA will produce bigger files. Besides, not all TGA files are compressed which means sometimes the size can vary noticeably.

I've mentioned TGA since I've encountered its usage in combination with RGBM, see https://github.com/box/hdrCompressor.

@MiiBond Does it make sense to use RGBM encoded TGA files on the web, too? Or should we focus on PNG?

[MiiBond]

Definitely png or maybe webp. Both of those support transparency and are good for the web.
Alternatively, Khronos is working on how to encode HDR data into BasisU-encoded KTX2 files.

ps. I forgot that this box/hdrCompressor existed out there. I created this years ago to get to a common, lossless format that I could then convert to something useful on the web. The output TGA wasn't intended to be used directly.