Alignment

[mrdoob]

What can't you do right now?

Recently I've been trying to generate USDZ files in JavaScript: mrdoob/three.js#21245

The USDZ file is essentially an zip file with 0 compression that contains a 3d file and textures.

So far fflate has been great for creating the file. However, yesterday I learned that in order to produce valid USDZ files I need to make sure that the files are aligned to 64 bytes. Currently my files are invalid:

From the spec page:

I'm aware this is quite an uncommon use case, but would you be interested in adding support for this?

[101arrowz]

Interesting, I always considered the "extra" field to be useless because literally no software (besides proprietary tools like Archive Utility) reads it. Adding support for this would not be too difficult, I'll publish an update in a few hours.

Since you need to align to 64 bytes, you probably need to know what the size of the header for each file is. I'll update with a calculation in a bit.

[101arrowz]

@mrdoob I've published v0.6.2 to NPM, which includes support for comments and extra fields. Both of these can hold arbitrary user data, e.g. a bunch of zeros until you reach a 64 byte boundary. However, only the extra field exists in local file headers, so we need to use it over null comments. Here's some sample code to get you started:

// Let's say you have a standard ZIPpable object:
const toZip = {
  'someFile': fflate.strToU8('someContent'),
  'otherStuff': fflate.strToU8('you get the point')
};

// We can mutate this object to add the padding

// Byte offset we are currently at
let offset = 0;

for (const filename in toZip) {
  const file = toZip[filename];
  // This is explained after this code block
  const headerSize = 34 + filename.length;
  offset += headerSize;

  // Bitwise AND with 63 is equivalent to mod 64
  const offsetMod64 = offset & 63;
  // If the offset is 4, after we remove the assumed
  // 4 bytes wasted in the extra field no matter what,
  // we are 0 mod 64 and don't need to add the extra
  // field to pad this file.
  if (offsetMod64 != 4) {
    // Adding this padding gets us to 0 mod 64
    const padLength = 64 - offsetMod64;
    const padding = new Uint8Array(padLength);
    toZip[filename] = [file, {
      extra: {
        // Any ID above 31, below 65536 works
        12345: padding
      }
    }];
  }
  // Reset offset. Although we could add the
  // file length, since we just became 0 mod 64,
  // setting instead of adding works too.
  offset = file.length;
  
}
// Zip as normal
const uszdOutput = fflate.zipSync(toZip, { level: 0 });

The local header size is 30 bytes + size of the filename in UTF-8 + extra fields sizes + number of extra fields * 4. Since you have 1 extra field and your filenames are all ASCII (presumably), this becomes 34 + length of filename + length of extra field. The above code uses that calculation to align each uncompressed file to a 64 byte boundary.

[mrdoob]

Awesome! I'll implement it right away 🙏

Hmm, I guess I need to generate a new fflate.min.js and fflate.module.min.js.

How did you generate these? I don't see scripts in this repo to generate them 🤔

mrdoob/three.js#20941 (comment)

[101arrowz]

You can take a look at the CDN instructions in the README and open those URLs in the browser. UMD and ESM (use the first link that is imported)

[mrdoob]

Excellent! Many thanks! 🙏
mrdoob/three.js@8000053