Files occasionally being truncated when downloading

[Jake7D]

Issue

We're seeing occasionally that files are being truncated when they are downloaded without the storage library throwing an error.
What we're trying to do is concurrently pull down a set of files from a location to then process them. We also exclude any 0 byte files when we download. We create multiple pubsub subscriptions when doing this as the node version doesn't support concurrent messaging in pubsub in the way we need it to.
We're seeing this happening around 5% of the time we download files.

Environment details

• OS: Windows
• Node.js version: 16.16.0
• npm version: 8.11.0
• @google-cloud/storage version: 6.2.0

Steps to reproduce

Difficult to explain so including a snippet of what we're doing here. We're using storage alongside the pubsub library so including the code for that, as we have a suspicion that might be causing some of the issues.

const pubsub = new PubSub({ projectId });
const storage = new Storage({ projectId });
const subscriptions = [...Array(5).keys()].map(() => pubsub.subscription(validReleaseSub, {
  ackDeadline: 600,
  flowControl: {
    allowExcessMessages: false,
    maxMessages: 5
  }
}));

for(const subscription of subscriptions) {
  subscription.on('message', async (message) => {
   const bucket = storage.bucket(message.bucket);
   const [files] = await bucket.getFiles({ prefix: `${message.location}/` });

   await Promise.all(
     files.filter(file => file.metadata.size != 0).map(async file => {
      const destination = file.name.replace(message.location, destinationPath);
      await file.download({destination});
    });
   );

  // process the files
  });
}

[Jake7D]

So we've also noticed this seems to start happening when the CPU ramps up on the box (we have files delivered from an external source to the bucket, so they come in sporadically in varying numbers)

[danielbankhead]

Hey @Jake7D, thanks for your patience - a fix has been merged. The next release will public shortly:

• chore(main): release 6.4.2 #2050

To better mitigate and catch issues like this in testing in the future we're planning to implement a revamp of our stream implementation in the next major release:

• refactor(deps)!: Remove duplexify dependency + Revamp Stream Implementation #2041

[Jake7D]

Perfect thanks @danielbankhead. We'll give it a try and feedback if we get any other problems