Cannot reliably use stream's pipeline function with an undici response's body in node v18.16.0

[fvictorio]

Bug Description

The pipeline function of stream/promises doesn't work well with a response.body in node v18.16.0 and v18.16.1. To be honest, I don't know if this is a problem in undici or in node.

Reproducible By

This snippet works, but if you uncomment the line that awaits 1s, then the process finishes in the middle of the piping:

const path = require("path")
const fs = require("fs")

async function main() {
  const { pipeline } = require("stream/promises");
  const { request } = require("undici");

  const response = await request("https://binaries.soliditylang.org/linux-amd64/list.json");

  //await new Promise(resolve => setTimeout(resolve, 1000))

  const filePath = path.resolve(__dirname, "list.json")
  await pipeline(response.body, fs.createWriteStream(filePath))

  console.log("success")
}

main()
 .catch(console.error)

The resulting list.json is corrupted (doesn't have the full JSON).

The 1s sleep is not realistic, it's just the best way I found to consistently trigger this issue on my machine. In the actual scenario, these are just a few fs calls.

Expected Behavior

The list.json file should be correctly created.

Environment

This happens in node v18.16.0 and v18.16.1, but it doesn't in v18.15.0.

Additional context

I tried uploading the same file to a GitHub gist and the same thing happened, so I don't think it's something related to that particular server.

[fvictorio]

In case someone lands here: one workaround is to immediately consume the response into a buffer and then write that:

const path = require("path")
const fsPromises = require("fs/promises")

async function main() {
  const { request } = require("undici");

  const response = await request("https://binaries.soliditylang.org/linux-amd64/list.json");

  const responseBuffer = await response.body.arrayBuffer();

  await new Promise(resolve => setTimeout(resolve, 1000))

  const filePath = path.resolve(__dirname, "list.json")
  await fsPromises.writeFile(filePath, responseBuffer)

  console.log("success")
}

main()
 .catch(console.error)

Of course, this is less performant.

[KhafraDev]

cc @ronag

[marcbachmann]

I can confirm that there's an issue with the piping. In my case the first file download never resolved with await pipeline(body, filewriteStream) (also with node 20.3.1), while subsequent requests succeeded. reading using an async iterator into memory worked properly.

const {body} = await request(...)
const chunks = []
for await (const chunk of body) chunks.push(chunk)
const buffer = Buffer.concat(chunks)

[mcollina]

I think this is fixed by nodejs/node#48435.

[marcbachmann]

The patch solves the issue with the reproduction in the issue description. I'll report back after we've tested our non-resolved promise case.