feat(gatsby-core-utils): improve fetch-remote-file #34758
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gatsbot
bot
added
the
status: triage needed
wardpeet
changed the title
wardpeet
added
topic: core
wardpeet
force-pushed
the
feat/remote-file-update
branch
from
9219a41 to
c221936
Compare
wardpeet
commented
Feb 8, 2022
| @@ -0,0 +1,269 @@ | |||
| import fs from "fs-extra" | |||
There was a problem hiding this comment.
This file stayed untouched
There was a problem hiding this comment.
I changed a few things after all
wardpeet
force-pushed
the
feat/remote-file-update
branch
from
c221936 to
c862f5a
Compare
wardpeet
force-pushed
the
feat/remote-file-update
branch
from
c862f5a to
3a30cf4
Compare
wardpeet
commented
Feb 8, 2022
| @@ -380,316 +363,6 @@ describe(`fetch-remote-file`, () => { | |||
| expect(gotStream).toBeCalledTimes(1) | |||
| }) | |||
|
|
|||
| it(`only writes the file once when multiple workers fetch at the same time`, async () => { | |||
There was a problem hiding this comment.
Moving this to integration test
wardpeet
force-pushed
the
feat/remote-file-update
branch
2 times, most recently
from
7114493 to
7d4a00e
Compare
LekoArts
reviewed
Feb 9, 2022
...gatsby-source-wordpress/src/steps/source-nodes/create-nodes/create-remote-file-node/index.js
Outdated
Show resolved
Hide resolved
wardpeet
commented
Feb 9, 2022
pieh
reviewed
Feb 10, 2022
wardpeet
force-pushed
the
feat/remote-file-update
branch
from
7d4a00e to
e40f1f6
Compare
wardpeet
commented
Feb 13, 2022
| // Called if we stall for 30s without receiving any data | ||
| const handleTimeout = async (): Promise<void> => { | ||
| fsWriteStream.close() | ||
| await fs.remove(tmpFilename) |
There was a problem hiding this comment.
made this async instead of sync for better windows support
There was a problem hiding this comment.
What the catch with sync fs.remove on windows?
| } | ||
|
|
||
| fsWriteStream.close() | ||
| await fs.remove(tmpFilename) |
There was a problem hiding this comment.
made this async instead of sync for better windows support
|
|
||
| // We have an incomplete download | ||
| if (!haveAllBytesBeenWritten) { | ||
| await fs.remove(tmpFilename) |
There was a problem hiding this comment.
made this async instead of sync for better windows support
wardpeet
force-pushed
the
feat/remote-file-update
branch
5 times, most recently
from
f813d2a to
b564e98
Compare
wardpeet
force-pushed
the
feat/remote-file-update
branch
from
0412bac to
56b467b
Compare
This was referenced May 28, 2022
This was referenced May 28, 2022
This was referenced Jul 21, 2022
This was referenced Mar 28, 2023
mwfrost
pushed a commit
to mwfrost/gatsby
that referenced
this pull request
Apr 20, 2023
* feat(gatsby-core-utils): improve fetch-remote-file * feat: use cacheKey in fetch-remote-file * use async file operations * Improve tests + low hanging fruit * revert wordpress * feat: make sure 304 works * fix jest config wordpress * add integration test * update tests
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Improving remote fetch by adding two bits:
fetch-remote-file is now decoupled from gatsby's cache. It still functions roughly the same but if you give it a proper cacheKey like contentdigest of a node we can keep the file around between builds even if package.json or gatsby got changed.
Documentation
FetchRemoteFile is not documented yet, will do that afterwards.
TODO