Accessing big files (greater than ~2GB) requires changing the File API to use 64-bit integers #400
Comments
|
This will be handled by godotengine/godot#27247, but due to the large scope of the change, it's on hold until 4.0. |
|
Oh, I didn't saw that issue because I didn't knew that was because of the 32 bit system to read data. :) |
fire
changed the title
akien-mga
changed the title
akien-mga
changed the title
|
I have played around the 2G problem under x64 windows, made minor playground changes in the file_access_windows.cpp. The .pck export went well and the project started ok for now... |
|
You may want to have a look to this old PR of mine. It's of course outdated, but I think the changes to the file system classes are still relevant: godotengine/godot#27247 It also can serve as a guide about other parts of the engine that can benefit. |
|
@RandomShaper I skimmed thru your commit. The move to switch to 64bits for file i/o api was a very good! I was happy that "it only needed" to fix file_access_windows.cpp. I'm not brave enough to make a pull request yet, because my fixed build was not tested/used enough in our environment. I also did not even try to compile for win32 at all... |
|
This is now handled in godotengine/godot#47254. |
oh no. the PCK file exceed 2GB and when we run the export it doesn't work. we don't know how to reduce the import less than 2GB as this is the Godot bug in file system access. - godotengine/godot#27168 pck more than 2.1 gb error - godotengine/godot#44363 manually put an export template in root folder of this project - https://godotengine.org/qa/75329/game-over-and-not-working-what-best-handle-this-godot-stable - godotengine/godot-proposals#400 it seems that it will only fix in Godot 4.0 - https://docs.godotengine.org/en/stable/getting_started/workflow/export/exporting_pcks.html - godotengine/godot#44363 - godotengine/godot#47254 akien is fixing, but this PR here is still WIP for now unfortunately so now we added 2 export template: Windows and Linux, both 64 bit in the project root folder. simply run HexagonEngine.bat for Windows or HexagonEngine.sh for Linux respectively. those script will print "Hexagon Engine by Perkedel Technologies" then run respective export template.
This changes the types of a big number of variables. General rules: - Using `uint64_t` in general. We also considered `int64_t` but eventually settled on keeping it unsigned, which is also closer to what one would expect with `size_t`/`off_t`. - We only keep `int64_t` for `seek_end` (takes a negative offset from the end) and for the `Variant` bindings, since `Variant::INT` is `int64_t`. This means we only need to guard against passing negative values in `core_bind.cpp`. - Using `uint32_t` integers for concepts not needing such a huge range, like pages, blocks, etc. In addition: - Improve usage of integer types in some related places; namely, `DirAccess`, core binds. Note: - On Windows, `_ftelli64` reports invalid values when using 32-bit MinGW with version < 8.0. This was an upstream bug fixed in 8.0. It breaks support for big files on 32-bit Windows builds made with that toolchain. We might add a workaround. Fixes godotengine#44363. Fixes godotengine/godot-proposals#400. Co-authored-by: Rémi Verschelde <[email protected]>
This changes the types of a big number of variables. General rules: - Using `uint64_t` in general. We also considered `int64_t` but eventually settled on keeping it unsigned, which is also closer to what one would expect with `size_t`/`off_t`. - We only keep `int64_t` for `seek_end` (takes a negative offset from the end) and for the `Variant` bindings, since `Variant::INT` is `int64_t`. This means we only need to guard against passing negative values in `core_bind.cpp`. - Using `uint32_t` integers for concepts not needing such a huge range, like pages, blocks, etc. In addition: - Improve usage of integer types in some related places; namely, `DirAccess`, core binds. Note: - On Windows, `_ftelli64` reports invalid values when using 32-bit MinGW with version < 8.0. This was an upstream bug fixed in 8.0. It breaks support for big files on 32-bit Windows builds made with that toolchain. We might add a workaround. Fixes godotengine#44363. Fixes godotengine/godot-proposals#400. Co-authored-by: Rémi Verschelde <[email protected]>
Describe the project you are working on:
I want to make a file manager, for making it easier to arange data.
Describe the problem or limitation you are having in your project:
Godot is not able to deal with data bigger than 2^31 -1 // (~2GB) bytes. So I can't seek behind that point or get the size (wich is just equal to seek_end() -> get_position()). So it's not possible to store or read data behind this value. And working with files beeing that big is very likely espacialy when managing "zip" folders or movies. I see no way to avoid that problem.
Describe how this feature / enhancement will help you overcome this problem or limitation:
It would be possible to accese the size of files bigger than ~ 2GB and data behind that point, e.g for compressing purpose. In Godot 3.2 it even it's not possible anymore to get the last date due the size limitaion.
Show a mock up screenshots/video or a flow diagram explaining how your proposal will work:
For e.g accesing the file's size:
var file : File = File.new()var err : int = File.open(Path)File.get_len()Describe implementation detail for your proposal (in code), if possible:
I see no solution in GDCode. I googled a lot to find a way, but it's just not possible to overcome the problem
If this enhancement will not be used often, can it be worked around with a few lines of script?:
Here goes the same as for the last question: I see now way in native Godot
Is there a reason why this should be core and not an add-on in the asset library?:
It is for internal use in the engine only.
The text was updated successfully, but these errors were encountered: