Open region file represented with []byte

[maxsupermanhd]

Hi, I am making http accept for regions and I want to parse it on the fly, is it possible?

[Tnze]

Theoretically, yes. The current API, no.
Can you explain why you want to do this? Is this a performance bottleneck? Parsing data on the fly will make memory occupied larger, because parsing does not access data in a completely linear order.

[maxsupermanhd]

I am working on server that stores chunks, basically a live world backupper and map. Clients submit chunk/region data to it and it stores raw byte arrays of them in database (takes very little space apparently compared to files on disk, 250mb to 1.7mb). I already made server manager, dimension manager and chunk storage. Already implemented map rendering by just literally ripping out example that you have and slapped monkey-primitive scaling to it and cut out threading because leaflet already threads it just by requesting a lot of tiles (5-8) at the same time.
I decided to implement region structure submit too because it is more optimal to do so if I have loads of chunks, it helps avoid rate limiting if present and theoretically faster because less http request/resonse mess. Single upload - 32*32 chunks dropped to database (takes up like 15 seconds to fully process region in single thread).
Currently I made it work via temp file: dropping http body to file and them feeding it to region reader and cleaning up when done.
This is what I currently have already: (rendered on the fly - no caching yet lol)
(yes it is watercubed obsidian monument from simplyvanilla.net)

[Tnze]

Sounds like a incremental backup system.

I don't quite understand. But it seems that the unit you read and process should be a single Chunk, that is, 16*16 Blocks. When store them into the database, render the map and cache it. In this way, when requested by leaflet, you only need to simply return the generated image.

[maxsupermanhd]

It actually is an incremental backup system for chunks. You understood everything correctly, data "unit" is a Chunk. In order to optimize Chunk submitting to the system I thought about sending over whole region file and parsing and breaking it up on server to compensate on traffic usage.

[Tnze]

And this is the reason why you want to parse mca files on the fly.

I decided to implement region structure submit too because it is more optimal to do so if I have loads of chunks, it helps avoid rate limiting if present and theoretically faster because less http request/resonse mess.

But block changes area usually are limited to the size of Chunk, aren't they? Submit the hole Region could increase the IO pressure.

Considering the first submitting, the http mess that you say could happen. You could packet chunks together in your own ways to avoid that mess. In this protocol, the coordinates of blocks are represented separately, so region cannot be used -- it can only store adjacent blocks.

[Tnze]

And parse regions on the fly will be support. I am working on it.

[maxsupermanhd]

Also speaking of chunks and loading, there is no support for uncompressed nbt int chunk load method.

[Tnze]

Commit 684d257 support parsing regions in memory. You may need to find an implement of io.ReadWriteSeeker, which doesn't exist in standard library as I know.

The loading and parsing of Chunks are still semi-finished products. The reason as you can see, due to frequent changes between versions, it is not easy to maintain. It also related to #110.