Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimize the memory usage of lightning when importing parquet data format files #53466

Closed
zeminzhou opened this issue May 22, 2024 · 3 comments · Fixed by #53474
Closed

Optimize the memory usage of lightning when importing parquet data format files #53466

zeminzhou opened this issue May 22, 2024 · 3 comments · Fixed by #53474
Labels
type/enhancement The issue or PR belongs to an enhancement.

Comments

@zeminzhou
Copy link
Contributor

zeminzhou commented May 22, 2024 •
edited
Loading

Enhancement

Update xitongsys/parquet-go to optimize the memory usage of lightning.
Before update:
img_v3_02b4_878ab795-c34b-42f4-8742-0236be09512g

After update:
img_v3_02b4_69009e5a-b4e3-4287-87ab-afbc3cafbe5g
@zeminzhou zeminzhou added the type/enhancement The issue or PR belongs to an enhancement. label May 22, 2024
@lance6716
Copy link
Contributor

there's another more official package https://pkg.go.dev/github.com/apache/arrow/go/parquet I don't know why we choose current one 😂

@zeminzhou
Copy link
Contributor Author

zeminzhou commented May 22, 2024 •
edited
Loading

It's really confusing, do you know why? /cc @D3Hunter

@zeminzhou zeminzhou mentioned this issue May 22, 2024
Merged
13 tasks
@D3Hunter
Copy link
Contributor

D3Hunter commented May 23, 2024 •
edited
Loading

It's really confusing, do you know why? /cc @D3Hunter

sadly no 😂

@zeminzhou zeminzhou changed the title Optimize the memory usage of lightnnig when importing parquet data format files Optimize the memory usage of lightning when importing parquet data format files May 23, 2024
ti-chi-bot bot pushed a commit that referenced this issue May 23, 2024
@zeminzhou
lightning: update parquet-go to optimize the memory usage when impo…
22dc064 
…rting parquet files. (#53474)

close #53466
RidRisR pushed a commit to RidRisR/tidb that referenced this issue May 27, 2024
@zeminzhou @RidRisR
lightning: update parquet-go to optimize the memory usage when impo…
61851c9 
…rting parquet files. (pingcap#53474)

close pingcap#53466
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Milestone
No milestone
3 participants
@lance6716 @D3Hunter @zeminzhou