Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: dataframe_serializer supports batching #293

Merged
merged 2 commits into from
Jul 29, 2021

Conversation

bednar
Copy link
Contributor

@bednar bednar commented Jul 26, 2021

Closes #291

Proposed Changes

  1. The dataframe_serializer supports batching - serialize DataFrame rows in batches corresponding to batch_size.
  2. Added an example how to import large DataFrame

Checklist

  • CHANGELOG.md updated
  • Rebased/mergeable
  • A test has been added if appropriate
  • pytest tests completes successfully
  • Commit messages are in semantic format
  • Sign CLA (if not already signed)

@codecov-commenter
Copy link

codecov-commenter commented Jul 26, 2021

Codecov Report

Merging #293 (b7070c2) into master (025e97e) will increase coverage by 0.11%.
The diff coverage is 97.93%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #293      +/-   ##
==========================================
+ Coverage   92.38%   92.50%   +0.11%     
==========================================
  Files          27       27              
  Lines        2168     2189      +21     
==========================================
+ Hits         2003     2025      +22     
+ Misses        165      164       -1     
Impacted Files Coverage Δ
...fluxdb_client/client/write/dataframe_serializer.py 98.01% <97.77%> (-0.77%) ⬇️
influxdb_client/client/write_api.py 99.06% <100.00%> (+<0.01%) ⬆️
influxdb_client/client/util/date_utils.py 96.29% <0.00%> (+7.40%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 025e97e...b7070c2. Read the comment docs.

@bednar bednar added the enhancement New feature or request label Jul 26, 2021
@bednar bednar marked this pull request as ready for review July 26, 2021 11:48
@bednar bednar requested a review from rhajek July 26, 2021 11:48
@bednar bednar merged commit cf21862 into master Jul 29, 2021
@bednar bednar deleted the feat/dataframe_serializer_support_batching branch July 29, 2021 05:31
@bednar bednar added this to the 1.20.0 milestone Jul 29, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Pandas Dataframe writing using tons of RAM with large datasets
3 participants