Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Need a method for writing and reading Pint-Pandas dataframes #33

Open
MichaelTiemannOSC opened this issue Apr 27, 2022 · 3 comments
Open

Need a method for writing and reading Pint-Pandas dataframes #33

MichaelTiemannOSC opened this issue Apr 27, 2022 · 3 comments
Assignees
@erikerlandson @caldeirav
Labels
help wanted Extra attention is needed

Comments

@MichaelTiemannOSC
Copy link
Contributor

The new pint-demo notebook illustrates the fact that we don't yet have a defined way to write (let alone read) Pint-Pandas dataframes. There are several directions we can go, which we have discussed. We should use this test case to make a decision, document it, and publish it.

https://github.com/os-climate/data-platform-demo/blob/master/notebooks/pint-demo.ipynb
@MichaelTiemannOSC MichaelTiemannOSC added the help wanted Extra attention is needed label Apr 27, 2022
MichaelTiemannOSC referenced this issue in os-climate/data-platform-demo Apr 28, 2022
@MichaelTiemannOSC
Demonstrate a hack-y way to support Pint<->SQL interop
0693da4 
pd.to_sql can deal with Pint Pandas dataframes (provided all columns are some kind of pint[] type.  But reading back the dataframe requires quite a bit of unpacking.

Addresses (somewhat) https://github.com/os-climate/data-platform-demo/issues/50
@MichaelTiemannOSC
Copy link
Contributor Author

Cross-referenced with: hgrecco/pint-pandas#123

@MichaelTiemannOSC
Copy link
Contributor Author

Updated the above notebook to provide complete round-tripping. Should provide an easy way to "borrow" functions into osc-ingest-tools.

@MichaelTiemannOSC MichaelTiemannOSC transferred this issue from os-climate/data-platform-demo May 27, 2022
@MichaelTiemannOSC
Copy link
Contributor Author

Potential candidate here: https://github.com/os-climate/PCAF-sovereign-footprint/blob/develop_mutlu-dbt/notebooks/pcaf_env.py

Especially the functions requantify_df and dequantify_df. So maybe we should migrate this functionality (that is duplicated across multiple ingestion pipeline repos) into osc-ingest-tools.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@erikerlandson erikerlandson

@caldeirav caldeirav

Labels
help wanted Extra attention is needed
Projects
Data Commons Platform
Status: No status
Milestone
No milestone
Development

No branches or pull requests
3 participants
@erikerlandson @caldeirav @MichaelTiemannOSC