Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to Convert the data fetched from Query to a Pandas Dataframe? #30080

Closed
Keerthesre1 opened this issue Jan 23, 2024 · 0 comments
Closed

How to Convert the data fetched from Query to a Pandas Dataframe? #30080

Keerthesre1 opened this issue Jan 23, 2024 · 0 comments

Comments

@Keerthesre1
Copy link

import apache_beam as beam
import pandas as pd
from psycopg.rows import dict_row
from datetime import datetime
import pandas as pd
from beam_postgres.io import ReadAllFromPostgres

with beam.Pipeline() as p:
print(datetime.now())
data = p | "Reading example records from database" >> ReadAllFromPostgres(
"host=host dbname=db user=user password=passowrd",
'SELECT _id FROM "62a9c0ec85354b161b80df98" LIMIT 10000000',
dict_row,
)
data | "Writing to stdout" >> beam.Map(print)
print(datetime.now())

From the above code i am getting ouput in the {"key": "value"} pair but i dont know how to convert to a Dataframe. Can someone help me with this.

@github-actions github-actions bot added this to the 2.61.0 Release milestone Nov 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant