Data is not inserting into InfluxDB 2.0 through python client influxdb_client

[ssaurav-1987]

I have written python script in that I am collecting data from spreadsheet and making list of json data that will be json object that have list of json data. I have configured bucket, organisation, token. And tried to insert DB but it is not inserting into DB and even I am not getting error into that
Please find below my code and kindly help me where I am missing:

#!/opt/anaconda3/bin/python3
from influxdb_client import InfluxDBClient
from influxdb_client.client.write_api import ASYNCHRONOUS
import random
import json
from datetime import datetime
import pandas as pd
import logging

logging.basicConfig(filename='/InfluxDB/test.log', format='%(filename)s: %(message)s',level=logging.DEBUG)
excel_data_df = pd.read_excel('/RouterSwitchData.xlsx')
json_data = json.loads(excel_data_df.to_json(orient='records'))
series=[]
counter=0
json_body={}
for item in json_data:
current_traffic_signal = random.uniform(1.00,8.00)
json_body = {
"measurement": "devicelink",
"tags": {
"devicename": item["DeviceName"],
"device_circle": item["Circle"],
"device_subcircle":item["SubCircle"],
},
"time": datetime.now().strftime("%Y-%m-%d %H:%M:%S.%f"),
"fields": {
"device_name": item["DeviceName"],
"device_ip_address":item["IP_Address_Device"],
"device_port":item["port"],
"device_circle":item["Circle"],
"device_subcircle":item["SubCircle"],
"neighbour_device_name":item["Neighbour_Device"],
"neighbour_device_ip_address":item["IP_Address_Neighbour_Dev"],
"neighbour_device_port": item["Neighbour_dev_port"],
"neighbour_device_circle":item["Neighbour_Circle"],
"neighbour_device_subcircle":item["Neighbour_subcircle"],
"device_link_capacity":item["capacity_of_link(gig/sec)"],
"device_threshold%":item["Threshold(%)"],
"current_taffic_signal":(current_traffic_signal),
"current_traffic_rate %":((current_traffic_signal/int(item['capacity_of_link(gig/sec)']))*100)

}

}
series.append(json_body)
#jsondata= json.dumps(series)
counter=counter+1
print("Total record : ",counter)
URL="http://localhost:9999"
TOKEN=$TOKEN
ORG="analytics"
BUCKET="DeviceAnalysis"
client = InfluxDBClient(url=URL, token=TOKEN, org=ORG)
write_api = client.write_api(write_options=ASYNCHRONOUS)
datasave = write_api.write(BUCKET, ORG, series) #Writing to InfluxDB
print("DB Written : ",datasave)

[bednar]

Hi @ssaurav-1987,

If you want to use the ASYNCHRONOUS write You should wait to result of async request... something like this:

client = InfluxDBClient(url=URL, token=TOKEN, org=ORG)
write_api = client.write_api(write_options=ASYNCHRONOUS)
datasave = write_api.write(BUCKET, ORG, series) #Writing to Influx
datasave.get()
print("DB Written : ", series)

Please consider use a batching API if your file is huge. We have an example for that: import_data_set.py.

Regards

[ssaurav-1987]

Hi

Hi @ssaurav-1987,

If you want to use the ASYNCHRONOUS write You should wait to result of async request... something like this:

client = InfluxDBClient(url=URL, token=TOKEN, org=ORG)
write_api = client.write_api(write_options=ASYNCHRONOUS)
datasave = write_api.write(BUCKET, ORG, series) #Writing to Influx
datasave.get()
print("DB Written : ", series)

Please consider use a batching API if your file is huge. We have an example for that: import_data_set.py.

Regards

Hi @bednar Thank you for helping me here, but have some confusion.

1. In the above code for Asynchronous part you are mentioning datasave.get(). Will datasave.get() return something status code or any further code need to add.
2. For batch prcocess the code you shared in that function is returning Point, how to process the batch if have bulk data.
3. If I have to write the continious query how to go though for continious query. I searched some content from influxDB forum and tried to execute on influxDB query section but it is giving error
   :

[ssaurav-1987]

I tried above continious query in influx 1.8

[bednar]

1. database.get() waits to finish async request. For successful it returns None or propagate exception.
2. just return your json_body instead of Point -

   influxdb-client-python/examples/import_data_set.py

   Line 47 in 58f653a

   return Point("financial-analysis") \

3. I don't have an experience with continuous query 😞 - try to ask on slack

[ssaurav-1987]

1. database.get() waits to finish async request. For successful it returns None or propagate exception.
   Saurav : I got None before using get() function but it did not save into DB, 1.8 influx is working fine but for 2.0 I modified code and followed all the instruction from forum but it didnot work
2. just return your json_body instead of Point -

   influxdb-client-python/examples/import_data_set.py

   Line 47 in 58f653a

   return Point("financial-analysis") \

   
   Saurav : I will check this also.
3. I don't have an experience with continuous query 😞 - try to ask on slack
   Saurav : Ok no problem, I have to make my DB like it has to generate graph in every 4-5 min and through Panda or Matplotlyb I need to populate to UI side

[bednar]

Let's try write a small piece of your data into InfluxDB 2 by synchronous configuration and we will see where is a problem:

write_api = client.write_api(write_options=SYNCHRONOUS)

[ssaurav-1987]

SYNCHRONOUS

I tried with SYNCHRONOUS now and for datasave.get() function it is giving error:
Total record : 1
Total record : 2
Total record : 3
Total record : 4
Total record : 5
Total record : 6
Total record : 7
Total record : 8
Traceback (most recent call last):
File "device-series_influx2.py", line 58, in
datasave.get()
AttributeError: 'NoneType' object has no attribute 'get'

and After removing of datasave.get() from code and executed code and below output:
Total record : 1
Total record : 2
Total record : 3
Total record : 4
Total record : 5
Total record : 6
Total record : 7
Total record : 8
DB Written : None

[ssaurav-1987]

But no data is entered into DB

[bednar]

datasave.get() is useles for SYNCHRONOUS write...

Try initiliaze client with debug and you will see pure HTTP request/response

client = InfluxDBClient(url="http://localhost:9999", token="my-token", org="my-org", debug=True)

[ssaurav-1987]

datasave.get() is useles for SYNCHRONOUS write...

Try initiliaze client with debug and you will see pure HTTP request/response

client = InfluxDBClient(url="http://localhost:9999", token="my-token", org="my-org", debug=True)

I enabled Debug and got this in response:

Total record : 1
Total record : 2
Total record : 3
Total record : 4
Total record : 5
Total record : 6
Total record : 7
Total record : 8
send: b'POST /api/v2/write?org=analytics&bucket=DeviceAnalysis&precision=ns HTTP/1.1\r\nHost: localhost:9999\r\nAccept-Encoding: identity\r\nContent-Length: 4190\r\nContent-Encoding: identity\r\nContent-Type: text/plain\r\nAccept: application/json\r\nAuthorization: Token 8Rx2zupinSsQlRaIqyoxbXc6wWVsmaQY6VN7YndKVK2w8DP7h-TbZvJGx5YKuKxsJpXsqT1IAYylN6GUR7z4lA==\r\nUser-Agent: influxdb-client-python/1.6.0\r\n\r\n'
send: b'devicelink,device_circle=Bangaluru,device_subcircle=Mahadevpura,devicename=Device1 current_taffic_signal=2.020232163106752,current_traffic_rate\ %=20.20232163106752,device_circle="Bangaluru",device_ip_address="192.168.1.4",device_link_capacity=10i,device_name="Device1",device_port=80i,device_subcircle="Mahadevpura",device_threshold%=80i,neighbour_device_circle="Mumbai",neighbour_device_ip_address="192.168.11.21",neighbour_device_name="Device11",neighbour_device_port=80i,neighbour_device_subcircle="powai" 1588773931367803136\ndevicelink,device_circle=Bangaluru,device_subcircle=Whitefield,devicename=Device2 current_taffic_signal=6.890348776098387,current_traffic_rate\ %=68.90348776098388,device_circle="Bangaluru",device_ip_address="192.168.1.5",device_link_capacity=10i,device_name="Device2",device_port=80i,device_subcircle="Whitefield",device_threshold%=80i,neighbour_device_circle="Bangaluru",neighbour_device_ip_address="192.168.11.22",neighbour_device_name="Device12",neighbour_device_port=80i,neighbour_device_subcircle="Marthahalli" 1588773931367878912\ndevicelink,device_circle=Bangaluru,device_subcircle=Belanduru,devicename=Device3 current_taffic_signal=1.8429319661482182,current_traffic_rate\ %=18.429319661482182,device_circle="Bangaluru",device_ip_address="192.168.1.6",device_link_capacity=10i,device_name="Device3",device_port=80i,device_subcircle="Belanduru",device_threshold%=80i,neighbour_device_circle="Goa",neighbour_device_ip_address="192.168.11.23",neighbour_device_name="Device13",neighbour_device_port=80i,neighbour_device_subcircle="panji" 1588773931367918848\ndevicelink,device_circle=Mumbai,device_subcircle=Navi\ Mumbai,devicename=Device4 current_taffic_signal=2.2468334890415376,current_traffic_rate\ %=22.468334890415377,device_circle="Mumbai",device_ip_address="192.168.1.7",device_link_capacity=10i,device_name="Device4",device_port=80i,device_subcircle="Navi Mumbai",device_threshold%=80i,neighbour_device_circle="Delhi",neighbour_device_ip_address="192.168.11.24",neighbour_device_name="Device14",neighbour_device_port=80i,neighbour_device_subcircle="Sarojni" 1588773931367960064\ndevicelink,device_circle=Mumbai,device_subcircle=Powai,devicename=Device5 current_taffic_signal=2.0540103900820403,current_traffic_rate\ %=20.5401039008204,device_circle="Mumbai",device_ip_address="192.168.1.8",device_link_capacity=10i,device_name="Device5",device_port=80i,device_subcircle="Powai",device_threshold%=80i,neighbour_device_circle="Delhi",neighbour_device_ip_address="192.168.11.25",neighbour_device_name="Device15",neighbour_device_port=80i,neighbour_device_subcircle="Dhaula Kuan" 1588773931368002816\ndevicelink,device_circle=Mumbai,device_subcircle=Bandra,devicename=Device6 current_taffic_signal=4.907719614189714,current_traffic_rate\ %=49.07719614189715,device_circle="Mumbai",device_ip_address="192.168.1.9",device_link_capacity=10i,device_name="Device6",device_port=80i,device_subcircle="Bandra",device_threshold%=80i,neighbour_device_circle="Bangaluru",neighbour_device_ip_address="192.168.11.26",neighbour_device_name="Device16",neighbour_device_port=80i,neighbour_device_subcircle="whitefield" 1588773931368051968\ndevicelink,device_circle=Delhi,device_subcircle=Saket,devicename=Device7 current_taffic_signal=7.479615015902969,current_traffic_rate\ %=74.79615015902968,device_circle="Delhi",device_ip_address="192.168.1.10",device_link_capacity=10i,device_name="Device7",device_port=80i,device_subcircle="Saket",device_threshold%=80i,neighbour_device_circle="Bangaluru",neighbour_device_ip_address="192.168.11.27",neighbour_device_name="Device17",neighbour_device_port=80i,neighbour_device_subcircle="Belanduru" 1588773931368104960\ndevicelink,device_circle=Delhi,device_subcircle=Rohini,devicename=Device8 current_taffic_signal=2.591839580922648,current_traffic_rate\ %=25.91839580922648,device_circle="Delhi",device_ip_address="192.168.1.11",device_link_capacity=10i,device_name="Device8",device_port=80i,device_subcircle="Rohini",device_threshold%=80i,neighbour_device_circle="Bangaluru",neighbour_device_ip_address="192.168.11.28",neighbour_device_name="Device18",neighbour_device_port=80i,neighbour_device_subcircle="BTM" 1588773931368166912'
reply: 'HTTP/1.1 204 No Content\r\n'
header: Date: Wed, 06 May 2020 08:35:31 GMT
DB Written : None

[bednar]

I think that the problem is in your timestamp. It should be in UTC - https://v2.docs.influxdata.com/v2.0/reference/syntax/line-protocol/#timestamp

[ssaurav-1987]

I think that the problem is in your timestamp. It should be in UTC - https://v2.docs.influxdata.com/v2.0/reference/syntax/line-protocol/#timestamp

Okay, let me convert to UTC and will try to submit again

[ssaurav-1987]

I think that the problem is in your timestamp. It should be in UTC - https://v2.docs.influxdata.com/v2.0/reference/syntax/line-protocol/#timestamp

Okay, let me convert to UTC and will try to submit again

Hi @bednar
Thank you for the help, just because of time format it was not submitting into DB.
Data is inserting into DB but I am unable to query to generate graph. As in influx 1.8 it is quite easy to understand and generate graph well my requirement to generate graph is from Database I have to take the mean(of traffic rate%) based on grouping of device and time interval like in 1.8 influx
my query execute this way :
SELECT mean("current_traffic_rate %") AS "mean_current_traffic_rate %" FROM "deviceanalysis"."devanalysis"."devicelink" WHERE time > :dashboardTime: AND time < :upperDashboardTime: GROUP BY time(:interval:), "devicename" FILL(null)

but in 2.0 this is quite confusing how to build query

[bednar]

It looks like you mix field that is string and int... I am not expert in Flux queries - try look into doc - https://v2.docs.influxdata.com/v2.0/reference/flux/stdlib/built-in/transformations/group/ or try to ask on slack

[ssaurav-1987]

It looks like you mix field that is string and int... I am not expert in Flux queries - try look into doc - https://v2.docs.influxdata.com/v2.0/reference/flux/stdlib/built-in/transformations/group/ or try to ask on slack

Hi @bednar
It is working now I have done mistake in sequence of building query I mean arrangement.

It is working, I will have to lot of R&D in there. Also sharing some continuous query link, please have a look one you free, I am unable to get how to set this query.
https://influxdbcom.readthedocs.io/en/latest/content/docs/v0.9/query_language/continuous_queries/

https://www.youtube.com/watch?v=0syXa8cBY-Y

Thank you once again, means alot to get help here.