regex is not able to select all the lines

[gopinath3759]

we are trying to send logs which is collected by fluent bit to splunk , in splunk we can see events triggers line by line . but we need to send all the lines as single message , so we used multiline with regex but it failing to select all the lines , please help us who to select all the logs as single message, we tired multiline paraser also .

regrex : Regex ^(?[0-9-]+ [0-9:,]+) - (?[^ ]+) - (?[^ ]+) - (?<json_payload>{.*})$

can you help us with right regex we need to use .

fluentbit configmap :

error: the server doesn't have a resource type "fb-agent-config"
[y000435a@cnpdcmldatagw01b ~]$ kubectl describe cm fb-agent-config -n runai-tai-agentai-ws
Name: fb-agent-config
Namespace: runai-tai-agentai-ws
Labels:
Annotations:

Data

fluent-bit.conf:

[SERVICE]
Flush 5
Log_Level debug
[INPUT]
Name tail
Path /home/agentai/proxyservice/*.log
Tag kube.runai-tai-agentai-ws.proxy
Multiline On
Parser multiline_json_logs
DB /var/log/fluentbit-offsets.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On

[INPUT]
Name tail
Path /home/agentai/ui/log/*.log
Tag kube.runai-tai-agentai-ws.ui
Multiline On
Parser multiline_json_logs
DB /var/log/fluentbit-offsets.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On

[FILTER]
Name kubernetes
Match kube.*
Kube_URL https://kubernetes.default.svc:443
Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
#Kube_Tag_Prefix kube.var.log.containers.
Merge_Log On
Merge_Log_Key log_processed
K8S-Logging.Parser On
K8S-Logging.Exclude On
[FILTER]
Name parser
Match kube.runai-tai-agentai-ws.proxy
Key_Name log
Parser custom_json_parser
Reserve_Data On
[FILTER]
Name parser
Match kube.runai-tai-agentai-ws.ui
Key_Name log
Parser custom_json_parser
Reserve_Data On

[FILTER]
Name modify
Match kube.runai-tai-agentai-ws.proxy
Add log-type application-log
Add namespace_name runai-tai-agentai-ws
Add cluster_name kube-stg
Add project-name machine-learning-platform
Add pod_name runai-tai-agentai-ws.proxy
[FILTER]
Name modify
Match kube.runai-tai-agentai-ws.ui
Add log-type application-log
Add namespace_name runai-tai-agentai-ws
Add cluster_name kube-stg
Add project-name machine-learning-platform
Add pod_name runai-tai-agentai-ws.ui

[OUTPUT]
Name http
Match kube.runai-tai-agentai-ws.proxy
Format json
Host elm-logs-dev.appl.kp.org
Port 31767
tls on
tls.verify off
header_tag machine-learning-platform
[OUTPUT]
Name http
Match kube.runai-tai-agentai-ws.ui
Format json
Host elm-logs-dev.appl.kp.org
Port 31767
tls on
tls.verify off
header_tag machine-learning-platform

parsers.conf:

[PARSER]
Name json
Format json
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z

[MULTILINE_PARSER]
Name multiline_json_logs
Type regex
Flush_Timeout 2
Rule "start_state" ^[0-9-]+ [0-9:,]+ - [^ ]+ - [^ ]+ - {$ "cont"
Rule "cont" ^\s+.* "cont"

[PARSER]
Name custom_json_parser
Format json
Time_Key timestamp
Time_Format %Y-%m-%d %H:%M:%S,%L

BinaryData

---

msg :

2024-12-05 07:32:45,905 - CustomLogger - INFO - {
"Model": "thriveai-llama",
"Messages": [
{
"role": "system",
"content": "You are a helpful assistant."
},
{
"role": "user",
"content": "\nGiven the following SQL query result which is in array format, summarize the data in a conversational, easy-to-understand format.\n \nPlease summarize the data in a friendly and engaging tone. The response should be short and crisp with the SQL query result that is provided.\nDo not include any other irrelavent messages or name before the output. Give the results as points.\nIf the query result is empty, give the results like example given below.\n\nExample 1:\nSQL query result:\n Related to Database or API downtime or An error occurred while fetching data.\n\nOutput:\nWe apologize for the inconvenience. It appears that the service is currently unavailable. Kindly try again after some time. Thank you for your understanding.\n\nExample 2:\nSQL query result:\n[['ORD1234', 'MRI Scan', 'Pending']['ORD0064', 'Blood Test', 'Pending']]\n \nOutput:\nSome messages related to the query result.\n\nMRI Scan and Blood test is pending.\n\nI have gone ahead and successfully booked the appointment for you. If you require any further assistance, please feel free to reach out.\n\nExample 3:\nSQL query result:\n[['ORD1234', 'MRI Scan', 'Completed']]\n \nOutput:\nSome messages related to the query result\n MRI Scan is Completed \n'Is something else i can help you with?', keep it as yes or no type of question.\n\nSQL Query Result:\n{result_str}\n[['Blood Test', 'lab', 'Pending']]"
}
],
"User": null,

}

2024-12-05 07:32:45,905 - CustomLogger - INFO - {
"Model": "thriveai-llama",
"Messages": [
{
"role": "system",
"content": "You are a helpful assistant."
},
{
"role": "user",
"content": "\nGiven the following SQL query result which is in array format, summarize the data in a conversational, easy-to-understand format.\n \nPlease summarize the data in a friendly and engaging tone. The response should be short and crisp with the SQL query result that is provided.\nDo not include any other irrelavent messages or name before the output. Give the results as points.\nIf the query result is empty, give the results like example given below.\n\nExample 1:\nSQL query result:\n Related to Database or API downtime or An error occurred while fetching data.\n\nOutput:\nWe apologize for the inconvenience. It appears that the service is currently unavailable. Kindly try again after some time. Thank you for your understanding.\n\nExample 2:\nSQL query result:\n[['ORD1234', 'MRI Scan', 'Pending']['ORD0064', 'Blood Test', 'Pending']]\n \nOutput:\nSome messages related to the query result.\n\nMRI Scan and Blood test is pending.\n\nI have gone ahead and successfully booked the appointment for you. If you require any further assistance, please feel free to reach out.\n\nExample 3:\nSQL query result:\n[['ORD1234', 'MRI Scan', 'Completed']]\n \nOutput:\nSome messages related to the query result\n MRI Scan is Completed \n'Is something else i can help you with?', keep it as yes or no type of question.\n\nSQL Query Result:\n{result_str}\n[['Blood Test', 'lab', 'Pending']]"
}
],
"User": null,

}

2024-12-05 07:32:45,905 - CustomLogger - INFO - {
"Model": "thriveai-llama",
"Messages": [
{
"role": "system",
"content": "You are a helpful assistant."
},
{
"role": "user",
"content": "\nGiven the following SQL query result which is in array format, summarize the data in a conversational, easy-to-understand format.\n \nPlease summarize the data in a friendly and engaging tone. The response should be short and crisp with the SQL query result that is provided.\nDo not include any other irrelavent messages or name before the output. Give the results as points.\nIf the query result is empty, give the results like example given below.\n\nExample 1:\nSQL query result:\n Related to Database or API downtime or An error occurred while fetching data.\n\nOutput:\nWe apologize for the inconvenience. It appears that the service is currently unavailable. Kindly try again after some time. Thank you for your understanding.\n\nExample 2:\nSQL query result:\n[['ORD1234', 'MRI Scan', 'Pending']['ORD0064', 'Blood Test', 'Pending']]\n \nOutput:\nSome messages related to the query result.\n\nMRI Scan and Blood test is pending.\n\nI have gone ahead and successfully booked the appointment for you. If you require any further assistance, please feel free to reach out.\n\nExample 3:\nSQL query result:\n[['ORD1234', 'MRI Scan', 'Completed']]\n \nOutput:\nSome messages related to the query result\n MRI Scan is Completed \n'Is something else i can help you with?', keep it as yes or no type of question.\n\nSQL Query Result:\n{result_str}\n[['Blood Test', 'lab', 'Pending']]"
}
],
"User": null,

}

2024-12-05 07:32:45,905 - CustomLogger - INFO - {
"Model": "thriveai-llama",
"Messages": [
{
"role": "system",
"content": "You are a helpful assistant."
},
{
"role": "user",
"content": "\nGiven the following SQL query result which is in array format, summarize the data in a conversational, easy-to-understand format.\n \nPlease summarize the data in a friendly and engaging tone. The response should be short and crisp with the SQL query result that is provided.\nDo not include any other irrelavent messages or name before the output. Give the results as points.\nIf the query result is empty, give the results like example given below.\n\nExample 1:\nSQL query result:\n Related to Database or API downtime or An error occurred while fetching data.\n\nOutput:\nWe apologize for the inconvenience. It appears that the service is currently unavailable. Kindly try again after some time. Thank you for your understanding.\n\nExample 2:\nSQL query result:\n[['ORD1234', 'MRI Scan', 'Pending']['ORD0064', 'Blood Test', 'Pending']]\n \nOutput:\nSome messages related to the query result.\n\nMRI Scan and Blood test is pending.\n\nI have gone ahead and successfully booked the appointment for you. If you require any further assistance, please feel free to reach out.\n\nExample 3:\nSQL query result:\n[['ORD1234', 'MRI Scan', 'Completed']]\n \nOutput:\nSome messages related to the query result\n MRI Scan is Completed \n'Is something else i can help you with?', keep it as yes or no type of question.\n\nSQL Query Result:\n{result_str}\n[['Blood Test', 'lab', 'Pending']]"
}
],
"User": null,

}