Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

span database #1608

Open
wants to merge 89 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 70 commits
Commits
Show all changes
89 commits
Select commit Hold shift + click to select a range
6e400c8
cleaning up old PR
sfc-gh-pmardziel Oct 17, 2024
a09c3a2
adding cost tracking to dummy endpoint
sfc-gh-pmardziel Oct 17, 2024
00e1705
finishing dummy and starting bedrock
sfc-gh-pmardziel Oct 18, 2024
0add8e0
finish up
sfc-gh-pmardziel Oct 18, 2024
4b748b0
replacement for track_all_costs_tally
sfc-gh-pmardziel Oct 18, 2024
9180fd0
docs
sfc-gh-pmardziel Oct 18, 2024
9ef5106
Merge remote-tracking branch 'origin/main' into piotrm/otel-cost-trac…
sfc-gh-pmardziel Oct 21, 2024
b8627c2
fixing up dummy api
sfc-gh-pmardziel Oct 22, 2024
1abb615
added huggingface cost tracking for otel
sfc-gh-pmardziel Oct 22, 2024
69b0e2a
working on awaitables
sfc-gh-pmardziel Oct 23, 2024
54ddad3
docs
sfc-gh-pmardziel Oct 23, 2024
5a36084
debugging async issue
sfc-gh-pmardziel Oct 24, 2024
4f4f624
work
sfc-gh-pmardziel Oct 24, 2024
ca3b5eb
async parallel fixes
sfc-gh-pmardziel Oct 24, 2024
4e88f2c
nit
sfc-gh-pmardziel Oct 24, 2024
62f8556
Merge remote-tracking branch 'origin/main' into piotrm/otel-cost-trac…
sfc-gh-pmardziel Oct 24, 2024
5686e3d
fix optional imports handling with fromlist
sfc-gh-pmardziel Oct 25, 2024
e491efe
remove print
sfc-gh-pmardziel Oct 25, 2024
a340a67
remove another print
sfc-gh-pmardziel Oct 25, 2024
2835286
doc format
sfc-gh-pmardziel Oct 25, 2024
1d6c2e0
put snowflake env var check back in test_endpoints
sfc-gh-pmardziel Oct 25, 2024
a8ec165
remove context.run
sfc-gh-pmardziel Oct 25, 2024
66dd6b7
make feature private
sfc-gh-pmardziel Oct 25, 2024
67d9cb6
making more things private
sfc-gh-pmardziel Oct 25, 2024
e703b6c
Merge remote-tracking branch 'origin/main' into piotrm/otel-cost-trac…
sfc-gh-pmardziel Oct 25, 2024
5fcc28c
Merge remote-tracking branch 'origin/main' into piotrm/otel-cost-trac…
sfc-gh-pmardziel Oct 25, 2024
e45fa5b
starting work on collector
sfc-gh-pmardziel Oct 26, 2024
d6ba709
working on collector
sfc-gh-pmardziel Oct 29, 2024
cb95a2f
Merge remote-tracking branch 'origin/main' into piotrm/otel_collector
sfc-gh-pmardziel Oct 29, 2024
166d9ec
typeargs
sfc-gh-pmardziel Oct 29, 2024
24f31e4
add type args
sfc-gh-pmardziel Oct 29, 2024
356bb2b
nits
sfc-gh-pmardziel Oct 29, 2024
0c3f1a0
span schema work
sfc-gh-pmardziel Oct 29, 2024
9a137c9
Merge remote-tracking branch 'origin/piotrm/otel-cost-tracking' into …
sfc-gh-pmardziel Oct 29, 2024
d875e68
db work
sfc-gh-pmardziel Oct 29, 2024
83d3215
Merge remote-tracking branch 'origin/piotrm/otel_collector' into piot…
sfc-gh-pmardziel Oct 29, 2024
ab0b16a
moved semconv to a different package, added span to/of orm
sfc-gh-pmardziel Oct 30, 2024
aa14ff8
nits
sfc-gh-pmardziel Oct 30, 2024
a487243
work
sfc-gh-pmardziel Oct 31, 2024
3ab16ae
nits
sfc-gh-pmardziel Oct 31, 2024
061091b
typed spans work
sfc-gh-pmardziel Nov 2, 2024
8537d57
nits
sfc-gh-pmardziel Nov 4, 2024
f34f8eb
Merge remote-tracking branch 'origin/main' into piotrm/otel_db
sfc-gh-pmardziel Nov 7, 2024
bc23ef5
Remaining files.
sfc-gh-pmardziel Nov 7, 2024
9a3f6c3
db work
sfc-gh-pmardziel Nov 9, 2024
a6d4573
get trace_record
sfc-gh-pmardziel Nov 10, 2024
8d78d04
fixup
sfc-gh-pmardziel Nov 11, 2024
a19216e
finishing up
sfc-gh-pmardziel Nov 13, 2024
7af1f2e
Merge remote-tracking branch 'origin/main' into piotrm/otel_db
sfc-gh-pmardziel Nov 13, 2024
5368e6b
nits
sfc-gh-pmardziel Nov 13, 2024
49de80a
Merge remote-tracking branch 'origin/main' into piotrm/otel_db
sfc-gh-pmardziel Nov 14, 2024
1233d66
doc nits
sfc-gh-pmardziel Nov 14, 2024
53338b4
remove accidental commit
sfc-gh-pmardziel Nov 14, 2024
f38efc3
docs nits
sfc-gh-pmardziel Nov 14, 2024
fe2f239
addressing PR comments
sfc-gh-pmardziel Nov 14, 2024
8ba5e37
print -> warning
sfc-gh-pmardziel Nov 14, 2024
363ee57
doc nits
sfc-gh-pmardziel Nov 14, 2024
9f46ed0
nits
sfc-gh-pmardziel Nov 14, 2024
54422ab
fixes
sfc-gh-pmardziel Nov 15, 2024
fd59fc9
docs additions
sfc-gh-pmardziel Nov 15, 2024
c8a296a
move to subfolder
sfc-gh-pmardziel Nov 15, 2024
1100ea6
Merge remote-tracking branch 'origin/main' into piotrm/otel_db
sfc-gh-pmardziel Nov 15, 2024
e2ce342
Fix TypeAlias import .
sfc-gh-pmardziel Nov 16, 2024
43e58c2
trying to get gitignore to work on the lock files
sfc-gh-pmardziel Nov 16, 2024
2dec4ee
I don't know what is going on
sfc-gh-pmardziel Nov 16, 2024
db57921
fix typealias issues
sfc-gh-pmardziel Nov 16, 2024
41b4bf8
remove another typealias
sfc-gh-pmardziel Nov 16, 2024
d911747
py3.8 fixes
sfc-gh-pmardziel Nov 16, 2024
672e8a4
another one
sfc-gh-pmardziel Nov 16, 2024
a7284db
write api
sfc-gh-pmardziel Nov 16, 2024
83d85eb
use compat type
sfc-gh-pmardziel Nov 16, 2024
5139874
alias weakset
sfc-gh-pmardziel Nov 16, 2024
f4bd770
docs
sfc-gh-pmardziel Nov 16, 2024
e8773b0
working on python 3.8 issues
sfc-gh-pmardziel Nov 16, 2024
92fb1ce
fix py38 issue
sfc-gh-pmardziel Nov 17, 2024
64fa350
remove debug cell
sfc-gh-pmardziel Nov 17, 2024
406059e
nits
sfc-gh-pmardziel Nov 17, 2024
5d809a1
Merge remote-tracking branch 'origin/main' into piotrm/otel_db
sfc-gh-dkurokawa Nov 22, 2024
30f27ae
Submit working `poetry.lock` file.
sfc-gh-dkurokawa Nov 22, 2024
bf796f8
Fix snowflake tests.
sfc-gh-dkurokawa Nov 22, 2024
93aea9f
Merge remote-tracking branch 'origin/main' into piotrm/otel_db
sfc-gh-dkurokawa Nov 23, 2024
0fe136e
Merge remote-tracking branch 'origin/main' into piotrm/otel_db
sfc-gh-dkurokawa Nov 25, 2024
5f449c9
Fix snowflake tests.
sfc-gh-dkurokawa Nov 25, 2024
2dd68ec
Update version.
sfc-gh-dkurokawa Nov 25, 2024
7f0abbc
Fix up some small issues.
sfc-gh-dkurokawa Nov 25, 2024
f2c72bb
Remove unnecessary `poetry.lock` files.
sfc-gh-dkurokawa Nov 25, 2024
4e5475c
Merge remote-tracking branch 'origin/main' into piotrm/otel_db
sfc-gh-dkurokawa Dec 3, 2024
812dfc2
Merge remote-tracking branch 'origin/main' into piotrm/otel_db
sfc-gh-dkurokawa Dec 5, 2024
5256462
Merge remote-tracking branch 'origin/main' into piotrm/otel_db
sfc-gh-dkurokawa Dec 5, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -70,3 +70,16 @@ data/
# Build objects
**/poetry.lock
!poetry.lock
src/apps/langchain/poetry.lock
src/apps/llamaindex/poetry.lock
src/apps/nemo/poetry.lock
src/connectors/snowflake/poetry.lock
src/core/poetry.lock
src/dashboard/poetry.lock
src/providers/bedrock/poetry.lock
src/providers/cortex/poetry.lock
src/providers/huggingface/poetry.lock
src/providers/langchain/poetry.lock
src/providers/litellm/poetry.lock
src/providers/openai/poetry.lock
src/trulens_eval/poetry.lock
106 changes: 106 additions & 0 deletions examples/experimental/collector_example.ipynb
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"! pip install opentelemetry-exporter-otlp\n",
"! pip install \"fastapi[standard]\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# TruLens OTLP Collector Usage"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import json\n",
"\n",
"from opentelemetry import trace\n",
"from opentelemetry.exporter.otlp.proto.http.trace_exporter import (\n",
" OTLPSpanExporter,\n",
")\n",
"from opentelemetry.propagate import inject\n",
"from opentelemetry.sdk.trace import TracerProvider\n",
"from opentelemetry.sdk.trace.export import BatchSpanProcessor\n",
"import requests\n",
"from trulens.experimental.otel_tracing.core.otel import Collector\n",
"\n",
"tracer = trace.get_tracer(__name__)\n",
"\n",
"tracer_provider = TracerProvider()\n",
"trace.set_tracer_provider(tracer_provider)\n",
"span_processor = BatchSpanProcessor(\n",
" OTLPSpanExporter(endpoint=\"http://localhost:5000\")\n",
")\n",
"tracer_provider.add_span_processor(span_processor)\n",
"\n",
"with tracer.start_as_current_span(\"request\") as span:\n",
" headers = {}\n",
" inject(headers)\n",
"\n",
"print(headers)\n",
"\n",
"c = Collector()\n",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explain what's going on in this cell? Like how do we know the exporter is pointing at this collector?

"c.start()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"res = requests.post(\"http://localhost:5000/\", json=json.dumps({\"key\": \"value\"}))\n",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This cell fails for me with the following error:

{
	"name": "ConnectionError",
	"message": "HTTPConnectionPool(host='localhost', port=5000): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x30f03be50>: Failed to establish a new connection: [Errno 61] Connection refused'))",
	"stack": "---------------------------------------------------------------------------
ConnectionRefusedError                    Traceback (most recent call last)
File ~/miniconda3/envs/trulens_3_11/lib/python3.11/site-packages/urllib3/connection.py:174, in HTTPConnection._new_conn(self)
    173 try:
--> 174     conn = connection.create_connection(
    175         (self._dns_host, self.port), self.timeout, **extra_kw
    176     )
    178 except SocketTimeout:

File ~/miniconda3/envs/trulens_3_11/lib/python3.11/site-packages/urllib3/util/connection.py:95, in create_connection(address, timeout, source_address, socket_options)
     94 if err is not None:
---> 95     raise err
     97 raise socket.error(\"getaddrinfo returns an empty list\")

File ~/miniconda3/envs/trulens_3_11/lib/python3.11/site-packages/urllib3/util/connection.py:85, in create_connection(address, timeout, source_address, socket_options)
     84     sock.bind(source_address)
---> 85 sock.connect(sa)
     86 return sock

ConnectionRefusedError: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

NewConnectionError                        Traceback (most recent call last)
File ~/miniconda3/envs/trulens_3_11/lib/python3.11/site-packages/urllib3/connectionpool.py:716, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    715 # Make the request on the httplib connection object.
--> 716 httplib_response = self._make_request(
    717     conn,
    718     method,
    719     url,
    720     timeout=timeout_obj,
    721     body=body,
    722     headers=headers,
    723     chunked=chunked,
    724 )
    726 # If we're going to release the connection in ``finally:``, then
    727 # the response doesn't need to know about the connection. Otherwise
    728 # it will also try to release it and we'll have a double-release
    729 # mess.

File ~/miniconda3/envs/trulens_3_11/lib/python3.11/site-packages/urllib3/connectionpool.py:416, in HTTPConnectionPool._make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)
    415     else:
--> 416         conn.request(method, url, **httplib_request_kw)
    418 # We are swallowing BrokenPipeError (errno.EPIPE) since the server is
    419 # legitimately able to close the connection after sending a valid response.
    420 # With this behaviour, the received response is still readable.

File ~/miniconda3/envs/trulens_3_11/lib/python3.11/site-packages/urllib3/connection.py:244, in HTTPConnection.request(self, method, url, body, headers)
    243     headers[\"User-Agent\"] = _get_default_user_agent()
--> 244 super(HTTPConnection, self).request(method, url, body=body, headers=headers)

File ~/miniconda3/envs/trulens_3_11/lib/python3.11/http/client.py:1303, in HTTPConnection.request(self, method, url, body, headers, encode_chunked)
   1302 \"\"\"Send a complete request to the server.\"\"\"
-> 1303 self._send_request(method, url, body, headers, encode_chunked)

File ~/miniconda3/envs/trulens_3_11/lib/python3.11/http/client.py:1349, in HTTPConnection._send_request(self, method, url, body, headers, encode_chunked)
   1348     body = _encode(body, 'body')
-> 1349 self.endheaders(body, encode_chunked=encode_chunked)

File ~/miniconda3/envs/trulens_3_11/lib/python3.11/http/client.py:1298, in HTTPConnection.endheaders(self, message_body, encode_chunked)
   1297     raise CannotSendHeader()
-> 1298 self._send_output(message_body, encode_chunked=encode_chunked)

File ~/miniconda3/envs/trulens_3_11/lib/python3.11/http/client.py:1058, in HTTPConnection._send_output(self, message_body, encode_chunked)
   1057 del self._buffer[:]
-> 1058 self.send(msg)
   1060 if message_body is not None:
   1061 
   1062     # create a consistent interface to message_body

File ~/miniconda3/envs/trulens_3_11/lib/python3.11/http/client.py:996, in HTTPConnection.send(self, data)
    995 if self.auto_open:
--> 996     self.connect()
    997 else:

File ~/miniconda3/envs/trulens_3_11/lib/python3.11/site-packages/urllib3/connection.py:205, in HTTPConnection.connect(self)
    204 def connect(self):
--> 205     conn = self._new_conn()
    206     self._prepare_conn(conn)

File ~/miniconda3/envs/trulens_3_11/lib/python3.11/site-packages/urllib3/connection.py:186, in HTTPConnection._new_conn(self)
    185 except SocketError as e:
--> 186     raise NewConnectionError(
    187         self, \"Failed to establish a new connection: %s\" % e
    188     )
    190 return conn

NewConnectionError: <urllib3.connection.HTTPConnection object at 0x30f03be50>: Failed to establish a new connection: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

MaxRetryError                             Traceback (most recent call last)
File ~/miniconda3/envs/trulens_3_11/lib/python3.11/site-packages/requests/adapters.py:667, in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
    666 try:
--> 667     resp = conn.urlopen(
    668         method=request.method,
    669         url=url,
    670         body=request.body,
    671         headers=request.headers,
    672         redirect=False,
    673         assert_same_host=False,
    674         preload_content=False,
    675         decode_content=False,
    676         retries=self.max_retries,
    677         timeout=timeout,
    678         chunked=chunked,
    679     )
    681 except (ProtocolError, OSError) as err:

File ~/miniconda3/envs/trulens_3_11/lib/python3.11/site-packages/urllib3/connectionpool.py:802, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    800     e = ProtocolError(\"Connection aborted.\", e)
--> 802 retries = retries.increment(
    803     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
    804 )
    805 retries.sleep()

File ~/miniconda3/envs/trulens_3_11/lib/python3.11/site-packages/urllib3/util/retry.py:594, in Retry.increment(self, method, url, response, error, _pool, _stacktrace)
    593 if new_retry.is_exhausted():
--> 594     raise MaxRetryError(_pool, url, error or ResponseError(cause))
    596 log.debug(\"Incremented Retry for (url='%s'): %r\", url, new_retry)

MaxRetryError: HTTPConnectionPool(host='localhost', port=5000): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x30f03be50>: Failed to establish a new connection: [Errno 61] Connection refused'))

During handling of the above exception, another exception occurred:

ConnectionError                           Traceback (most recent call last)
Cell In[3], line 1
----> 1 res = requests.post(\"http://localhost:5000/\", json=json.dumps({\"key\": \"value\"}))
      2 res.content

File ~/miniconda3/envs/trulens_3_11/lib/python3.11/site-packages/requests/api.py:115, in post(url, data, json, **kwargs)
    103 def post(url, data=None, json=None, **kwargs):
    104     r\"\"\"Sends a POST request.
    105 
    106     :param url: URL for the new :class:`Request` object.
   (...)
    112     :rtype: requests.Response
    113     \"\"\"
--> 115     return request(\"post\", url, data=data, json=json, **kwargs)

File ~/miniconda3/envs/trulens_3_11/lib/python3.11/site-packages/requests/api.py:59, in request(method, url, **kwargs)
     55 # By using the 'with' statement we are sure the session is closed, thus we
     56 # avoid leaving sockets open which can trigger a ResourceWarning in some
     57 # cases, and look like a memory leak in others.
     58 with sessions.Session() as session:
---> 59     return session.request(method=method, url=url, **kwargs)

File ~/miniconda3/envs/trulens_3_11/lib/python3.11/site-packages/requests/sessions.py:589, in Session.request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
    584 send_kwargs = {
    585     \"timeout\": timeout,
    586     \"allow_redirects\": allow_redirects,
    587 }
    588 send_kwargs.update(settings)
--> 589 resp = self.send(prep, **send_kwargs)
    591 return resp

File ~/miniconda3/envs/trulens_3_11/lib/python3.11/site-packages/requests/sessions.py:703, in Session.send(self, request, **kwargs)
    700 start = preferred_clock()
    702 # Send the request
--> 703 r = adapter.send(request, **kwargs)
    705 # Total elapsed time of the request (approximately)
    706 elapsed = preferred_clock() - start

File ~/miniconda3/envs/trulens_3_11/lib/python3.11/site-packages/requests/adapters.py:700, in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
    696     if isinstance(e.reason, _SSLError):
    697         # This branch is for urllib3 v1.22 and later.
    698         raise SSLError(e, request=request)
--> 700     raise ConnectionError(e, request=request)
    702 except ClosedPoolError as e:
    703     raise ConnectionError(e, request=request)

ConnectionError: HTTPConnectionPool(host='localhost', port=5000): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x30f03be50>: Failed to establish a new connection: [Errno 61] Connection refused'))"
}

"res.content"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from opentelemetry.exporter import otlp"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"dir(otlp.proto.grpc)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "trulens-9bG3yHQd-py3.11",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
2 changes: 1 addition & 1 deletion examples/experimental/dummy_example.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -418,7 +418,7 @@
],
"metadata": {
"kernelspec": {
"display_name": "trulens-9bG3yHQd-py3.9",
"display_name": "trulens-9bG3yHQd-py3.11",
"language": "python",
"name": "python3"
},
Expand Down
Loading
Loading