[source-facebook-marketing] Difficult to debug connector logs?

[ThierryDeruyttere]

Topic

No response

Relevant information

Hi, I'm making some changes to the fb market place source api but I find it extremely difficult to debug any errors that are in the sources. All the airbyte stuff is ran in a separate process and hence my debugger is not catching it. So I'm wondering how people are debugging this without losing more time to finding where the problem is than actually resolving it. Thanks!

[marcosmarxm]

Can you provide... more information? What are you trying to do? Are you using the Airbyte Platform with the Facebook Marketing connector? Which steps you're not getting the feedback or error logs you need?

[ThierryDeruyttere]

Yea sure, sorry about the lack of info. We're using the airbyte python SDK.

For instance, we're trying to add hashes to the image stream. We're using the following code.

source = ab.get_source(
        "source-facebook-marketing",
        local_executable=local_executable_path,
        config={
            "account_ids": [ad_account_id],
            "access_token": access_token,
            "start_date": start_date,
            "hashes": hashes
        },
    )

    source.check()
    source.select_streams(["images"])

  cache_dir = Path("./.cache/default_cache")
  base_cache = airbyte.caches.DuckDBCache(
      db_path=cache_dir / "default_cache.duckdb",
      cache_dir=cache_dir,
  )
  result = source.read(cache=base_cache)

The error is thrown on the result = source.read(cache=base_cache) line.

This is the error:

Failed `source-facebook-marketing` read operation at 12:57:19.
Traceback (most recent call last):
  File "/Users/thierryderuyttere/miniconda3/envs/AdLean/lib/python3.11/site-packages/airbyte/sources/base.py", line 580, in _execute
    for line in self.executor.execute(args):
  File "/Users/thierryderuyttere/miniconda3/envs/AdLean/lib/python3.11/site-packages/airbyte/_executor.py", line 466, in execute
    with _stream_from_subprocess([str(self.path), *args]) as stream:
  File "/Users/thierryderuyttere/miniconda3/envs/AdLean/lib/python3.11/contextlib.py", line 144, in __exit__
    next(self.gen)
  File "/Users/thierryderuyttere/miniconda3/envs/AdLean/lib/python3.11/site-packages/airbyte/_executor.py", line 142, in _stream_from_subprocess
    raise exc.AirbyteSubprocessFailedError(
airbyte.exceptions.AirbyteSubprocessFailedError: AirbyteSubprocessFailedError: Subprocess failed.
    Run Args: ['integration_fb/source_facebook_marketing/executable', 'read', '--config', '/var/folders/4y/zntfmw7963l16186_wcld_v80000gn/T/tmppwisgxrp.json', '--catalog', '/var/folders/4y/zntfmw7963l16186_wcld_v80000gn/T/tmp9q6mjp_v.txt', '--state', '/var/folders/4y/zntfmw7963l16186_wcld_v80000gn/T/tmpvqwg3uc7.txt']
    Exit Code: 1

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/thierryderuyttere/miniconda3/envs/AdLean/lib/python3.11/site-packages/airbyte/sources/base.py", line 734, in read
    cache_processor.process_airbyte_messages(
  File "/Users/thierryderuyttere/miniconda3/envs/AdLean/lib/python3.11/site-packages/airbyte/_future_cdk/record_processor.py", line 186, in process_airbyte_messages
    for message in messages:
  File "/Users/thierryderuyttere/miniconda3/envs/AdLean/lib/python3.11/site-packages/airbyte/sources/base.py", line 548, in _read_with_catalog
    yield from self._tally_records(
  File "/Users/thierryderuyttere/miniconda3/envs/AdLean/lib/python3.11/site-packages/airbyte/sources/base.py", line 605, in _tally_records
    for message in messages:
  File "/Users/thierryderuyttere/miniconda3/envs/AdLean/lib/python3.11/site-packages/airbyte/sources/base.py", line 593, in _execute
    raise exc.AirbyteConnectorReadError(
airbyte.exceptions.AirbyteConnectorReadError: AirbyteConnectorReadError: Error when reading from the connector.

Log output: 
          File "/Users/thierryderuyttere/miniconda3/envs/AdLean/lib/python3.11/site-packages/airbyte_cdk/entrypoint.py", line 235, in launch

        for message in source_entrypoint.run(parsed_args):

      File "/Users/thierryderuyttere/miniconda3/envs/AdLean/lib/python3.11/site-packages/airbyte_cdk/entrypoint.py", line 122, in run

        yield from map(AirbyteEntrypoint.airbyte_message_to_string, self.read(source_spec, config, config_catalog, state))

      File "/Users/thierryderuyttere/miniconda3/envs/AdLean/lib/python3.11/site-packages/airbyte_cdk/entrypoint.py", line 166, in read

        for message in self.source.read(self.logger, config, catalog, state):

      File "/Users/thierryderuyttere/miniconda3/envs/AdLean/lib/python3.11/site-packages/airbyte_cdk/sources/abstract_source.py", line 184, in read

        raise AirbyteTracedException(message=error_message, failure_type=FailureType.config_error)

    airbyte_cdk.utils.traced_exception.AirbyteTracedException: None

    During the sync, the following streams did not sync successfully: images: AirbyteTracedException("string indices must be integers, not 'str'")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/thierryderuyttere/miniconda3/envs/AdLean/lib/python3.11/site-packages/airbyte/sources/base.py", line 745, in read
    raise exc.AirbyteConnectorFailedError(
airbyte.exceptions.AirbyteConnectorFailedError: AirbyteConnectorFailedError: Connector failed.

Log output: 
          File "/Users/thierryderuyttere/miniconda3/envs/AdLean/lib/python3.11/site-packages/airbyte_cdk/entrypoint.py", line 235, in launch

        for message in source_entrypoint.run(parsed_args):

      File "/Users/thierryderuyttere/miniconda3/envs/AdLean/lib/python3.11/site-packages/airbyte_cdk/entrypoint.py", line 122, in run

        yield from map(AirbyteEntrypoint.airbyte_message_to_string, self.read(source_spec, config, config_catalog, state))

      File "/Users/thierryderuyttere/miniconda3/envs/AdLean/lib/python3.11/site-packages/airbyte_cdk/entrypoint.py", line 166, in read

        for message in self.source.read(self.logger, config, catalog, state):

      File "/Users/thierryderuyttere/miniconda3/envs/AdLean/lib/python3.11/site-packages/airbyte_cdk/sources/abstract_source.py", line 184, in read

        raise AirbyteTracedException(message=error_message, failure_type=FailureType.config_error)

    airbyte_cdk.utils.traced_exception.AirbyteTracedException: None

    During the sync, the following streams did not sync successfully: images: AirbyteTracedException("string indices must be integers, not 'str'")

I know there's a problem with the hashes because when I remove them from the configuration, they work. But how does one go about debugging the code here? Because we've put breakpoints where we assign the hashes on the Image class, and it's not getting triggered. We think it's because Airbyte is using this local executable path to run the program.

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import re
import sys
from run import run
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(run())

And because this is spawned into a separate process, the debugger is not catching anything in the actual Facebook marketing code. So actually our question is, is there another way to use airbyte without spawning a separate process?

[marcosmarxm]

Transfer to PyAirbyte repo.