Lineage from Databricks "child" notebooks

[97nitt]

I have a Databricks notebook that, among other things, runs another notebook (via dbutils.notebook.run). I followed this and added the programmatic initialization of the Spline agent to the parent notebook. That works fine, but I noticed no lineage was produced for the stuff that is happening in the child notebook. I'm sure this is happening because, under the covers, that call to run the child notebook gets submitted as a separate job on the same cluster. So I tried adding the programmatic initialization to the child notebook, but that child job now fails with ModuleNotFoundError: No module named 'za'.

I'm sure this isn't an issue with the Spline agent, it's more weirdness with the way Databricks runs Notebook jobs. Just wondering if anyone in this community has experienced the same problem and has discovered any workarounds, aside from not using Notebooks :)

[wajda]

The error message means that a child job is created without Spline library in the class path. I've never used child notebooks, so all I'll say below is just my logical guess.
The Databricks doc says:

run command (dbutils.notebook.run)

Runs a notebook and returns its exit value. The notebook will run in the current cluster by default.

If I understand it correctly a child job is executed as a separate process, that apparently uses the same runtime setup as the parent process. In theory it should mean that child jobs should have the same set of libraries as its parent, but the fact is opposite.
The reason could be related to the way the Spline library is installed into the cluster. Somehow the libraries that are installed via the UI dialog are loaded at a later stage in runtime, and do not become a part of Spark cluster in a sense that when child processes are created from the same runtime setup (classpath) the custom libraries do not get loaded automatically. Try, instead of using a UI dialog install the Spark agent using the approach described in this post - #316 (reply in thread). If my deduction is correct it should fix the issue. If not, sorry :)

[97nitt]

👍 will give this a try... thanks.