✨ Xircuits Entry Handler for Components + Submodule Components Download

adr/0002-Enable Data Passing via Context.md

@@ -0,0 +1,40 @@
+# Enable Data Passing Via Context
+
+## Status
+Accepted
+
+Proposed by: Eduardo Gonzalez (17/01/2022)
+
+Discussed with: the team
+
+## Context
+We allow data to be passed between components via Context (ctx). Previously, the only way to pass data between components is by connecting the outPort to inPort. While visually intuitive, variables that are constantly used in most if not all components (such as SparkSession in Spark xircuits)  must be linked again and again.
+
+## Decision
+- We add `ctx` to the base component `execute()` method. 
+
+- A component that would like to pass data to the context can be treated as a dict, and would be implemented as the following:
+```python
+from xai_components.base import Component, xai_component
+
+@xai_component()
+class HelloContext(Component):
+
+    def execute(self, ctx):
+        context_dict = {"new ctx": "Hello Xircuits!"}
+        ctx.update(context_dict)
+```
+- In codegen, by default `ctx` will contain the xircuits args and hyperparameters.
+```python
+# python example.py --epoch=554
+{'args': Namespace(epoch=554, experiment_name='2022-04-22 16:13:40')}
+```
+
+## Consequences
+
+### Advantages
+* A convenient way to pass data between components.
+* Provide an alternative to access hyperparameters.
+
+### Disadvantages
+* The need to add `ctx` to each component.

adr/0003-Refactor Component Libraries as Submodules.md

@@ -0,0 +1,37 @@
+# Refactor Component Libraries as Submodules
+
+## Status
+Accepted
+
+Proposed by: Fahreza (12/04/2022)
+
+Discussed with: Eduardo Gonzalez
+
+## Context
+
+As the repository matures, the available Xircuits component libraries will only grow and with it the number of packages required to install the `full` version as well as the examples for each component library. Adding or modifying a component library also requires a PR to update. 
+
+Implementing component libraries as submodules would allow users to easy create their own component library repositories and port them.
+
+## Decision
+
+
+Moving forward, there'll be a clearer description on how xai component libraries are organized.
+
+1. The default [full] installation of Xircuits will only include deep learning related libraries (Tensorflow, Pytorch, Onnx) as well as PySpark components.
+2. XAI component libraries that uses special frameworks like Pycaret or Selenium will be added as submodules. 
+3. The examples of those component libraries are moved from the `examples` folder to their respective component library directories.
+
+This has been implemented in https://github.com/XpressAI/xircuits/pull/148
+
+## Consequences
+
+### Advantages
+
+* Improves scalability and reduce maintainability costs.
+* Component library authors to have more control over their own component libraries.
+* Improves the housekeeping of the examples.
+* Users can modify the submodules and update if needed.
+
+### Disadvantages
+* Require an extra step to fetch component submodules.

adr/0004-Define Component Libraries and Templates Implementation.md

@@ -0,0 +1,70 @@
+# Define Component Libraries and Templates Implementation
+
+## Status
+Accepted
+
+Proposed by: Fahreza (13/04/2022)
+
+Discussed with: Eduardo Gonzalez
+
+## Context
+
+In [ADR-0003](./0003-Refactor%20Component%20Libraries%20as%20Submodules.md), we have defined the core component libraries and refactored the rest into submodules. In this ADR, we further define what is a submodule component library, what is a template, how to structure the repository and the workflow with Xircuits.
+
+
+## Decision
+
+### Component Library:
+* A Xircuits component library is a repository or directory that contains xircuits component codes render and run in a Xircuits canvas.
+* A component library should be framework focused (tensorflow, pyspark, pycaret, etc).
+* A component library repository should focus on one framework at a time and keep other library / frameworks imports to a minimum.
+* The structure should contain `__init__.py`, the component_code.py, the requirements.txt, and optionally an `examples` directory, as below: 
+```
+# +-- examples
+# |   |
+# |   +-- example1.xircuits
+# |   +-- example2.xircuits
+# |
+# +-- __init__.py
+# +-- component_code.py
+# +-- requirements.txt
+```
+* In Xircuits they are implemented as xai_component submodules.
+* The workflow of using a component library would be simply copying / cloning the directory to your working Xircuits directory.
+
+* Example: a pycaret component library
+
+### Template:
+* A Xircuits template is a project repository that utilizes Xircuits as its engine.
+* A single project should have only one Xircuits template at a time.
+* A template should be application focused (object detection, BERT training, etc).
+* A template repository can have as much component libraries as the application needs.
+* The template repository should be structured so that Xircuits can be launched directly. Hence it will need the `.xircuits` config directory, `xai_components` with the `base.py` and whatever component libraries needed, as well as the xircuits to run the application template, as shown below: 
+```
+# working directory
+# |
+# +-- .xircuits
+# |   +-- config.ini
+# |
+# +-- xai_components
+# |   +-- xai_lib_1
+# |   |   +-- __init__.py
+# |   |   +-- component_code.py
+# |   |   +-- requirements.txt
+# |   |
+# |   +-- __init__.py
+# |   +-- base.py
+# |
+# +-- ApplicationTemplate.xircuits
+# +-- requirements.txt
+```
+* The workflow of using a template would be to clone the template repository then installing requirements.txt with a specified xircuits version inside.
+
+## Consequences
+
+### Advantages
+* A clear distinction between a xircuits component library and xircuits template.
+* Xircuits templates should always work out of the box as the xircuits and component definitions are frozen in requirements.txt.
+
+### Disadvantages
+* Xircuits template repositories might not always have the most latest features.

requirements.txt

@@ -10,5 +10,6 @@ jinja2>=3.0.3
 y-py>=0.3.0,<0.4.0
 jupyterlab>=3.0.0,<4.0.0
 requests
+gitpython
 pygithub
 tqdm

setup.py

@@ -89,7 +89,8 @@
     entry_points ={
         'console_scripts': [
             'xircuits = xircuits.start_xircuits:main',
-            'xircuits-examples = xircuits.start_xircuits:download_examples'
+            'xircuits-examples = xircuits.start_xircuits:download_examples',
+            'xircuits-components = xircuits.start_xircuits:download_component_library'
             ]}
 )
 

xircuits/handlers/request_folder.py

@@ -28,7 +28,7 @@ def request_folder(folder, repo_name="XpressAi/Xircuits", branch="master"):
         if file_content.type=='dir':
             if not os.path.exists(file_content.path):
                 os.mkdir(file_content.path)
-            contents.extend(repo.get_contents(file_content.path))
+            contents.extend(repo.get_contents(file_content.path, ref=branch))
 
         else:
             file_url = base_url + "/" + parse.quote(file_content.path)

xircuits/handlers/request_submodule.py

@@ -0,0 +1,47 @@
+from urllib import request
+from git import Repo
+from git.remote import RemoteProgress
+
+class Progress(RemoteProgress):
+    def update(self, *args):
+        print(self._cur_line, end='\r', flush=True)
+
+def get_submodule_config(user_query):
+
+    import configparser
+    config = configparser.ConfigParser()
+
+    config.read('.xircuits/.gitmodules')
+
+    submodule_keys = [submodule for submodule in config.sections() if user_query in submodule]
+    if len(submodule_keys) == 0:
+        print(user_query + " component library submodule not found.")
+        return
+
+    if len(submodule_keys) > 1:
+        print("Multiple '" + user_query + "' found. Returning first instance.")
+
+    submodule_key = submodule_keys.pop(0)
+
+    submodule_path = config[submodule_key]["path"]
+    submodule_url = config[submodule_key]["url"]
+
+    return submodule_path, submodule_url
+
+
+def request_submodule_library(component_library_query):
+
+    module_url = "https://raw.githubusercontent.com/XpressAI/xircuits/master/.gitmodules"
+    request.urlretrieve(module_url, ".xircuits/.gitmodules")
+
+    # ensure syntax is as xai_components/xai_library_name
+    if "xai" not in component_library_query:
+        component_library_query = "xai_" + component_library_query
+
+    if "xai_components" not in component_library_query:
+        component_library_query = "xai_components/" + component_library_query
+
+    submodule_path, submodule_url = get_submodule_config(component_library_query)
+
+    print("Cloning " + submodule_path + " from " + submodule_url)
+    Repo.clone_from(submodule_url, submodule_path, progress=Progress())

xircuits/start_xircuits.py

@@ -7,57 +7,71 @@
 import argparse
 
 from .handlers.request_folder import request_folder
+from .handlers.request_submodule import request_submodule_library
 
 def init_xircuits():
+
     url = "https://raw.githubusercontent.com/XpressAI/xircuits/master/.xircuits/config.ini"
     path = ".xircuits"
     os.mkdir(path)
     request.urlretrieve(url, path+"/config.ini")
 
-def start_xircuits(branch_name):
-    print(
-'''
-======================================
-__   __  ___                _ _       
-\ \  \ \/ (_)_ __ ___ _   _(_) |_ ___ 
- \ \  \  /| | '__/ __| | | | | __/ __|
- / /  /  \| | | | (__| |_| | | |_\__ \\
-/_/  /_/\_\_|_|  \___|\__,_|_|\__|___/
-
-======================================
-'''
-    )
 
-    config_path = Path(os.getcwd()) / ".xircuits"
-    component_library_path = Path(os.getcwd()) / "xai_components"
+def download_examples():
+
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--branch', nargs='?', default="master", help='pull files from a xircuits branch')
 
-    if not config_path.exists():
-        init_xircuits()
+    args = parser.parse_args()
 
-    if not component_library_path.exists():
-        val = input("Xircuits Component Library is not found. Would you like to load it in the current path (Y/N)? ")
-        if val.lower() == ("y" or "yes"):
-            request_folder("xai_components", branch=branch_name)
-
-    os.system("jupyter lab")
+    request_folder("examples", branch=args.branch)
+    request_folder("datasets", branch=args.branch)
 
 
-def download_examples():
+def download_component_library():
+
     parser = argparse.ArgumentParser()
-    parser.add_argument('--branch', nargs='?', help='pull files from a xircuits branch')
+    parser.add_argument('--branch', nargs='?', default="master", help='pull files from a xircuits branch')
+    parser.add_argument('--sublib', nargs='*', help='pull component library from a xircuits submodule')
 
     args = parser.parse_args()
-    branch_name = args.branch if args.branch else "master"
+    if not args.sublib:
+        request_folder("xai_components", branch=args.branch)
+    else:
+        for component_lib in args.sublib:
+            request_submodule_library(component_lib)
 
-    request_folder("examples", branch=branch_name)
-    request_folder("datasets", branch=branch_name)
-
 
-def main(argv=None):
+def main():
+
     parser = argparse.ArgumentParser()
-    parser.add_argument('--branch', nargs='?', help='pull files from a xircuits branch')
+    parser.add_argument('--branch', nargs='?', default="master", help='pull files from a xircuits branch')
 
     args = parser.parse_args()
-    branch_name = args.branch if args.branch else "master"
+    branch_name = args.branch
+
+    component_library_path = Path(os.getcwd()) / "xai_components"
+
+    if not component_library_path.exists():
+        val = input("Xircuits Component Library is not found. Would you like to load it in the current path (Y/N)? ")
+        if val.lower() == ("y" or "yes"):
+            request_folder("xai_components", branch=branch_name)
+
+    os.system("jupyter lab")
+
+print(
+'''
+======================================
+__   __  ___                _ _       
+\ \  \ \/ (_)_ __ ___ _   _(_) |_ ___ 
+ \ \  \  /| | '__/ __| | | | | __/ __|
+ / /  /  \| | | | (__| |_| | | |_\__ \\
+/_/  /_/\_\_|_|  \___|\__,_|_|\__|___/
+
+======================================
+'''
+)
 
-    start_xircuits(branch_name)
+config_path = Path(os.getcwd()) / ".xircuits"
+if not config_path.exists():
+        init_xircuits()