Bring NMODL transpiler under NEURON project

[pramodk]

• Script to trim and clean NMODL repository
• Test NMODL repository
• Merge nmodl trimed repo into NEURON - A Dry Run!
• "Marriage" of CMake - merging/linking NEURON and NMODL CMake
• Cleanup and remove unnecessary/duplicate files

Steps To Follow

Here are 5 major steps that we need to follow:

Clone NRN and NMODL Repos

#!/bin/bash

BASE_DIR=$(pwd)/nrn-nmodl-merge
mkdir -p ${BASE_DIR} && cd ${BASE_DIR}

git clone -b master --single-branch [email protected]:BlueBrain/nmodl.git
git clone -b master --single-branch [email protected]:neuronsimulator/nrn.git

Install git-filter-repo

pip install git-filter-repo

# TODO: change if necessary, `git-filter-repo` should be in $PATH
export PATH=$HOME/.local/bin/:$PATH

Prepare NMODL Repo

Use git filter-repo to do all hard work 👷

#!/bin/bash


cd ${BASE_DIR}/nmodl

# create a commit hook file that will be used by `git filter-repo`
cat <<EOF >> commit_hook.py

commit_message = commit.message.decode()
original_id = commit.original_id.decode()

# lets first remove trailing new lines
commit_message = commit_message.rstrip("\n")

# append now the original SHA in the commit message
commit_message = commit_message + "\n\nNMODL Repo SHA: BlueBrain/nmodl@" + original_id

# make links to PRs
matches = re.findall('#[0-9]+', commit_message, re.DOTALL)
if matches:
    for match in set(matches):
        # rename the PR IDs as they could be misleading / point to nrn PRs
        match.replace('#', '#nmodl-')
        new_prefix = "BlueBrain/nmodl" + match
        commit_message = commit_message.replace(match, new_prefix)

commit.message = commit_message.encode()
EOF

# first change commit messages and PR references
git filter-repo --force --preserve-commit-hashes --commit-callback commit_hook.py
rm commit_hook.py

# rename tags
git filter-repo --tag-rename '':'nmodl-'

# remove files that we don't need after merging in nrn
git filter-repo \
    --path .bbp-project.yaml \
    --path .clang-format.changes \
    --path .clang-tidy \
    --path .clang-tidy \
    --path .gitignore \
    --path .gitlab-ci.yml \
    --path .github \
    --path .gitmodules \
    --path .sanitizers \
    --path .cmake-format.changes.yaml \
    --path AUTHORS.txt \
    --path MANIFEST.in \
    --path azure-pipelines.yml \
    --path ci \
    --path ext \
    --path packaging \
    --path pyproject.toml \
    --path requirements.txt \
    --path setup.cfg \
    --path sonar-project.properties \
    --invert-paths

# now start renaming and remapping of directories and files 
git filter-repo --path-rename cmake/:cmake/nmodl/
git filter-repo --path-rename CMakeLists.txt:cmake/nmodl/CMakeLists.txt
git filter-repo --path-rename CONTRIBUTING.rst:docs/nmodl/transpiler/CONTRIBUTING.rst
git filter-repo --path-rename docs/:docs/nmodl/transpiler/
git filter-repo --path-rename INSTALL.rst:docs/nmodl/transpiler/INSTALL.rst
git filter-repo --path-rename LICENSE:src/nmodl/LICENSE
git filter-repo --path-rename python/:src/nmodl/python/
git filter-repo --path-rename README.rst:src/nmodl/README.rst
git filter-repo --path-rename share/nrnunits.lib:share/nmodl/nrnunits.lib
git filter-repo --path-rename src/:src/nmodl/
git filter-repo --path-rename test/:test/nmodl/transpiler/

We now have a "cleaned" version of NMODL repository. See if generated structure is what you want:

$ tree -L 3
.
├── cmake
│   └── nmodl
│       ├── CMakeLists.txt
│       ├── ClangTidyHelper.cmake
│       ├── CompilerHelper.cmake
│       ├── Config.cmake.in
│       ├── ExternalProjectHelper.cmake
│       ├── FlexHelper.cmake
│       ├── GitRevision.cmake
│       ├── PythonLinkHelper.cmake
│       ├── RpathHelper.cmake
│       └── hpc-coding-conventions
├── docs
│   └── nmodl
│       └── transpiler
├── share
│   └── nmodl
│       └── nrnunits.lib
├── src
│   └── nmodl
│       ├── CMakeLists.txt
│       ├── ast
│       ├── codegen
│       ├── config
│       ├── language
│       ├── lexer
│       ├── main.cpp
│       ├── nmodl
│       ├── parser
│       ├── printer
│       ├── pybind
│       ├── solver
│       ├── symtab
│       ├── units
│       ├── utils
│       └── visitors
└── test
    └── nmodl
        └── transpiler

We can now push this trimmed-down version of the repo into neuronsimulator org to verify if the commits and PRs are properly linked to original NMODL repo:

git remote add  neuronsimulator  [email protected]:neuronsimulator/nmodl-test.git
git push -f --all neuronsimulator

See the result at https://github.com/neuronsimulator/nmodl-test.

Merge NMODL Repo into NEURON

Now comes the part of merging the two repositories. Add "cleaned" nmodl repo as remote and merge the two repos as:

cd ${BASE_DIR}/nrn
git remote add nmodl file://${BASE_DIR}/nmodl
git fetch  nmodl --tags
git merge --allow-unrelated-histories nmodl/master

This should merge the repos cleanly. We can push this to nrn branch:

git push -f origin HEAD:pramodk/nmodl-merge

"Marriage" of CMake and Build System

We have now nmodl and nrn repo merged. But there are various things that needs to be happened:

• Add any submodules needed for NMODL
• Update CMakeLists.txt and connect nmodl's CMakeLists.txt
• Check requirements.txt and other build dependencies
• Update Spack recipe (?)
• Update CIs
• Remove unnecessary files from docs, cmake, tests, etc

[sonarcloud]

Quality Gate passed

Issues
993 New issues
0 Accepted issues

Measures
3 Security Hotspots
0.0% Coverage on New Code
9.7% Duplication on New Code

See analysis details on SonarQube Cloud

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 67.07%. Comparing base (5fe797a) to head (ab66e5e).

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #3265   +/-   ##
=======================================
  Coverage   67.07%   67.07%           
=======================================
  Files         571      571           
  Lines      111095   111095           
=======================================
  Hits        74521    74521           
  Misses      36574    36574           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[azure-pipelines]

✔️ ab66e5e -> Azure artifacts URL

[bbpbuildbot]

Logfiles from GitLab pipeline #239671 (:no_entry:) have been uploaded here!

Status and direct links:

• ✅ mac_m1_cmake_build: [cmake, ON, OFF, OFF, address]
• ✅ spack_setup
• ✅ build:nmodl
• ✅ build:neuron:nmodl:intel:legacy
• ✅ build:neuron:nmodl:intel:shared
• ✅ build:neuron:nmodl:nvhpc:acc:legacy
• ✅ build:neuron:nmodl:nvhpc:acc:shared
• ✅ build:neuron:nmodl:nvhpc:omp:legacy
• ✅ build:neuron:nmodl:nvhpc:omp
• ✅ test:neuron:nmodl:intel:legacy
• ✅ test:neuron:nmodl:intel:shared
• ✅ test:neuron:nmodl:nvhpc:acc:legacy
• ✅ test:neuron:nmodl:nvhpc:acc:shared
• ⛔ test:neuron:nmodl:nvhpc:omp:legacy
• ✅ test:neuron:nmodl:nvhpc:omp