Create a pure Python SWC -> NeuroML converter

[pgleeson]

Is your feature request related to a problem? Please describe.
There should be a pure Python way to import an SWC file and convert it to the latest valid NeuroML 2 format

Describe the solution you'd like
A file in here: https://github.com/NeuroML/pyNeuroML/tree/master/pyneuroml/swc, which can be used in scripts, or potentially at the command line, to take an SWC file and create a valid NML file. Some options could be included for handing ambiguities, e.g. when the soma is absent, e.g. see here

Describe alternatives you've considered
The current recommended way to get SWC to NeuroML is to use neuroConstruct, but any updates to this would require updates to the Java implementation here. See discussions here.

Additional context
The closest thing to a reference implementation of SWC is the https://github.com/pgleeson/Cvapp-NeuroMorpho.org which is based on @robertcannon's original CVapp application. This is used on NeuroMorpho.Org for viewing SWC files.

The main issue when importing SWC files is deciding what to do when the soma is missing/badly specified/only present as a traced outline, and how to handle connection to the first real point (on dendrites etc.). There are a number of examples here: https://github.com/pgleeson/Cvapp-NeuroMorpho.org/tree/master/caseExamples for the various options, as well as a doc discussing these: https://github.com/pgleeson/Cvapp-NeuroMorpho.org/blob/master/caseExamples/SomaFormat-NMOv5.3.pdf

@MRIO and colleagues are interested in working on this functionality.

[pgleeson]

Note 1: The notebook here: https://github.com/NeuralEnsemble/libNeuroML/blob/development/notebooks/CellMorphology.ipynb may be useful for libNeuroML helper methods for morphologies.
Note 2: Ideally for a simple SWC file exported to NML via neuroConstruct and via this new script, they should be pretty much identical.

cc @WardDPeeters

[WardDPeeters]

Thanks! That should really help speed the process along! I'll probably also approach you via Slack regarding some other functions and/or specific variables in the near future.

[WardDPeeters]

Hi!

I had a couple of questions regarding a few different variables within a cell morphology in nml;

My first question is regarding the value 'proximal'. Every segment has a point for proximal and distal - where the segment begins and where it ends, but in some nml files I've looked at I've noticed that for some segments, proximal points are missing while for others they are not. I figured this maybe had to do with assuming that the distal point of the parent segment is also the proximal point of the next, but I'm not sure on that one.
My second question is alro regarding the segments, namely the value 'fraction_along'. SWC-files appear to only be able to handle values that are 0 or 1 here, but what does this value actually represent exactly?

[pgleeson]

Hi @WardDPeeters. Regarding the first point, yes, if the proximal is missing you use a point on the parent segment. If fractionAlong is missing or 1 (default) it's the distal point of the parent, and if fractionAlong==0, it's the proximal point. For vals between 0 nad 1 use interpolation The helper method: cell.get_actual_proximal(seg.id) can help you get the real proximal point, see https://github.com/NeuralEnsemble/libNeuroML/blob/master/notebooks/CellMorphology.ipynb.

Regarding the second point, there are no segments in swc, only points. Points are connected to their parent points. The simplest interpretation of 2 connected points is to make a corresponding NML segment for this. The complexities arise when the start point is the soma (often it's intentionally left out of the swc) and whether the daughter dendrites inherit the soma's (larger) radius... The pointers in the first comment above outline the various cases...

[WardDPeeters]

Hello @pgleeson! I have some good news, I have a first working version of a converter! I need to finetune it a bit, specifically to incorporate the detection and handling of weird structures like loops, or anything of the like.
Regarding this, I had the question, what should I do with the loops that might be present in a morphology? Do they need to be processed, or just detected and throw an error message? Is it even possible to have a loop encoded in an swc file?

[pgleeson]

@WardDPeeters Apologies for the delay in getting back to you. Glad you have an initial version.

Regarding loops, certainly NeuroML doesn't allow loops, and probably SWC too, though there's nothing stopping someone entering a sequence of points in an SWC file forming a loop. For your purposes here it's best to throw an error giving the specifics on which points are connected in loops.

[sanjayankur31]

Hi @WardDPeeters

How are you getting on with this task? Is there anything we can help you with?

Would the code you have so far be available somewhere for us to look at too please?

Cheers,

[sanjayankur31]

Hi @WardDPeeters , would this be the current state of the converter please?

https://github.com/WardDPeeters/Morphologies

[WardDPeeters]

@sanjayankur31 Yess! The main converter is in the Main branch of the folder (it's called SWC_to_NML_Converter_Dev.py), it should work just fine, but I haven't touched the code in quite a while, so there might be some unresolved bugs that I did not yet know of.

Furthermore, the file that you want to convert can only be pasted into the code, not yet converted through a command terminal yet, however there should be someone working on that in the coming time.

[sanjayankur31]

That's great, thanks very much. I'll play with it when I can, and I'll get in touch before I jump in to making any tweaks.

[MRIO]

Thanks Ward, and indeed, Ankur, there’s a new student (Cindy Steward) working on making the script a python command line function. And as we’d like this function to be as general as possible, we’d like it to be able handle “difficult" .swc, e.g., those with not entirely consistent annotations/parenthoods. Potentially going with user input to resolve ambiguities. To that end, I was wondering if you have any examples of typical annotation problems we could keep an eye out for?

…

On 6 Sep 2022, at 16:15, WardDPeeters ***@***.***> wrote: @sanjayankur31 <https://github.com/sanjayankur31> Yess! The main converter is in the Main branch of the folder (it's called SWC_to_NML_Converter_Dev.py), it should work just fine, but I haven't touched the code in quite a while, so there might be some unresolved bugs that I did not yet know of. Furthermore, the file that you want to convert can only be pasted into the code, not yet converted through a command terminal yet, however there should be someone working on that in the coming time. — Reply to this email directly, view it on GitHub <#89 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB5EGFH7OA73U4W7VLGMKVTV45GXNANCNFSM4Y6TQRQQ>. You are receiving this because you were mentioned.

[WardDPeeters]

I've already handled a few of them, I believe I put in error messages/resolutions for both "circular" dendrites (so segments with parents such that it loops back on itself after a while), and somas that are indicated using a soma outline instead of a singular node that represents the whole soma.
Other typical ambiguities should still be taken full care of.

[MRIO]

Great! Does anyone here by any chance have a set of funky swc’s they would be willing to share as test set? Else I’ll ask Cindy to put one together by scraping neuromorpho.org <http://neuromorpho.org/>.

…

On 7 Sep 2022, at 11:16, WardDPeeters ***@***.***> wrote: I've already handled a few of them, I believe I put in error messages/resolutions for both "circular" dendrites (so segments with parents such that it loops back on itself after a while), and somas that are indicated using a soma outline instead of a singular node that represents the whole soma. Other typical ambiguities should still be taken full care of. — Reply to this email directly, view it on GitHub <#89 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB5EGFADP5N2TKGCMMDUSCTV5BMOHANCNFSM4Y6TQRQQ>. You are receiving this because you were mentioned.

[pgleeson]

Do have a look again at https://github.com/pgleeson/Cvapp-NeuroMorpho.org/tree/master/caseExamples, particularly the pdf there for some simplified examples of different ways to handle somas in SWC.

[MRIO]

Brilliant, thank you!

[sanjayankur31]

@AdityaBITMESRA : assigning to you so folks know this is being worked on.

[MRIO]

Hi Sietse and Sotirios, make sure you advertise here! Cheers, M. On 17 Jun 2024, at 11:58, Ankur Sinha ***@***.***> wrote: @AdityaBITMESRA<https://github.com/AdityaBITMESRA> : assigning to you so folks know this is being worked on. — Reply to this email directly, view it on GitHub<#89 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AB5EGFEC6J4PGCBZVTCMHV3ZH2XLLAVCNFSM6AAAAABJNVMLBCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNZSHEZTKOJXGA>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[Sietse20]

Hi everyone!

I'm currently working on an SWC to NML converter.
You can find a detailed explanation of the converter in the converter_notebook.ipynb file on the following GitHub page:
https://github.com/Sietse20/BEP-morphologies

This notebook not only provides an overview of how the converter works but can also be used to convert your own files.
If you have any questions or suggestions, please feel free to leave a comment!

Best,
Sietse