add multiple experts per moe layer

[jonhilgart22]

Add the ability to use multiple types of networks per MoE layer instead of only one network.

[awan-10]

@jonhilgart22 - thanks for the PR! I do not fully understand the need for this PR without looking at the client-side code. Do you have any example code to explain the usage of this new feature?

[jonhilgart22]

@jonhilgart22 - thanks for the PR! I do not fully understand the need for this PR without looking at the client-side code. Do you have any example code to explain the usage of this new feature?

My intention is to attempt to train MoE with existing fine-tuned models as the experts. So, instead of having many-to-one experts to network types (e.g. 10 experts of MLP layers) you could instead train differing experts per layer ( t5 as one and gpt2 as another).

[awan-10]

@jonhilgart22. Sounds good. Can you please either modify the existing MoE unit test or add one so we know that this does not introduce any issues for the standard MoE API?

[awan-10]

@jonhilgart22 - please see that failing MoE unit tests. I think that is an indication that something is broken with the new changes.

[jonhilgart22]

closing as this is not exactly what I'm looking for. BTM is closer.