JIT model multiple inputs

[fteufel]

Hi, just came across your project, Great work, seem to be the only go bindings that actually work!
I have a jitted model taking two inputs and returning multiple outputs.
In C++, this seems to be done like this:
pytorch/pytorch#18337

std::vector<torch::jit::IValue> i;
std::vector<float> x = {2.0, 1.0, 0.5, 3.0, 4.0};
std::vector<float> y = {3.0, 2.0};
i.push_back(torch::tensor(x));
i.push_back(torch::tensor(y));

torch::Tensor output = module->forward(i).toTensor();

How would I do this in gotch? Forward only accepts a single tensor.

[fteufel]

Just managed to make the multi-inputs work.

input_ids, _ := ts.Ones([]int64{2,73}, gotch.Int64, gotch.CPU)
input_mask, _ := ts.Ones([]int64{2,73}, gotch.Int64, gotch.CPU)
inputs := []ts.Tensor{*input_ids,*input_mask}

model, _ := ts.ModuleLoad("distilled_scripted.pt")

predictTmp,err := model.ForwardTs(inputs) //ForwardTs(Sclice)
fmt.Println(err)
fmt.Printf("%i", predictTmp)

However I'm still clueless for the multi-outputs. I coudn't find any function that allows the output to be a slice.
Libtorch API Error: forward did not return a tensor

[sugarme]

@fteufel

Although JIT API has not been complete yet, gotch provides 2 APIs to forward pass CModule:

• func (cm *CModule) ForwardTs(tensors []Tensor) (*Tensor, error)
• func (cm *CModule) ForwardIs(ivalues []IValue) (*IValue, error)

Have a look at JIT unit test for how to use.

Also, there 2 separate examples of using JIT in example folder: inference and train

Not sure if that what you're after?

[sugarme]

Just managed to make the multi-inputs work.

input_ids, _ := ts.Ones([]int64{2,73}, gotch.Int64, gotch.CPU)
input_mask, _ := ts.Ones([]int64{2,73}, gotch.Int64, gotch.CPU)
inputs := []ts.Tensor{*input_ids,*input_mask}

model, _ := ts.ModuleLoad("distilled_scripted.pt")

predictTmp,err := model.ForwardTs(inputs) //ForwardTs(Sclice)
fmt.Println(err)
fmt.Printf("%i", predictTmp)

However I'm still clueless for the multi-outputs. I coudn't find any function that allows the output to be a slice.
Libtorch API Error: forward did not return a tensor

Have you handled error in model, _ := ts.ModuleLoad("distilled_scripted.pt")? This error coming from C and just make sure the JIT module is loaded properly.

[fteufel]

Hi, Thanks for the hints!

the error was coming from ForwardTs, because the python signature of the model looks like this:
forward(input_ids, input_mask) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor] which ForwardTs can't handle.

Converting the input tensors to IValue and using ForwardIs fixed it:

input_ids, _ := ts.Ones([]int64{2,73}, gotch.Int64, gotch.CPU)
input_mask, _ := ts.Ones([]int64{2,73}, gotch.Int64, gotch.CPU)
input_ids_ival := ts.NewIValue(*input_ids)
input_mask_ival := ts.NewIValue(*input_mask)

inputs := []ts.IValue{*input_ids_ival,*input_mask_ival}


model, _ := ts.ModuleLoad("distilled_scripted.pt")
prediction,err := model.ForwardIs(inputs) 

Now I would just figure out how to get the three Tensors back out of the prediction IValue.

[sugarme]

@fteufel

prediction is IValue type. You may try to use func (iv *IValue) Value() interface{} and cast return value to your expected type.

Something like

var output []ts.Tensor
output = prediction.Value().([]ts.Tensor)

[fteufel]

prediction.Kind() gives []tensor.IValue, that should be ok I suppose.
The expected type casting however is not doing it.
panic: interface conversion: interface {} is []interface {}, not []tensor.Tensor

I'm really new to Go, as you can tell - and the concept of interface especially. I actually just have a codebase that uses tfgo and I need to swap out the core model for pytorch.
Thanks a lot for your help!

[sugarme]

@fteufel

I think the issue is related to type casting inside gotch JIT itself and it may fall into one of the unsupported cases.

As you mentioned output is Tuple[torch.Tensor, torch.Tensor, torch.Tensor], gotch may cast output to 'GenericList' and end up with []inferface{} instead of 'TensorList' which is []Tensor.

Can you try to cast in []interface{} type. I would suspect some error: IValueFromC method call - GenericList case: Unsupported item type....

var output []interface{}
output = prediction.Value().([]interface{})

// Or just
output := prediction.Value()

If you can share your JIT module .pt or any simplified version, I can trace up and may add more support to the library.

[fteufel]

Hi,

gave it a try:
Casting works, yielding output type: []interface {} as expected.

Here's the jit model:
https://mega.nz/file/uZ0DVQ7b#oDDerv_8mJmj1ypC_CX8OaWBxfYe4dXDvHGVgOiwCks
It's rather large (essentially Bert+custom prediction head), but it's probably better to check the real thing. As stated, it takes two Int64 tensors of size (batch_size, 73) as inputs.

Update:
I changed the signature in Python to List[torch.Tensor, torch.Tensor, torch.Tensor]. Now

output := prediction.Value().([]ts.Tensor)

works. Easy fix for the problem, as there is no advantage of using Tuple over List really.

[sugarme]

@fteufel

A fix #20 done on master and v0.3.8. Now you can cast output to []ts.Tensor as below. If that what you expect, please close this issue.

package main

import (
	"fmt"
	"log"

	"github.com/sugarme/gotch"
	ts "github.com/sugarme/gotch/tensor"
)

func main() {
	input_ids, _ := ts.Ones([]int64{2, 73}, gotch.Int64, gotch.CPU)
	input_mask, _ := ts.Ones([]int64{2, 73}, gotch.Int64, gotch.CPU)
	input_ids_ival := ts.NewIValue(*input_ids)
	input_mask_ival := ts.NewIValue(*input_mask)

	inputs := []ts.IValue{*input_ids_ival, *input_mask_ival}

	model, err := ts.ModuleLoad("distilled_scripted.pt")
	if err != nil {
		log.Fatal(err)
	}

	prediction, err := model.ForwardIs(inputs)
	if err != nil {
		log.Fatal(err)
	}

	xs := prediction.Value().([]ts.Tensor)
	for _, x := range xs {
		fmt.Printf("%i", &x)
	}
}

// Output:

TENSOR INFO:
        Shape:          [2 6]
        DType:          float32
        Device:         {CPU 1}
        Defined:        true

TENSOR INFO:
        Shape:          [2 70 37]
        DType:          float32
        Device:         {CPU 1}
        Defined:        true

TENSOR INFO:
        Shape:          [2 70]
        DType:          int64
        Device:         {CPU 1}
        Defined:        true