gramine-sgx hold at step of loading

[chongloubbt]

When I exec a program "xsce" with gramine-sgx, it hold at step of loading…

Is there any log to debug this? And what conditions can cause this problem？

[P1:T1:xsce] debug: Creating pipe: pipe.srv:33ae186edfd95bdcdd6f65630930dcc497bdda64c078b8cf83fd0fec87158b17
[P1:T1:xsce] debug: Allocated stack at 0x8f6cb000 (size = 0x4000000)
[P1:T1:xsce] debug: loading "file:xsce"

[dimakuv]

Could you try to use loader.log_level = "all" in the Gramine manifest file? I would find it very surprising if Gramine would just hang on loading a file...

[chongloubbt]

I have tried using loader.log_level = "all" configuration, but there are no more logs.

[dimakuv]

Can you show us the command with which you're running?

And the complete log that Gramine prints.

Also, what is the xsce program? Is there anything special about it?

[chongloubbt]

logs.txt
The command is "gramine-sgx xsce".

The logs.txt file is the full logs.

The xsce program is my test code, and it can be runned success on 2023.03.27.

I think there may be some limited in resources, but there is no logs to help me to debug.

[chongloubbt]

There is not anything special in the program. When I remove some code, it can be runned, but there is nothing special about the removed code.

[chongloubbt]

In the manifest, I config 64 thread. gramine-sgx can run 64 threads before, but now it hang at loading. The thread is only one as following:

loader.pal_internal_mem_size = "64M"
sys.stack.size = "32M"
sys.brk.max_size = "256M"

sys.insecure__allow_eventfd = true
sys.enable_sigterm_injection = true
sgx.thread_num = 64
sgx.debug = true
sgx.enclave_size = "4G"
sgx.max_threads = 64
sgx.file_check_policy = "allow_all_but_log"
sgx.nonpie_binary = true

[dimakuv]

it can be runned success on 2023.03.27.

What does this mean? What happened after this date? You modified the xsce program?

Looking at your log, you're using v1.3.1 of Gramine (debug: Gramine was built from commit: 1.3.1). Could you try the latest Gramine?

Also, have you tried running with gramine-direct? Does it work?

[chongloubbt]

1. I decrease the threads num in the xsce program, the modification should not affect program. And it has not yet run to the place where I modified it.
2. gramine-direct mode can run success.
3. 1.4 can't run success, because is_oot in gramine-sgx-get-token check failed.

[chongloubbt]

When I gdb attach the loader of gramine-sgx, it seems hang in async_exit_pointer as following:

[dimakuv]

GDB on the release version of Gramine won't show anything interesting. You'll need to build Gramine from sources and in debug mode (see https://gramine.readthedocs.io/en/stable/devel/building.html).

1.4 can't run success, because is_oot in gramine-sgx-get-token check failed.

I don't know what you mean by this, but you should try to resolve this issue first. Are you using a very old Intel CPU? Why do you need to OOT legacy SGX driver?

[chongloubbt]

gramine-sgx-get-token will return in the following code.

[chongloubbt]

I have tried to build source code follow the building doc, but there are many building errors.

[chongloubbt]

enclave_size is set as 32G，the example of ra-tls-mbedtls hang on, But enclave_size is set as 8G，it can run. Is there any info to debug that?

[dimakuv]

What is your machine? How much RAM does it have (the output of free -h)? How much EPC does it have? You can run is-sgx-available to learn about the EPC size.

I have a suspicion that you have little EPC, and probably also little RAM, so trying to run a 32GB enclave leads to page swapping and dramatic drop in performance.

[chongloubbt]

root@ubuntu:/home# is-sgx-available
SGX supported by CPU: true
SGX1 (ECREATE, EENTER, ...): true
SGX2 (EAUG, EACCEPT, EMODPR, ...): true
Flexible Launch Control (IA32_SGXPUBKEYHASH{0..3} MSRs): true
SGX extensions for virtualizers (EINCVIRTCHILD, EDECVIRTCHILD, ESETCONTEXT): false
Extensions for concurrent memory management (ETRACKC, ELDBC, ELDUC, ERDINFO): false
CET enclave attributes support (See Table 37-5 in the SDM): false
Key separation and sharing (KSS) support (CONFIGID, CONFIGSVN, ISVEXTPRODID, ISVFAMILYID report fields): true
Max enclave size (32-bit): 0x80000000
Max enclave size (64-bit): 0x100000000000000
EPC size: 0xfe000000
SGX driver loaded: true
AESMD installed: true
SGX PSW/libsgx installed: true

root@ubuntu:/home# free -h
total used free shared buff/cache available
Mem: 247Gi 43Gi 92Gi 717Mi 111Gi 201Gi
Swap: 9.0Gi 0B 9.0Gi

[dimakuv]

You have only 4GB of EPC size, which means that any SGX workload above 4GB will be slow. And the greater the size of the workload (like 32GB), the worse the performance will get. This explains why your workload hangs, I guess (have you tried waiting for several hours?).

[chongloubbt]

You have only 4GB of EPC size, which means that any SGX workload above 4GB will be slow. And the greater the size of the workload (like 32GB), the worse the performance will get. This explains why your workload hangs, I guess (have you tried waiting for several hours?).

On 2023.03.27, it run success after about 15minutes. Later on 2023.03.28, it hang for more than 10 hours, and I kill it.

[dimakuv]

On 2023.03.27, it run success after about 15minutes. Later on 2023.03.28, it hang for more than 10 hours, and I kill it.

Unfortunately, I do not know what changed in your environment on those days. From what I understand, the Gramine installation was the same. So, it doesn't look like a problem with Gramine.

[chongloubbt]

Is there any docker images, and the sgx driver/sdk/psw has been installed in it, then the gramine program can be built direct. Thanks.

[dimakuv]

There is one Docker image that we publish -- https://hub.docker.com/r/gramineproject/gramine (currently has Gramine v1.4). Please also read the README carefully on that web page.

[chongloubbt]

Thank you very much.
Can the Docker image be used to build the gramine source code？ Or can gramine source code can be built in docker？When I build the source code in the docker, it depend sgx.h and some more files of sgx driver. There is not those files in the docker.

[dimakuv]

You can circumvent the problem of missing sgx.h by just downloading it into some directory and specifying it during meson build, as is done here: https://github.com/gramineproject/gsc/blob/77b1f70e297a286a82708ec6c76b8e4243a5c1bd/templates/Dockerfile.common.compile.template#L22-L37

Alternatively, you can expose some volumes from the host OS to the Docker container, but I don't have a snippet at hand that you could copy-paste. (But this is a cleaner approach than the first one, if you want to do it properly.)

[woju]

Or can gramine source code can be built in docker？When I build the source code in the docker, it depend sgx.h and some more files of sgx driver. There is not those files in the docker.

`sgx.h` is from `linux-libc-dev` package, starting from version 5.11. You can use `debian:bullseye-backports` docker image to build gramine just fine (probably with intel-sgx repo added).

[chongloubbt]

OK, thanks.

[chongloubbt]

Run with gramine of debug mode, it seems hang in _PalVirtualMemoryAlloc function? Maybe the size of memset is too large??

[dimakuv]

Yeah, as I mentioned previously, it feels like you are limited in EPC, and you have a lot of page swapping.

Though one thing I don't understand -- why is it about xsce? Can you show ls -lah for the xsce binary? Could it be that it's so huge?

[chongloubbt]

When I build the source code of gramine v1.4 with debug, and it seems hang in ecall_enclave_start function.

[chongloubbt]

Thans.

I modify the sgx.enclave_size = "4G" to sgx.enclave_size = "2G", it can run success.
Maybe this is the reason, can Gramine output some debug/warn log info to show this condition.

[dimakuv]

I modify the sgx.enclave_size = "4G" to sgx.enclave_size = "2G", it can run success.

So the problem was indeed in the hardware limitation (limited EPC), which resulted in very slow loading of the xsce binary? And by decreasing the enclave size, the EPC limitation was not hit? Sorry, I'm just not sure I understand what is really happenning in your case.

Maybe this is the reason, can Gramine output some debug/warn log info to show this condition.

What is the reason? And what would you expect Gramine to output?

[chongloubbt]

So the problem was indeed in the hardware limitation (limited EPC), which resulted in very slow loading of the xsce binary? And by decreasing the enclave size, the EPC limitation was not hit? Yes, it is. I decrease the enclave size, it is not hang.

What is the reason? And what would you expect Gramine to output?
—> The EPC size limitation maybe cause gramine hang or slow loading. When this happened, if there some logs to show, such as warning: EPC size may cause program loading to be too slow or trace: some function in, some function out, the debugging will be faster.