Multi-Threading Performance Issue with TBB in Gramine-SGX

[BaoSKY]

I am attempting to use multi-threading with the TBB library to optimize the performance of my algorithm. The implementation works well in both REE and Gramine-direct modes, but I am encountering a significant slowdown in Gramine-SGX.

Here are the results of my experiments:

• Max threads set to 1:
  The performance is comparable across REE, Gramine-direct, and Gramine-SGX.

• Max threads set to 16:
  The performance remains comparable between REE and Gramine-direct. However, in Gramine-SGX, the performance slows down by approximately 5x.

Notably, the algorithm does not involve any I/O operations, so I suspect this is not an issue related to Ocall/Ecall overhead.

Questions:

• Does Gramine-SGX have limited support for TBB or multi-threading in general?

• Could this slowdown be inherent to the SGX architecture's multi-threading support?

I would greatly appreciate any insights or guidance on this issue. Thank you!

[kailun-qin]

Max threads set to 1:
Max threads set to 16:

Do you mean the setting in the TBB library? Have you adjusted the sgx.max_threads Gramine manifest option accordingly? Pls see https://gramine.readthedocs.io/en/latest/manifest-syntax.html#number-of-threads for details.

[BaoSKY]

Sure!

Here is is-sgx-available:

SGX supported by CPU: true
SGX1 (ECREATE, EENTER, ...): true
SGX2 (EAUG, EACCEPT, EMODPR, ...): true
Flexible Launch Control (IA32_SGXPUBKEYHASH{0..3} MSRs): true
SGX extensions for virtualizers (EINCVIRTCHILD, EDECVIRTCHILD, ESETCONTEXT): false
Extensions for concurrent memory management (ETRACKC, ELDBC, ELDUC, ERDINFO): false
CET enclave attributes support (See Table 37-5 in the SDM): false
Key separation and sharing (KSS) support (CONFIGID, CONFIGSVN, ISVEXTPRODID, ISVFAMILYID report fields): true
Max enclave size (32-bit): 0x80000000
Max enclave size (64-bit): 0x100000000000000
EPC size: 0x1fcf800000
SGX driver loaded: true
AESMD installed: true
SGX PSW/libsgx installed: true
#PF/#GP information in EXINFO in MISC region of SSA supported: true
#CP information in EXINFO in MISC region of SSA supported: false

And here is Gramine Manifest:

sgx.debug = false
sgx.edmm_enable = {{ 'true' if env.get('EDMM', '0') == '1' else 'false' }}
sgx.enclave_size = "2G"
sgx.max_threads = 20

And here is build information:

Attributes (required for enclave measurement):
size: 0x80000000
edmm: False
max_threads: 20

[kailun-qin]

Thanks, nothing looks suspicious to me.

My guess is that tbb::global_control c(tbb::global_control::max_allowed_parallelism, MAX_THREAD_NUM) only specifies the max number of threads while TBB dynamically manages the exact number of threads used in your workloads, which may for some reasons differ between the native, gramine-direct and gramine-sgx execution. Can you pls share the "trace"-level log of your -direct and -sgx execution?

[BaoSKY]

Do you mean setting loader.log_level = "trace" in the Gramine manifest?
Below are the "trace"-level logs for both -direct and -sgx executions.
trace_gramine_direct.log
trace_gramine_sgx.log

[kailun-qin]

[P1:T9:main] ...

It looks like under Gramine-SGX, less than 16 threads have been spawned. Is the workload (inside your parallel region) substantial enough to keep all threads busy? If not, can you create a more substantial benchmark e.g., performing multiple iterations of matrix multiplication with large size of the matrices and re-check the perf diff between -direct and -sgx? I would expect the overhead to be sth ~2x rather than ~5x.

[BaoSKY]

Is the workload (inside your parallel region) substantial enough to keep all threads busy?

Thanks for your comment!

I'm working on an experiment to simulate continuous user queries to evaluate my algorithm.
A few days ago, I tested with 1000 queries, and the performance was good across both the native and gramine-direct environments, but not with gramine-sgx.

Now, when I increased the number of queries to 10,000 or more, the performance in gramine-sgx improved significantly. Now, it performs within a factor of 2x compared to both native and gramine-direct.