"workers" auto setup #11

uprtdev · 2017-09-07T13:47:52Z

Added the feature of setting the number of workers for the application
to the number of CPU cores (if "workers" parameter == 0)

to the number of cpu cores if "workers" parameter == 0

mar0x

Agree with special meaning of '0', but default value '1' is more predictable and safe.

uprtdev · 2017-09-08T07:26:40Z

Well, I agree with you. Let the default value will be '1'.

Can be reproduced by test/test_rewrite.py::test_rewrite_njs with enabled UndefinedBehaviorSanitizer: src/nxt_http_js.c:169:52: runtime error: applying zero offset to null pointer #0 0x10255b044 in nxt_http_js_ext_get_args nxt_http_js.c:169 nginx#1 0x102598ad0 in njs_value_property njs_value.c:1175 nginx#2 0x10259c2c8 in njs_vm_object_prop njs_vm.c:1398 nginx#3 0x102559d74 in nxt_js_call nxt_js.c:445 nginx#4 0x1023c0da0 in nxt_tstr_query nxt_tstr.c:276 nginx#5 0x102516ec4 in nxt_http_rewrite nxt_http_rewrite.c:56 nginx#6 0x1024fd86c in nxt_http_request_action nxt_http_request.c:565 nginx#7 0x1024d71b0 in nxt_h1p_request_body_read nxt_h1proto.c:998 nginx#8 0x1023f5c48 in nxt_event_engine_start nxt_event_engine.c:542 nginx#9 0x1023e2838 in nxt_thread_trampoline nxt_thread.c:126 nginx#10 0x18133e030 in _pthread_start+0x84 (libsystem_pthread.dylib:arm64e+0x7030) nginx#11 0x181338e38 in thread_start+0x4 (libsystem_pthread.dylib:arm64e+0x1e38) SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior src/nxt_http_js.c:169:52 Same fix was introduced in NJS: http://hg.nginx.org/njs/rev/4fba78789fe4

Can be reproduced by test/test_rewrite.py::test_rewrite_njs with enabled UndefinedBehaviorSanitizer: src/nxt_http_js.c:169:52: runtime error: applying zero offset to null pointer #0 0x10255b044 in nxt_http_js_ext_get_args nxt_http_js.c:169 nginx#1 0x102598ad0 in njs_value_property njs_value.c:1175 nginx#2 0x10259c2c8 in njs_vm_object_prop njs_vm.c:1398 nginx#3 0x102559d74 in nxt_js_call nxt_js.c:445 nginx#4 0x1023c0da0 in nxt_tstr_query nxt_tstr.c:276 nginx#5 0x102516ec4 in nxt_http_rewrite nxt_http_rewrite.c:56 nginx#6 0x1024fd86c in nxt_http_request_action nxt_http_request.c:565 nginx#7 0x1024d71b0 in nxt_h1p_request_body_read nxt_h1proto.c:998 nginx#8 0x1023f5c48 in nxt_event_engine_start nxt_event_engine.c:542 nginx#9 0x1023e2838 in nxt_thread_trampoline nxt_thread.c:126 nginx#10 0x18133e030 in _pthread_start+0x84 (libsystem_pthread.dylib:arm64e+0x7030) nginx#11 0x181338e38 in thread_start+0x4 (libsystem_pthread.dylib:arm64e+0x1e38) SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior src/nxt_http_js.c:169:52 Same fix was introduced in NJS: <http://hg.nginx.org/njs/rev/4fba78789fe4>

Can be reproduced by test/test_rewrite.py::test_rewrite_njs with enabled UndefinedBehaviorSanitizer: src/nxt_http_js.c:169:52: runtime error: applying zero offset to null pointer #0 0x10255b044 in nxt_http_js_ext_get_args nxt_http_js.c:169 nginx#1 0x102598ad0 in njs_value_property njs_value.c:1175 nginx#2 0x10259c2c8 in njs_vm_object_prop njs_vm.c:1398 nginx#3 0x102559d74 in nxt_js_call nxt_js.c:445 nginx#4 0x1023c0da0 in nxt_tstr_query nxt_tstr.c:276 nginx#5 0x102516ec4 in nxt_http_rewrite nxt_http_rewrite.c:56 nginx#6 0x1024fd86c in nxt_http_request_action nxt_http_request.c:565 nginx#7 0x1024d71b0 in nxt_h1p_request_body_read nxt_h1proto.c:998 nginx#8 0x1023f5c48 in nxt_event_engine_start nxt_event_engine.c:542 nginx#9 0x1023e2838 in nxt_thread_trampoline nxt_thread.c:126 nginx#10 0x18133e030 in _pthread_start+0x84 (libsystem_pthread.dylib:arm64e+0x7030) nginx#11 0x181338e38 in thread_start+0x4 (libsystem_pthread.dylib:arm64e+0x1e38) SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior src/nxt_http_js.c:169:52 Same fix was introduced in NJS: <http://hg.nginx.org/njs/rev/4fba78789fe4> Reviewed-by: Andrew Clayton <[email protected]>

Can be reproduced by test/test_rewrite.py::test_rewrite_njs with enabled UndefinedBehaviorSanitizer: src/nxt_http_js.c:169:52: runtime error: applying zero offset to null pointer #0 0x10255b044 in nxt_http_js_ext_get_args nxt_http_js.c:169 #1 0x102598ad0 in njs_value_property njs_value.c:1175 #2 0x10259c2c8 in njs_vm_object_prop njs_vm.c:1398 #3 0x102559d74 in nxt_js_call nxt_js.c:445 #4 0x1023c0da0 in nxt_tstr_query nxt_tstr.c:276 #5 0x102516ec4 in nxt_http_rewrite nxt_http_rewrite.c:56 #6 0x1024fd86c in nxt_http_request_action nxt_http_request.c:565 #7 0x1024d71b0 in nxt_h1p_request_body_read nxt_h1proto.c:998 #8 0x1023f5c48 in nxt_event_engine_start nxt_event_engine.c:542 #9 0x1023e2838 in nxt_thread_trampoline nxt_thread.c:126 #10 0x18133e030 in _pthread_start+0x84 (libsystem_pthread.dylib:arm64e+0x7030) #11 0x181338e38 in thread_start+0x4 (libsystem_pthread.dylib:arm64e+0x1e38) SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior src/nxt_http_js.c:169:52 Same fix was introduced in NJS: <http://hg.nginx.org/njs/rev/4fba78789fe4> Reviewed-by: Andrew Clayton <[email protected]>

Can be reproduced by test/test_rewrite.py::test_rewrite_njs with enabled UndefinedBehaviorSanitizer: src/nxt_http_js.c:169:52: runtime error: applying zero offset to null pointer #0 0x10255b044 in nxt_http_js_ext_get_args nxt_http_js.c:169 nginx#1 0x102598ad0 in njs_value_property njs_value.c:1175 nginx#2 0x10259c2c8 in njs_vm_object_prop njs_vm.c:1398 nginx#3 0x102559d74 in nxt_js_call nxt_js.c:445 nginx#4 0x1023c0da0 in nxt_tstr_query nxt_tstr.c:276 nginx#5 0x102516ec4 in nxt_http_rewrite nxt_http_rewrite.c:56 nginx#6 0x1024fd86c in nxt_http_request_action nxt_http_request.c:565 nginx#7 0x1024d71b0 in nxt_h1p_request_body_read nxt_h1proto.c:998 nginx#8 0x1023f5c48 in nxt_event_engine_start nxt_event_engine.c:542 nginx#9 0x1023e2838 in nxt_thread_trampoline nxt_thread.c:126 nginx#10 0x18133e030 in _pthread_start+0x84 (libsystem_pthread.dylib:arm64e+0x7030) nginx#11 0x181338e38 in thread_start+0x4 (libsystem_pthread.dylib:arm64e+0x1e38) SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior src/nxt_http_js.c:169:52 Same fix was introduced in NJS: <http://hg.nginx.org/njs/rev/4fba78789fe4> Reviewed-by: Andrew Clayton <[email protected]>

This issue was found with oss-fuzz. ==18420==WARNING: MemorySanitizer: use-of-uninitialized-value #0 0x55dd798a5797 in nxt_vsprintf unit/src/nxt_sprintf.c:163:31 #1 0x55dd798d5bdb in nxt_conf_vldt_error unit/src/nxt_conf_validation.c:1525:11 #2 0x55dd798dd4cd in nxt_conf_vldt_var unit/src/nxt_conf_validation.c:1560:16 #3 0x55dd798dd4cd in nxt_conf_vldt_if unit/src/nxt_conf_validation.c:1592:16 nginx#4 0x55dd798d55f4 in nxt_conf_vldt_object unit/src/nxt_conf_validation.c:2815:23 nginx#5 0x55dd798d6f84 in nxt_conf_vldt_access_log unit/src/nxt_conf_validation.c:3426:11 nginx#6 0x55dd798d55f4 in nxt_conf_vldt_object unit/src/nxt_conf_validation.c:2815:23 nginx#7 0x55dd798d47bd in nxt_conf_validate unit/src/nxt_conf_validation.c:1421:11 nginx#8 0x55dd79871c82 in LLVMFuzzerTestOneInput unit/fuzzing/nxt_json_fuzz.c:67:5 nginx#9 0x55dd79770620 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:614:13 nginx#10 0x55dd7975adb4 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:327:6 nginx#11 0x55dd7976084a in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:862:9 nginx#12 0x55dd7978cc42 in main /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20:10 nginx#13 0x7e8192213082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/libc-start.c:308:16 nginx#14 0x55dd7975188d in _start Uninitialized value was created by an allocation of 'error.i' in the stack frame #0 0x55dd798dd42b in nxt_conf_vldt_var unit/src/nxt_conf_validation.c:1557:5 #1 0x55dd798dd42b in nxt_conf_vldt_if unit/src/nxt_conf_validation.c:1592:16 The issue was in nxt_tstr_test() where we create an error message with nxt_sprintf(), where this error message is then later used with the '%s' format specifier which expects a nul-terminated string, but by default nxt_sprintf() doesn't nul-terminate, you must use the '%Z' specifier to signify a '\0' at the end of the string. Link: <https://github.com/google/oss-fuzz> Link: <https://oss-fuzz.com/testcase-detail/5545917827055616> Co-developed-by: Zhidao HONG <[email protected]> Signed-off-by: Zhidao HONG <[email protected]> Reviewed-by: Andrew Clayton <[email protected]> Signed-off-by: Arjun <[email protected]> [ Commit message/subject - Andrew ] Signed-off-by: Andrew Clayton <[email protected]>

This issue was found with oss-fuzz. ==18420==WARNING: MemorySanitizer: use-of-uninitialized-value #0 0x55dd798a5797 in nxt_vsprintf unit/src/nxt_sprintf.c:163:31 #1 0x55dd798d5bdb in nxt_conf_vldt_error unit/src/nxt_conf_validation.c:1525:11 #2 0x55dd798dd4cd in nxt_conf_vldt_var unit/src/nxt_conf_validation.c:1560:16 #3 0x55dd798dd4cd in nxt_conf_vldt_if unit/src/nxt_conf_validation.c:1592:16 nginx#4 0x55dd798d55f4 in nxt_conf_vldt_object unit/src/nxt_conf_validation.c:2815:23 nginx#5 0x55dd798d6f84 in nxt_conf_vldt_access_log unit/src/nxt_conf_validation.c:3426:11 nginx#6 0x55dd798d55f4 in nxt_conf_vldt_object unit/src/nxt_conf_validation.c:2815:23 nginx#7 0x55dd798d47bd in nxt_conf_validate unit/src/nxt_conf_validation.c:1421:11 nginx#8 0x55dd79871c82 in LLVMFuzzerTestOneInput unit/fuzzing/nxt_json_fuzz.c:67:5 nginx#9 0x55dd79770620 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:614:13 nginx#10 0x55dd7975adb4 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:327:6 nginx#11 0x55dd7976084a in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:862:9 nginx#12 0x55dd7978cc42 in main /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20:10 nginx#13 0x7e8192213082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/libc-start.c:308:16 nginx#14 0x55dd7975188d in _start Uninitialized value was created by an allocation of 'error.i' in the stack frame #0 0x55dd798dd42b in nxt_conf_vldt_var unit/src/nxt_conf_validation.c:1557:5 #1 0x55dd798dd42b in nxt_conf_vldt_if unit/src/nxt_conf_validation.c:1592:16 The issue was in nxt_tstr_test() where we create an error message with nxt_sprintf(), where this error message is then later used with the '%s' format specifier which expects a nul-terminated string, but by default nxt_sprintf() doesn't nul-terminate, you must use the '%Z' specifier to signify a '\0' at the end of the string. Signed-off-by: Arjun <[email protected]> Co-developed-by: Zhidao HONG <[email protected]> Signed-off-by: Zhidao HONG <[email protected]> Link: <https://github.com/google/oss-fuzz> Link: <https://oss-fuzz.com/testcase-detail/5545917827055616> Reviewed-by: Andrew Clayton <[email protected]> [ Commit message/subject - Andrew ] Signed-off-by: Andrew Clayton <[email protected]>

This issue was found with oss-fuzz. ==18420==WARNING: MemorySanitizer: use-of-uninitialized-value #0 0x55dd798a5797 in nxt_vsprintf unit/src/nxt_sprintf.c:163:31 #1 0x55dd798d5bdb in nxt_conf_vldt_error unit/src/nxt_conf_validation.c:1525:11 #2 0x55dd798dd4cd in nxt_conf_vldt_var unit/src/nxt_conf_validation.c:1560:16 #3 0x55dd798dd4cd in nxt_conf_vldt_if unit/src/nxt_conf_validation.c:1592:16 nginx#4 0x55dd798d55f4 in nxt_conf_vldt_object unit/src/nxt_conf_validation.c:2815:23 nginx#5 0x55dd798d6f84 in nxt_conf_vldt_access_log unit/src/nxt_conf_validation.c:3426:11 nginx#6 0x55dd798d55f4 in nxt_conf_vldt_object unit/src/nxt_conf_validation.c:2815:23 nginx#7 0x55dd798d47bd in nxt_conf_validate unit/src/nxt_conf_validation.c:1421:11 nginx#8 0x55dd79871c82 in LLVMFuzzerTestOneInput unit/fuzzing/nxt_json_fuzz.c:67:5 nginx#9 0x55dd79770620 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:614:13 nginx#10 0x55dd7975adb4 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:327:6 nginx#11 0x55dd7976084a in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:862:9 nginx#12 0x55dd7978cc42 in main /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20:10 nginx#13 0x7e8192213082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/libc-start.c:308:16 nginx#14 0x55dd7975188d in _start Uninitialized value was created by an allocation of 'error.i' in the stack frame #0 0x55dd798dd42b in nxt_conf_vldt_var unit/src/nxt_conf_validation.c:1557:5 #1 0x55dd798dd42b in nxt_conf_vldt_if unit/src/nxt_conf_validation.c:1592:16 The issue was in nxt_tstr_test() where we create an error message with nxt_sprintf(), where this error message is then later used with the '%s' format specifier which expects a nul-terminated string, but by default nxt_sprintf() doesn't nul-terminate, you must use the '%Z' specifier to signify a '\0' at the end of the string. Signed-off-by: Arjun <[email protected]> Co-developed-by: Zhidao HONG <[email protected]> Signed-off-by: Zhidao HONG <[email protected]> Link: <https://github.com/google/oss-fuzz> Reviewed-by: Andrew Clayton <[email protected]> [ Commit message/subject - Andrew ] Signed-off-by: Andrew Clayton <[email protected]>

Under Ubuntu 24.04 the pytest for test/test_php_isolation.py::test_php_isolation_rootfs fails due to Unit aborting (SIGABRT) in the PHP language module due to FORIFY_SOURCE hardening detecting a buffer overflow 2024/10/16 16:46:54 [info] 11661#11661 "phpinfo" application started *** buffer overflow detected ***: terminated 2024/10/16 16:46:54 [alert] 11660#11660 app process 11661 exited on signal 6 After spending an extraordinary amount of time faffing around with Ubuntu and pytests (they don't make for a pleasant combination) I was able to reproduce it. The crash was occurring here nginx#4 0x00007ebe818288ff in __GI_abort () at ./stdlib/abort.c:79 nginx#5 0x00007ebe818297b6 in __libc_message_impl ( fmt=fmt@entry=0x7ebe819ce765 "*** %s ***: terminated\n") at ../sysdeps/posix/libc_fatal.c:132 nginx#6 0x00007ebe81936c19 in __GI___fortify_fail ( msg=msg@entry=0x7ebe819ce74c "buffer overflow detected") at ./debug/fortify_fail.c:24 nginx#7 0x00007ebe819365d4 in __GI___chk_fail () at ./debug/chk_fail.c:28 nginx#8 0x00007ebe8134a055 in mempcpy (__len=10, __src=0x7ebe8160ade8, __dest=0x571ba9bd0930) at /usr/include/x86_64-linux-gnu/bits/string_fortified.h:45 nginx#9 fake_data_segment (info=0x0, sysdb=0x571ba9bcf080) at /usr/src/php8.1-8.1.30-1+ubuntu24.04.1+deb.sury.org+1/ext/date/lib/parse_tz.c:921 nginx#10 timelib_builtin_db () at /usr/src/php8.1-8.1.30-1+ubuntu24.04.1+deb.sury.org+1/ext/date/lib/parse_tz.c:1084 nginx#11 0x00007ebe812e0885 in zm_info_date (zend_module=0x571ba9a14420) [Well as best as I can tell, as this is from the php 8.1 packages from <https://github.com/oerdnj/deb.sury.org>, I don't know where the packages (I'm assuming it's packages) shivammathur/setup-php@v2 installs come from.] So we get killed in fake_data_segment(), the thing is, that function (as well as timelib_builtin_db()) doesn't exist in upstream PHP. It turns out these come from a patch that is applied by distributions to make PHP use the system installed timezone database rather than the one built into PHP. I was unable to reproduce this with vanilla PHP 8.1. It can be triggered on affected builds with the following config { "listeners": { "[::1]:8080": { "pass": "applications/php" } }, "applications": { "php": { "type": "php", "root": "/app/php", "isolation": { "rootfs": "/tmp/unit-root", "namespaces": { "mount": true, "credential": true, "pid": true } } } } } The crux of the issue seems to come down to in this case PHP can't open the tz database as it's not contained in the new mount namespace. 190437 openat(AT_FDCWD, "/usr/share/zoneinfo/", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = -1 ENOENT (No such file or directory) 190437 openat(AT_FDCWD, "/usr/share/zoneinfo/zone.tab", O_RDONLY) = -1 ENOENT (No such file or directory) 190437 writev(2, [{iov_base="*** ", iov_len=4}, {iov_base="buffer overflow detected", iov_len=24}, {iov_base=" ***: terminated\n", iov_len=17}], 3) = 45 ... 190437 --- SIGABRT {si_signo=SIGABRT, si_code=SI_TKILL, si_pid=2, si_uid=65534} --- 190437 +++ killed by SIGABRT +++ Specifically the issue is with the following code in the patch (certainly an earlier version of the patch, this is from a Debian patch <https://sources.debian.org/src/php8.2/8.2.20-1~deb12u1/debian/patches/0007-Add-support-for-use-of-the-system-timezone-database.patch/>) + data = malloc(3 * sysdb->index_size + 7); + + p = mempcpy(data, FAKE_HEADER, sizeof(FAKE_HEADER) - 1); If the zone file hasn't been found then sysdb->index_size is 0. So we malloc(3) a total of 7 bytes. However, sizeof(FAKE_HEADER) - 1 is 10. (Hence the __len=10 in the mempcpy(3) in the above backtrace). Of course 10 doesn't fit into 7 and the FORTIFY_SOURCE hardening kicks in and SIGABRTs the process. Now, it's worth noting that this issue doesn't occur with PHP 8.2 and 8.3. As can been seen from the Fedora patch for this <https://src.fedoraproject.org/rpms/php/blob/rawhide/f/php-8.4.0-systzdata-v24.patch> They actually have a fix incorporated r23: fix possible buffer overflow So the above patch now does + data = malloc(3 * sysdb->index_size + sizeof(FAKE_HEADER) - 1); + + p = mempcpy(data, FAKE_HEADER, sizeof(FAKE_HEADER) - 1); So you will always get at least the required 10 bytes allocated. I assume the PHP 8.2 & 8.3 packages either no longer use this patch or have the fixed version. I don't know... I haven't found the sources... Anyway the above was more about satisfying myself that the problem wasn't with Unit. PHP 8.1 is now in security maintenance mode and people are actively encouraged to upgrade to 8.2/8.3 So lets just remove 8.1 from our testing... [It's also worth noting that after all this, the ubuntu-latest runners seemed to have switched back from 24.04 to 22.04. However lets stick with this and the other ci fixes as who knows when it'll go back to 24.04 (or some other version) again...] Link: <https://www.php.net/supported-versions.php> Signed-off-by: Andrew Clayton <[email protected]>

Added the feature of setting the number of workers for the application

190b613

to the number of cpu cores if "workers" parameter == 0

mar0x reviewed Sep 7, 2017

View reviewed changes

Set 'workers' default value to 1 from 'auto'

f1aeda4

VBart closed this Sep 11, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

"workers" auto setup #11

"workers" auto setup #11

uprtdev commented Sep 7, 2017

mar0x left a comment

uprtdev commented Sep 8, 2017 •

edited

Loading

"workers" auto setup #11

"workers" auto setup #11

Conversation

uprtdev commented Sep 7, 2017

mar0x left a comment

Choose a reason for hiding this comment

uprtdev commented Sep 8, 2017 • edited Loading

uprtdev commented Sep 8, 2017 •

edited

Loading