application crash on HTTPS connection using a self-signed certificate without keyCertSig bit

[fknx]

Version: 14.16.0
Platform: Microsoft Windows NT 10.0.19041.0 x64
Subsystem: https

Background

Self-signed certificates without the keyCertSig bit were rejected by OpenSSL. This was fixed in version 1.1.1h. Trying to connect to a server using such a certificate using older Node.js versions resulted in an UNABLE_TO_VERIFY_LEAF_SIGNATURE error.

What steps will reproduce the bug?

Connecting to a HTTPS Server that is using a self-signed certificate without the keyCertSig bit causes the application to hang/crash when this certificate is passed to the ca option of the https.Agent.

Client:

const fs = require('fs');
const https = require('https');

const certificate = fs.readFileSync('localhost.cer', 'utf8');

https.request({ hostname: '127.0.0.1', port: 4001, ca: certificate }, (res) => {
    console.log(res);
});

Server:

const fs = require('fs');
const https = require('https');

const options = {
    cert: fs.readFileSync('localhost.cer', 'utf8'),
    key: fs.readFileSync('localhost.key', 'utf8')
};

https.createServer(options, (req, res) => {
    res.writeHead(200, {'Content-Type': 'text/plain'});
    res.write('Hello World!');
    res.end();
}).listen(4001);

certificates.zip

How often does it reproduce? Is there a required condition?

The issue occurs on every connection attempt.

What is the expected behavior?

The connection to the server should be established.

What do you see instead?

The application is unresponsive and starts to use more and more memory and crashes when the machine runs out of memory:

<--- Last few GCs --->

[25716:000001D3C7BCBB40]   165619 ms: Mark-sweep (reduce) 4092.7 (4105.3) -> 4092.6 (4106.3) MB, 3008.1 / 0.0 ms  (average mu = 0.084, current mu = 0.009) allocation failure scavenge might not succeed
[25716:000001D3C7BCBB40]   168348 ms: Mark-sweep (reduce) 4093.6 (4103.3) -> 4093.5 (4105.3) MB, 2701.4 / 0.0 ms  (average mu = 0.045, current mu = 0.010) allocation failure scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: MarkCompactCollector: young object promotion failed Allocation failed - JavaScript heap out of memory
 1: 00007FF627CE052F napi_wrap+109311
 2: 00007FF627C85256 v8::internal::OrderedHashTable<v8::internal::OrderedHashSet,1>::NumberOfElementsOffset+33302
 3: 00007FF627C86026 node::OnFatalError+294
 4: 00007FF62855163E v8::Isolate::ReportExternalAllocationLimitReached+94
 5: 00007FF6285364BD v8::SharedArrayBuffer::Externalize+781
 6: 00007FF6283E094C v8::internal::Heap::EphemeronKeyWriteBarrierFromCode+1516
 7: 00007FF6283CB58B v8::internal::NativeContextInferrer::Infer+59243
 8: 00007FF6283B0ABF v8::internal::MarkingWorklists::SwitchToContextSlow+57327
 9: 00007FF6283C470B v8::internal::NativeContextInferrer::Infer+30955
10: 00007FF6283BB82D v8::internal::MarkCompactCollector::EnsureSweepingCompleted+6269
11: 00007FF6283C395E v8::internal::NativeContextInferrer::Infer+27454
12: 00007FF6283C78EB v8::internal::NativeContextInferrer::Infer+43723
13: 00007FF6283D1142 v8::internal::ItemParallelJob::Task::RunInternal+18
14: 00007FF6283D10D1 v8::internal::ItemParallelJob::Run+641
15: 00007FF6283A49D3 v8::internal::MarkingWorklists::SwitchToContextSlow+7939
16: 00007FF6283BBCDC v8::internal::MarkCompactCollector::EnsureSweepingCompleted+7468
17: 00007FF6283BA524 v8::internal::MarkCompactCollector::EnsureSweepingCompleted+1396
18: 00007FF6283B8088 v8::internal::MarkingWorklists::SwitchToContextSlow+87480
19: 00007FF6283E66D1 v8::internal::Heap::LeftTrimFixedArray+929
20: 00007FF6283E87B5 v8::internal::Heap::PageFlagsAreConsistent+789
21: 00007FF6283DDA61 v8::internal::Heap::CollectGarbage+2033
22: 00007FF6283DBC65 v8::internal::Heap::AllocateExternalBackingStore+1317
23: 00007FF6283FDB71 v8::internal::Factory::NewJSArrayBufferView+241
24: 00007FF628400CF6 v8::internal::Factory::NewJSTypedArray+726
25: 00007FF62854BD96 v8::Uint8Array::New+294
26: 00007FF627CA81A5 node::Buffer::New+213
27: 00007FF627B49D2E uv_loop_size+23262
28: 00007FF627B4842C uv_loop_size+16860
29: 00007FF627B5812D v8_inspector::protocol::Binary::operator=+41741
30: 00007FF6284FE85F v8::internal::Builtins::builtin_handle+321471
31: 00007FF6284FDDF4 v8::internal::Builtins::builtin_handle+318804
32: 00007FF6284FE0E7 v8::internal::Builtins::builtin_handle+319559
33: 00007FF6284FDF33 v8::internal::Builtins::builtin_handle+319123
34: 00007FF6285DA0CD v8::internal::SetupIsolateDelegate::SetupHeap+464173
35: 00007FF6285729D2 v8::internal::SetupIsolateDelegate::SetupHeap+40498
36: 00007FF6285729D2 v8::internal::SetupIsolateDelegate::SetupHeap+40498
37: 00007FF6285729D2 v8::internal::SetupIsolateDelegate::SetupHeap+40498
38: 00007FF6285729D2 v8::internal::SetupIsolateDelegate::SetupHeap+40498
39: 00007FF6285729D2 v8::internal::SetupIsolateDelegate::SetupHeap+40498
40: 00007FF62857068E v8::internal::SetupIsolateDelegate::SetupHeap+31470
41: 00007FF62857027C v8::internal::SetupIsolateDelegate::SetupHeap+30428
42: 00007FF628440761 v8::internal::Execution::CallWasm+1649
43: 00007FF62843FFCF v8::internal::Execution::Call+191
44: 00007FF62852B917 v8::Function::Call+615
45: 00007FF627D0459B node::CallbackScope::~CallbackScope+1659
46: 00007FF627CFC274 v8::internal::compiler::Operator::EffectOutputCount+228
47: 00007FF627B43EEC std::basic_streambuf<char,std::char_traits<char> >::xsputn+60028
48: 00007FF627E6668D SSL_set_tlsext_max_fragment_length+57005
49: 00007FF627E6F9F3 SSL_is_init_finished+2547
50: 00007FF627E97934 SSL_set_default_read_buffer_len+3684
51: 00007FF627E904B0 i2d_SSL_SESSION+9456
52: 00007FF627E90421 i2d_SSL_SESSION+9313
53: 00007FF627E86BA7 SSL_write_ex+6071
54: 00007FF627E83D72 SSL_read+66
55: 00007FF627B40922 std::basic_streambuf<char,std::char_traits<char> >::xsputn+46258
56: 00007FF627B40C40 std::basic_streambuf<char,std::char_traits<char> >::xsputn+47056
57: 00007FF627B4317D std::basic_streambuf<char,std::char_traits<char> >::xsputn+56589
58: 00007FF627BAA039 v8::internal::MicrotaskQueue::microtasks_policy+1257
59: 00007FF627D1E2AB uv_tty_set_vterm_state+9211
60: 00007FF627D33E0C uv_loop_init+924
61: 00007FF627D3411A uv_run+202
62: 00007FF627C40255 v8::internal::OrderedHashTable<v8::internal::OrderedHashSet,1>::NumberOfBucketsOffset+9365
63: 00007FF627CB39B7 node::Start+311
64: 00007FF627B1686C RC4_options+339820
65: 00007FF628AB619C v8::internal::compiler::RepresentationChanger::Uint32OverflowOperatorFor+153532
66: 00007FFFC57F7034 BaseThreadInitThunk+20
67: 00007FFFC6222651 RtlUserThreadStart+33

Additional information

Node.js 12.21.0 shows the same behavior. Node.js 12.16.3 terminates the connection with the previously mentioned UNABLE_TO_VERIFY_LEAF_SIGNATURE error but does not hang or crash.

[addaleax]

I'll close this as a duplicate of #37757, but please correct me if that's wrong. This contains value information either way 👍