Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

src: add /json/protocol endpoint to inspector #7491

Merged
merged 1 commit into from
Sep 23, 2016

Conversation

bnoordhuis
Copy link
Member

@bnoordhuis bnoordhuis commented Jun 29, 2016

Embed the compressed and minified protocol.json from the bundled
v8_inspector and make it available through the /json/protocol endpoint.

Refs: nodejs/diagnostics#52
CI: https://ci.nodejs.org/job/node-test-pull-request/3133/
EDIT: https://ci.nodejs.org/job/node-test-pull-request/3135/

@bnoordhuis bnoordhuis added c++ Issues and PRs that require attention from people who are familiar with C++. inspector Issues and PRs related to the V8 inspector protocol labels Jun 29, 2016
@nodejs-github-bot nodejs-github-bot added the zlib Issues and PRs related to the zlib subsystem. label Jun 29, 2016
strm.opaque = Z_NULL;
CHECK_EQ(Z_OK, inflateInit(&strm));
static const size_t kDecompressedSize =
PROTOCOL_JSON[0] * 0x1000u +
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0x10000u?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed. I'll fix that.

@jasnell
Copy link
Member

jasnell commented Jun 30, 2016

LGTM but can you add a quick statement to the commit log about why? :-)

result += flags_match.group(1).strip().split()
# PORT should match the definition in test/common.js.
env = { 'PORT': int(os.getenv('NODE_COMMON_PORT', '12346')) }
env['PORT'] += self.thread_id * 100
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding thread_id * 100 to the port number and requiring the port to match common.PORT seem to be kind of contradictory?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't make it up, exports.PORT += process.env.TEST_THREAD_ID * 100; is what test/common.js does.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, sorry, didn’t see that value is changed later in test/common.js.

@addaleax
Copy link
Member

This CI failure looks related

@joshgav
Copy link
Contributor

joshgav commented Jun 30, 2016

Does this commit intend to canonicalize V8's embedded protocol.json as Node's protocol.json? I don't think we should finalize that yet, that's the goal of the discussion in nodejs/diagnostics#52.

@bnoordhuis
Copy link
Member Author

This CI failure looks related

Looks like I got the dependency chain wrong. It also looks like we're doing that wrong in more places. I added a fix-up that makes the inspector build depend explicitly on the generated header.

New CI: https://ci.nodejs.org/job/node-test-pull-request/3137/

Does this commit intend to canonicalize V8's embedded protocol.json as Node's protocol.json?

No.

import zlib

if __name__ == '__main__':
fp = open(sys.argv[1])
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure if important for this kind of script, but fp is not explicitly closed. Since we depend on 2.6-2.7, maybe use a with statement instead?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The script is short-lived so I went for simplicity.

@addaleax
Copy link
Member

Apparently, that did not do the trick… https://ci.nodejs.org/job/node-test-commit-arm/3865/nodes=armv7-wheezy/console

@bnoordhuis
Copy link
Member Author

bnoordhuis commented Jun 30, 2016

I'm starting to think the bug might not be in my changes to node.gyp. Other buildbots do this:

  LD_LIBRARY_PATH=/home/iojs/build/workspace/node-test-commit-arm/nodes/armv7-ubuntu1404/out/Release/lib.host:/home/iojs/build/workspace/node-test-commit-arm/nodes/armv7-ubuntu1404/out/Release/lib.target:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; cd ../.; mkdir -p /home/iojs/build/workspace/node-test-commit-arm/nodes/armv7-ubuntu1404/out/Release/obj/gen; python tools/js2c.py "/home/iojs/build/workspace/node-test-commit-arm/nodes/armv7-ubuntu1404/out/Release/obj/gen/node_natives.h" lib/internal/bootstrap_node.js lib/_debug_agent.js lib/_debugger.js lib/assert.js lib/buffer.js lib/child_process.js lib/console.js lib/constants.js lib/crypto.js lib/cluster.js lib/dgram.js lib/dns.js lib/domain.js lib/events.js lib/fs.js lib/http.js lib/_http_agent.js lib/_http_client.js lib/_http_common.js lib/_http_incoming.js lib/_http_outgoing.js lib/_http_server.js lib/https.js lib/_linklist.js lib/module.js lib/net.js lib/os.js lib/path.js lib/process.js lib/punycode.js lib/querystring.js lib/readline.js lib/repl.js lib/stream.js lib/_stream_readable.js lib/_stream_writable.js lib/_stream_duplex.js lib/_stream_transform.js lib/_stream_passthrough.js lib/_stream_wrap.js lib/string_decoder.js lib/sys.js lib/timers.js lib/tls.js lib/_tls_common.js lib/_tls_legacy.js lib/_tls_wrap.js lib/tty.js lib/url.js lib/util.js lib/v8.js lib/vm.js lib/zlib.js lib/internal/child_process.js lib/internal/cluster.js lib/internal/freelist.js lib/internal/linkedlist.js lib/internal/net.js lib/internal/module.js lib/internal/process/next_tick.js lib/internal/process/promises.js lib/internal/process/stdio.js lib/internal/process/warning.js lib/internal/process.js lib/internal/readline.js lib/internal/repl.js lib/internal/socket_list.js lib/internal/util.js lib/internal/v8_prof_polyfill.js lib/internal/v8_prof_processor.js lib/internal/streams/lazy_transform.js lib/internal/streams/BufferList.js deps/v8/tools/splaytree.js deps/v8/tools/codemap.js deps/v8/tools/consarray.js deps/v8/tools/csvparser.js deps/v8/tools/profile.js deps/v8/tools/profile_view.js deps/v8/tools/logreader.js deps/v8/tools/tickprocessor.js deps/v8/tools/SourceMap.js deps/v8/tools/tickprocessor-driver.js ./config.gypi src/notrace_macros.py src/nolttng_macros.py src/perfctr_macros.py
  touch /home/iojs/build/workspace/node-test-commit-arm/nodes/armv7-ubuntu1404/out/Release/obj.target/node_dtrace_header.stamp
  touch /home/iojs/build/workspace/node-test-commit-arm/nodes/armv7-ubuntu1404/out/Release/obj.target/node_dtrace_provider.stamp
  touch /home/iojs/build/workspace/node-test-commit-arm/nodes/armv7-ubuntu1404/out/Release/obj.target/node_dtrace_ustack.stamp
  touch /home/iojs/build/workspace/node-test-commit-arm/nodes/armv7-ubuntu1404/out/Release/obj.target/node_etw.stamp
  touch /home/iojs/build/workspace/node-test-commit-arm/nodes/armv7-ubuntu1404/out/Release/obj.target/node_perfctr.stamp
  touch /home/iojs/build/workspace/node-test-commit-arm/nodes/armv7-ubuntu1404/out/Release/obj.target/specialize_node_d.stamp
  LD_LIBRARY_PATH=/home/iojs/build/workspace/node-test-commit-arm/nodes/armv7-ubuntu1404/out/Release/lib.host:/home/iojs/build/workspace/node-test-commit-arm/nodes/armv7-ubuntu1404/out/Release/lib.target:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; cd ../.; mkdir -p /home/iojs/build/workspace/node-test-commit-arm/nodes/armv7-ubuntu1404/out/Release/obj/gen; python tools/compress_json.py deps/v8_inspector/platform/v8_inspector/js_protocol.json "/home/iojs/build/workspace/node-test-commit-arm/nodes/armv7-ubuntu1404/out/Release/obj/gen/v8_inspector_protocol_json.h"

But the armv7-wheezy buildbot only does this:

  LD_LIBRARY_PATH=/home/iojs/build/workspace/node-test-commit-arm/nodes/armv7-wheezy/out/Release/lib.host:/home/iojs/build/workspace/node-test-commit-arm/nodes/armv7-wheezy/out/Release/lib.target:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; cd ../.; mkdir -p /home/iojs/build/workspace/node-test-commit-arm/nodes/armv7-wheezy/out/Release/obj/gen; python tools/js2c.py "/home/iojs/build/workspace/node-test-commit-arm/nodes/armv7-wheezy/out/Release/obj/gen/node_natives.h" lib/internal/bootstrap_node.js lib/_debug_agent.js lib/_debugger.js lib/assert.js lib/buffer.js lib/child_process.js lib/console.js lib/constants.js lib/crypto.js lib/cluster.js lib/dgram.js lib/dns.js lib/domain.js lib/events.js lib/fs.js lib/http.js lib/_http_agent.js lib/_http_client.js lib/_http_common.js lib/_http_incoming.js lib/_http_outgoing.js lib/_http_server.js lib/https.js lib/_linklist.js lib/module.js lib/net.js lib/os.js lib/path.js lib/process.js lib/punycode.js lib/querystring.js lib/readline.js lib/repl.js lib/stream.js lib/_stream_readable.js lib/_stream_writable.js lib/_stream_duplex.js lib/_stream_transform.js lib/_stream_passthrough.js lib/_stream_wrap.js lib/string_decoder.js lib/sys.js lib/timers.js lib/tls.js lib/_tls_common.js lib/_tls_legacy.js lib/_tls_wrap.js lib/tty.js lib/url.js lib/util.js lib/v8.js lib/vm.js lib/zlib.js lib/internal/child_process.js lib/internal/cluster.js lib/internal/freelist.js lib/internal/linkedlist.js lib/internal/net.js lib/internal/module.js lib/internal/process/next_tick.js lib/internal/process/promises.js lib/internal/process/stdio.js lib/internal/process/warning.js lib/internal/process.js lib/internal/readline.js lib/internal/repl.js lib/internal/socket_list.js lib/internal/util.js lib/internal/v8_prof_polyfill.js lib/internal/v8_prof_processor.js lib/internal/streams/lazy_transform.js lib/internal/streams/BufferList.js deps/v8/tools/splaytree.js deps/v8/tools/codemap.js deps/v8/tools/consarray.js deps/v8/tools/csvparser.js deps/v8/tools/profile.js deps/v8/tools/profile_view.js deps/v8/tools/logreader.js deps/v8/tools/tickprocessor.js deps/v8/tools/SourceMap.js deps/v8/tools/tickprocessor-driver.js ./config.gypi src/notrace_macros.py src/nolttng_macros.py src/perfctr_macros.py
  touch /home/iojs/build/workspace/node-test-commit-arm/nodes/armv7-wheezy/out/Release/obj.host/node_js2c.stamp

The output of ./configure on the two buildbots isn't hugely different. arm_fpu and arm_version are only used by V8.

--- a.txt       2016-06-30 10:39:49.475993197 +0200
+++ b.txt       2016-06-30 10:40:04.387993197 +0200
@@ -1,7 +1,7 @@
   'variables': { 'arm_float_abi': 'hard',
-                 'arm_fpu': 'vfpv3',
+                 'arm_fpu': 'vfp',
                  'arm_thumb': 0,
-                 'arm_version': '7',
+                 'arm_version': '6',
                  'asan': 0,
                  'debug_devtools': 'node',
                  'gas_version': '2.24',

cc @nodejs/build - ideas on what might be causing that?

EDIT: To be clear, CI runs for other pull requests show the exact same behavior. It's not isolated to this pull request.

@bnoordhuis
Copy link
Member Author

Ping @nodejs/build. Even just a 'no clue' would be appreciated. :-)

@pavelfeldman
Copy link
Contributor

It might make sense to expose this via the protocol itself i.e. Protocol.getJSON method would return the protocol definition. That way clients could reuse same web socket connection and hence save on CORS, etc. for the separate request.

@joaocgreis
Copy link
Member

On my local machine this compiles with make -j10 but fails with make -j1. make -n shows that making v8_inspector_protocol_json.h is one of the last things that make tries to do, after compiling inspector_agent.cc. What about adding 'v8_inspector_compress_protocol_json' to the dependencies in line 306 instead of the last fixup?

@bnoordhuis
Copy link
Member Author

@joaocgreis Good suggestion, updated. CI: https://ci.nodejs.org/job/node-test-pull-request/3227/

It might make sense to expose this via the protocol itself i.e. Protocol.getJSON method would return the protocol definition. That way clients could reuse same web socket connection and hence save on CORS, etc. for the separate request.

@pavelfeldman Correct me if I'm wrong but doesn't that introduce a chicken-and-egg problem? The client won't know what protocol to speak to request the protocol JSON.

@pavelfeldman
Copy link
Contributor

Correct me if I'm wrong but doesn't that introduce a chicken-and-egg problem?

It does, but it is manageable. There could be runtimes with different sets of capabilities available via the same endpoint. If we do use json/protocol to expose those capabilities, then it should be on the connection level in a form of RTTI. We have not implemented these capabilities yet, so it is up for debate.

@jasnell
Copy link
Member

jasnell commented Aug 9, 2016

ping @bnoordhuis :-)

@bnoordhuis
Copy link
Member Author

Let's take the debian approach: when faced with several options, do 'em all. Does publishing the protocol over both WS and HTTP sound good?

@pavelfeldman
Copy link
Contributor

@bnoordhuis we can do /json/protocol first to cover the immediate needs. we'll follow up with the same in chromium for consistency.

@bnoordhuis
Copy link
Member Author

Rebased, new CI: https://ci.nodejs.org/job/node-test-pull-request/3676/

Can I get a LGTM or two?

@@ -322,6 +322,7 @@
'dependencies': [
'deps/v8_inspector/third_party/v8_inspector/platform/'
'v8_inspector/v8_inspector.gyp:v8_inspector_stl',
'v8_inspector_compress_protocol_json#host',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: is there a reason for the inconsistent indent between this and the two lines above?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The previous line is a continuation of the one before it. Note how the first line doesn't have a comma at the end.

@bnoordhuis bnoordhuis force-pushed the diag52 branch 2 times, most recently from 68206a9 to 2655fb9 Compare August 23, 2016 19:05
@bnoordhuis
Copy link
Member Author

Rebased against master. Can someone sign off on this?

CI: https://ci.nodejs.org/job/node-test-pull-request/3808/

@jasnell
Copy link
Member

jasnell commented Aug 23, 2016

LGTM

1 similar comment
@cjihrig
Copy link
Contributor

cjihrig commented Aug 23, 2016

LGTM

@bnoordhuis
Copy link
Member Author

bnoordhuis commented Sep 11, 2016

Rebased. New CI run: https://ci.nodejs.org/job/node-test-pull-request/4008/

Barring nays or CI failures, I'll merge this tomorrow or the day after.

EDIT: Again, because "stderr: fatal: unable to access 'https://github.com/nodejs/node.git/': gnutls_handshake() failed: A TLS packet with unexpected length was received."

https://ci.nodejs.org/job/node-test-pull-request/4009/

@joshgav
Copy link
Contributor

joshgav commented Sep 15, 2016

CI again so we can get this merged: https://ci.nodejs.org/job/node-test-pull-request/4064

EDIT: Problems on ARM and AIX which are out of my realm to troubleshoot 😢

@jasnell
Copy link
Member

jasnell commented Sep 20, 2016

Ping @bnoordhuis

@bnoordhuis
Copy link
Member Author

@bnoordhuis
Copy link
Member Author

Known flake parallel/test-tick-processor-unknown on smartos, see #8725; everything else is green/yellow. I'm going to land this.

Embed the compressed and minified protocol.json from the bundled
v8_inspector and make it available through the /json/protocol endpoint.

Refs: nodejs/diagnostics#52
PR-URL: nodejs#7491
Reviewed-By: Colin Ihrig <[email protected]>
Reviewed-By: James M Snell <[email protected]>
@bnoordhuis bnoordhuis closed this Sep 23, 2016
@bnoordhuis bnoordhuis deleted the diag52 branch September 23, 2016 16:54
@bnoordhuis bnoordhuis merged commit 782620f into nodejs:master Sep 23, 2016
MylesBorins pushed a commit to MylesBorins/node that referenced this pull request Sep 28, 2016
Embed the compressed and minified protocol.json from the bundled
v8_inspector and make it available through the /json/protocol endpoint.

Refs: nodejs/diagnostics#52
PR-URL: nodejs#7491
Reviewed-By: Colin Ihrig <[email protected]>
Reviewed-By: James M Snell <[email protected]>
jasnell pushed a commit that referenced this pull request Sep 29, 2016
Embed the compressed and minified protocol.json from the bundled
v8_inspector and make it available through the /json/protocol endpoint.

Refs: nodejs/diagnostics#52
PR-URL: #7491
Reviewed-By: Colin Ihrig <[email protected]>
Reviewed-By: James M Snell <[email protected]>
MylesBorins pushed a commit that referenced this pull request Sep 30, 2016
Embed the compressed and minified protocol.json from the bundled
v8_inspector and make it available through the /json/protocol endpoint.

Refs: nodejs/diagnostics#52
PR-URL: #7491
Reviewed-By: Colin Ihrig <[email protected]>
Reviewed-By: James M Snell <[email protected]>
cyrus-and added a commit to cyrus-and/chrome-remote-interface that referenced this pull request Oct 5, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c++ Issues and PRs that require attention from people who are familiar with C++. inspector Issues and PRs related to the V8 inspector protocol zlib Issues and PRs related to the zlib subsystem.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

10 participants