Replace Celluloid::IO with native code (for TCP) #246
Comments
mechanicalduck
changed the title
|
+1, I've spent hours trying to get the TCP forwarding to work in Windows (with Guard on a VM) but as soon as something tries to query whatever port I have it set up on, it crashes. |
|
+1 |
|
It's best to run with LISTEN_GEM_DEBUGGING=2 environment variable, because when something in listen silently crashes, all kinds of errors can happen - unrelated to the problem. Also, master branch has unreleased fixes to handling TCP options (which I screwed up while refactoring some time ago) and showing how ports are used (socket/server). Also, celluloid-0.16.0 was released, so I'd try upgrading it first. |
|
Test (celluloid/celluloid-io#74 (comment)) [Error is provoked by connecting (e.g. using telnet client)] However, it doesn't seem to increase detail in logging. With best regards |
|
@mechanicalduck, @noinkling - I'm closing this, because of #258. If anyone could provide a separate Listen TCP server implementation (that works for all platforms), I'd move/remove the server code from Listen itself. |
e2
changed the title
|
Any updates on this? Is there any currently working solution to run guard in a vagrant with some listen solution on a win32 host? |
|
@joshco - probably best to just work inside a VM completely. Liste 3.x doesn't use Celluloid anymore, but TCP also isn't implemented. I personally don't use Windows and there are other areas in Listen I need to work on first. As a workaround, you could just setup a TCP server based on Listen that sends changes to a TCP listener inside Guard directly. Guard has an internal API for simulating file messages. If someone creates a proof-of concept on the Guard side, I could set up an official Guard TCP client plugin. |
|
Thanks for the quick response. Moving completely inside the vagrant VM isn't an option. (too many other things become much more difficult ) I might be able to write something that uses listen and sends messages. |
You can have any protocol you want since you'd be implementing both sides.
That file implements the Pry "change" command, which simulates a file. I'm pretty sure the I'm not sure about the overall speed, but it's a start. Listen 3.x is heavily optimized to be responsive and fit within "deadlines", so if you send the changes as soon as you get them, it should be enough - especially since Guard will also respond immediately to the changes. You can use Let me know if you need anything else. |
|
@joshco - if you want to create something more "future-oriented", make sure to include a "protocol version" in your protocol. Listen didn't do this and it was a mistake. |
|
this is super helpful. Starting to code, but one question Can it sit blocked on a socket read loop? Or should I start up my own thread for to monitor the socket for incoming data and then just call the async_queue_add from there? Re protocol, yup, I hear ya. |
You can just start it in a thread. Later you can stop the connection and do a async_queue_add is designed to be called from anywhere - it uses Thread::Queue and the interactor+commands are handled asynchronously. |
|
roger |
|
do I want to put my tcp code in the proc that is called like run_on_additions, or do I want to put it in the do block? |
|
You're only responding to :start and :stop. So it's probably something like this: data = [nil, nil]
starter = proc do |data, _, _|
th = Thread.new(0.1) do
data.last = TCPSocket.new('mypc', 4000);
while (line = data.last.gets)
# (...)
Guard.async_queue.......
end
data.last.close
end
data.first = th
end
finisher = proc do |data, _, _|
data.last.close # tcp
data.first.join
end
guard :yield, { start: starter, stop: finisher, object: data } do
# nothing to watch or do
end |
|
Or you can just go OO from the start with e.g. guard :yield, { start: proc { |con| con.start }, stop: proc { |con| con.stop }, object: MyTCPConnection.new } do |
|
got it On Wed, Jun 1, 2016 at 1:30 AM, Cezary Baginski [email protected]
_Josh Co_hen [email protected] | @joshco | LinkedIn Looking for a contract developer to work on APIs, integrations, servers, |
Success!@e2 Your protocol versioning got me thinking deeply, and so i decided to just use HTTP. My solution also reverses the client/server model that you had because it seems to fit my mind better. The existing model always seemed strange since the guard instance would contact the listen server, but then be receiving pushed data back to it. (But then again, that's websockets style notifications. huh) Why muck around with masrhalling payloads and metadata over raw sockets? The change notifications, which are relatively infrequent, are reasonably well modeled as webhooks. The version problem is handled RESTishly as part of the URL, which takes a JSON body. Leveraging HTTP's message framing also opens the door to other functions at different URL endpoints and versioning (and evil too!) FS ListenerWin_listen.rb, fires up the listen gem to watch a directory and takes a URL to POST change notifications to. Notification Receiver (Guardfile)Using your advice on guard-yield, I embedded a sinatra server inside the Guardfile that accepts the webhooks and calls the async queue add. https://gist.github.com/joshco/6c84fd30dfbe0118bc42570000eb268d I will pretty this up and package it how you like for others to more easily reuse in a week or so. Right now I need to get back to the current deadline, which hopefully this fix will make faster. |
I actually wanted to suggest that. I thought it was overkill, but it actually makes sense since HTTP servers are designed to process requests quickly anyway, so the overhead isn't huge at all. It actually doesn't matter since listen accumulates changes to avoid spamming if there are lots of disk changes. So it's only minor overhead (wait_for_delay + tcp overhead + http request overhead). You can probably reduce the wait_for_delay as much as possible, e.g. 0.01. You'll know if the value is too small if you're editor starts generating other than 1 event for saving files. So the value can be as low as your editor and editor settings allow. Overall response time here is important, because it boosts productivity if there's less lag between changing a file and the action. (At least for me).
I actually think it's unreasonable to say raw TCP should be used here, unless there are numbers/benchmarks proving otherwise. The
Yeah, I had the same idea when I first tried to work out why Listen's TCP seemed "reversed". It doesn't really matter. The idea to have the server in Listen comes from the fact that if you restart guard, the connection is broken. And usually the Listener should be watching for files non-stop anyway, so that's where the server should be. It seems backwards, but in fact Guard is just a client listening for changes.
I think it's better to use plain Guard for the listener - especially since it already handles relative directories. It's a bit trickier if you want the Guardfile outside the project, but it's doable. So I actually think it's best to use Guard on both sides - Guard is a thin layer on top of Listen anyway and allows more flexibility with transforming paths if needed. Guard::Yield on both sides should do the job without needing to write plugins. I'd also consider putting BOTH the server and client functionality into a single Guard plugin. And it would be up to the user to decide which instance is the server (accepting changes) and the client (sending changes). Also, having a server on both sides means you can e.g. pause the listener remotely and do other cool stuff, like restart Guard with a new config if a file changes. If you like, wrap this up into a small repo using e.g. Awesome work! |
Great minds think alike.
Compared to the beast that emerged in the Celluloid implementation it's quite simple.
Yeah, but HTTP is sufficiently quick, especially compared to the rails/rspec startup time.
Makes sense for a session oriented protocol like TCP. Since HTTP is more transaction oriented (connections don't represent session state), restarting the receiving guard doesn't reall cause a problem. (excepting a downed guard that might miss a change notification, of course)
Ya I'm down to do the work to make this reusable for others.
Hmm. I'm on the fence about using Guard on the sending side. Something like a 'notify-webhook' gem would be useful as a general tool, but it's also simple and elegant for the sender side in a guard scenario. The user doesn't have to worry about setting up a Guardfile or if both configs are part of the same Guardfile having to detect if it is supposed to be the sender/host or receiver/guest, avoid doing any test running on the sender/host side. Plus, once the notify-webhook gem/binary is installed, it can just be run immediate without config wherever the user likes, eg:
or if not watching cwd
That's not to say that there isn't some value in using it within guard (like file matching you described) and we could make that possible too, but in terms of a developer being in the "I just need to get this running asap!" mental state, leaving the Guardfile mental complexity to what they are used to (the receiver side) and the quick direct notify-webhook gem binary could be simpler. thoughts? |
So much so, it makes discussions almost superfluous :D
I can't agree more. It's hard to justify using raw TCP nowadays.
I agreed. That's not important if you're using an editor to create changes.
Not for the sending itself. Listen vs Guard is like TCP vs HTTP in this case.
It's fine by me. Guard has some nice bonuses, though. It's possible to have a Guardfile config in the home directory, and just use that config in any directory. (No project-level Guardfile necessary).
I was thinking more along the lines of just running
I don't mind either way (Guard or not). I was thinking that it's easier to manage and update/change/test if both server and listener are handled by a single gem. So a user just does Also I think it's "easier" for users to follow a documentation where everything is in the README of one project. (Instead of telling them to install a different gem, etc.). Personally I think having 2 projects to manage is overhead that isn't worth it. You can always split a project into 2 parts if needed later. Or, the plugin can just have a bin script for the listener. (Like Listen used to). The idea is: after a project exists, people can send PRs for whatever they need. |
Referring to issue: celluloid/celluloid-io#74
Celluloid::IO got locking issues on Windows systems.
One of guard/listen use cases is to use it on a Windows system to efficiently monitor
(with kernel-support (gem wdm), without polling)
its local file system by remote clients using shared directories on it.
As proposed by celluloid/celluloid-io repository owner @tarcieri,
celloluid-io could be replaced with native code to resolve issues as guard/listen doesn't heavily rely on it.
The text was updated successfully, but these errors were encountered: