treemacs-git-mode not working

[CSRaghunandan]

Here is my treemacs config: https://github.com/CSRaghunandan/.emacs.d/blob/master/setup-files/setup-treemacs.el

emacs version: GNU Emacs 26.0.91 (build 1, x86_64-apple-darwin16.7.0, NS appkit-1504.83 Version 10.12.6 (Build 16G1212)) of 2018-02-26

treemacs version: 20180221.1229

After modifying or adding any new files, I'm not seeing any changes to faces for the modified/added files. Any idea why this is happening?

[CSRaghunandan]

Ok looks like I've to manually refresh the treemacs buffer for it to show git related modifications. I've treemacs-filewatch-mode enabled. But looks like that's not helping ?

[Alexander-Miller]

The answer is twofold:

• filewatch-mode only applies to nodes you expand after its activation. If you turn it on when treemacs is already open it's easiest to just kill and reopen treemacs.
• filewatch-mode will update when files are changed, which is enough to cover many, but not all use-cases. Treemacs cannot know about e.g. adding or committing files in git, since these actions don't make any changes to the actual files. For these a manual refresh is the only option, though for the future I want to ask the maintainers of magit to provide hooks that treemacs can use.

[CSRaghunandan]

filewatch-mode will update when files are changed, which is enough to cover many, but not all use-cases. Treemacs cannot know about e.g. adding or committing files in git, since these actions don't make any changes to the actual files. For these a manual refresh is the only option, though for the future I want to ask the maintainers of magit to provide hooks that treemacs can use.

Can use magit-post-refresh-hook to refresh treemacs. I've similar settings to update diff-hl in open buffer after running any magit command.

filewatch-mode only applies to nodes you expand after its activation. If you turn it on when treemacs is already open it's easiest to just kill and reopen treemacs.

Oh I didn't realize filewatch-mode works that way.

[Alexander-Miller]

Oh I didn't realize filewatch-mode works that way.

Makes things much easier on my part. I figured filewatch-mode is something you either turn on from the start or not at all.

Can use magit-post-refresh-hook to refresh treemacs.

That's a stopgap solution at best.

1. Treemacs knows how to run a local refresh for a single directory. With this hook I'd have no choice but to refresh everything.
2. magit-refresh runs many more times than treemacs requires refreshing. Depending on the size of your filetree and your hardware refreshing everything can incur a visible delay.

I'll get around this eventually, but first it requires a) magit's cooperation and b) me finishing my current project, which is teaching treemacs to display multiple file trees at once.

[Compro-Prasad]

You can keep a watch on the .git directory itself for any changes. Normal git status/log shouldn't mutate the state of the .git directory. Most of the other commands might change the .git directory. Doing a checkout means all files are changed along with the value of HEAD in .git, so you are still going to update the file tree.

[Alexander-Miller]

I ran a short test - just watched .git and checked what changes it produces. Adding an untracked file - one of the actions that treemacs cannot react to now - didn't do much other than produce a whole lot of of changes to index.lock. The index would also get locked every time I fire git status, so whenever you expand a directory node.

So for this use-case I could not detect the event I need to react to and only burdened myself with additional interference.

Doing a checkout means all files are changed along with the value of HEAD in .git

A checkout should not be an issue. Anything that changes actual files will be detected. The problem is when git does somethinh that does change the underlying file system. AFAIK that should be limited to adding untracked files and making commits.

[Compro-Prasad]

Should we really care about untracked files?

[Compro-Prasad]

What I can suggest is run a git status to collect untracked files when:

• Project is added to treemacs buffer
• Force refresh is done
• A new file is created

Watching for the whole project directory might be a thing which can help out in such situations but it will also count on the ignored files.

[Alexander-Miller]

Should we really care about untracked files? Yes, of course. Appropriately displaying ignored, changed, added, untracked or conflicted files is the entire point of git-mode. Watching for the whole project directory might be a thing which might also help out in such situations It sounds like you don't know how filewatch-mode works. Watching the entire project directory for filesystem changes - or, more accurately, the visible parts - is exactly what it does. Your three points are likewise just a subset of everything treemacs already pays attention to. The problem lies only in detecting those events that make no changes to the underlying file system. That's the part I'll need magit hooks for.

[Compro-Prasad]

What is the thing that doesn't make changes to the underlying file system but is important for us to watch?

[Alexander-Miller]

For example making a commit. That'll change a bunch of your files from being modified to being clean again.

[Compro-Prasad]

For that we can watch for changes in the .git directory.

[Alexander-Miller]

Not really. As already said above, adding an untracked file does nothing but produce a dozen index.lock changes (in addition to the general interference every time I fire git status). That's not a good indication for anything.

[Compro-Prasad]

Are we out of options?

[Alexander-Miller]

When I tackle this issue next I'll ask magit to provide the necessary hooks for treemacs to use. I've no good solution for directly working with the command line.

[Alexander-Miller]

I did some talking over at magit/magit#3496 and I now have enough material to close those gaps when using magit. It'll involve some non-trivial additions, so I'll be publishing a treemacs-magit package soon.

[Alexander-Miller]

I've created a new issue specifically for dealing with integrating magit: #223

[memeplex]

Hi, I'm seeing all sorts of weird behavior regarding the file watcher. My treemacs configuration is:

(use-package treemacs
  :config
  (treemacs-filewatch-mode 1)
  (treemacs-git-mode 'simple)
  :bind ("C-c t" . treemacs))
(use-package treemacs-magit
  :after (treemacs magit))

I created a simple repository with just one file. I repeatedly made changes to this single file and committed them to the repo. Each time I made a change it wasn't reflected in treemacs until I explicitly refreshed its buffer by pressing g. But after committing each change treemacs did automatically refresh itself. That is, it seems it's detecting the commit event but not the change after the commit.

Now, if I disable magit integration treemacs isn't auto-refreshed in any case. So maybe the case that was working before was just because of some magit event (no pun intended). Nevertheless, if I add a file to the directory from the terminal, it immediately gets reflected in treemacs, so some sync is indeed happening.

[memeplex]

Did a little debugging:

touch z
=>
1 -> (treemacs--filewatch-callback (16 created "/Users/carlos/Other/repo/z"))
1 <- treemacs--filewatch-callback: nil

echo foobar > z
=>
Nothing

rm z
=>
1 -> (treemacs--filewatch-callback (16 deleted "/Users/carlos/Other/repo/z"))
1 <- treemacs--filewatch-callback: nil
======================================================================
1 -> (treemacs--filewatch-callback (16 changed "/Users/carlos/Other/repo/.git"))
1 <- treemacs--filewatch-callback: nil

Tracing treemacs--process-file-events instead adds nothing new, echo foobar > z doesn't trigger it

[memeplex]

I did this simple experiment:

(defun my-watcher (event) (message "Not in my watch: %s" event))

(file-notify-add-watch "/tmp/some-dir" '(change) #'my-watcher)

and again:

touch /tmp/some-dir/x => Not in my watch: (18 created /tmp/some-dir/x)
echo dsds > /tmp/some-dir/x => Nothing
rm  /tmp/some-dir/x => Not in my watch: (18 deleted /tmp/some-dir/x)

Maybe it's a bug in emacs-28, I don't know, I'm on macOS Big Sur.

[memeplex]

Well, indeed it makes sense according to the docstring:

FLAGS is a list of conditions to set what will be watched for.  It can
include the following symbols:

  ‘change’           -- watch for file changes
  ‘attribute-change’ -- watch for file attributes changes, like
                        permissions or modification time

If FILE is a directory, ‘change’ watches for file creation or
deletion in that directory.  This does not work recursively.  <------

If I instead track "/tmp/some-dir/x" I do get:

Not in my watch: (20 changed /tmp/some-dir/x)

Maybe treemacs is not watching individual files? I don't know, I need to sleep now, I'll try to dig deeper later, HIH.

[memeplex]

Just one more thing, the docstring quoted above contradicts https://www.gnu.org/software/emacs/manual/html_node/elisp/File-Notifications.html:

If file is a directory, changes for all files in that directory will be notified. This does not work recursively.

[memeplex]

According to https://news.ycombinator.com/item?id=9063910 the difference I noticed is in fact a difference between kqueue (BSD/macOS) and inotify (Linux).

Also: https://stackoverflow.com/a/30019533/2012920

But this is the only disclaimer the elisp manual does wrt kqueue:

Likewise, the kqueue library does not report reliably file attribute changes when watching a directory.

So nothing about file changes. Either the feature is misdocumented or there is a bug. If there is a bug then there doesn't seem to be an easy cure, at least if we are to believe what was said in the ycombinator thread.

PS: I reported this to emacs bug tracker.

[memeplex]

Eli Zaretski answered that this should be considered a bug in macOS and not in emacs.

I believe it's not a bug in macOS but a characteristic of kqueue, although I agree that it's better to leave emacs alone (except that the documentation should still be fixed)

See for example this remark in the documentation of fswatch [1]:

The kqueue monitor requires a file descriptor to be opened for every
file being watched. As a result, this monitor scales badly with the
number of files being observed and may begin to misbehave as soon as
the fswatch process runs out of file descriptors. In this case,
fswatch dumps one error on standard error for every file that cannot
be opened so that users are notified and can take action, including
terminating the fswatch session. Beware that on some systems the
maximum number of file descriptors that can be opened by a process
is set to a very low value (values as low as 256 are not uncommon),
even if the operating system may allow a much larger value.

The implications are rather negative. fswatch also provides:

The FSEvents monitor, available only on Apple macOS, has no known limitations and scales very well with the number of files being observed. In fact, I observed no performance degradation when testing fswatch observing changes on a filesystem of 500 GB over long periods of time. This is the default monitor on Apple macOS.

But I don't see any implementation of file-notify-add-watch on top of FSEvents or fswatch coming anytime soon, so for treemacs the options regarding kqueue based implementations seem to be:

• Don't support the filewatch git extension for them.
• Individually track each file and document that this will scale badly for large projects.
• Require fswatch as a dependency and call it as a subprocess.

I'll keep you updated.

[Alexander-Miller]

Yesterday I got to learn that shells on macos are so bungled that a simple whatis ls lookup can take 5 seconds and now this. Good job, apple.

Anyway, I'll go with option number one. Filewatch is complex enough as it is, and I have plans to refactor some chunks anyway - I am not going to build a second implementation for an operating system I do not even use.

[memeplex]

Ok, no problem, just don't throw the baby with the bathwater, filewatch for file adding/removal works just fine, the integration with git is more problematic since it depends on content change notification, so I would just disable the git/magit modules.

Regarding Apple, sadly I'm a refugee after > 20 years in Linux. My last 3 years, since laptops started to ship with FHD screens and external monitors become UHD, have been so painful in Linux that I decided to stay away until Wayland gets its act together (maybe in 20 years, who knows, I'll probably be dead by then). ATM there is no real support for mixed/fractional scaling factors, except for xrandr hacks, and some video drivers are in bad shape. Not that I'm blaming anyone, hardware is complex, changes fast and there is no big money in the Linux desktop. At the time I also left Emacs because of the lack of support for HiDPI, hopefully the pgtk backend will fix that.

That said, in this case I think it's unfair to blame Apple because kqueue is a general BSD thing.

[Alexander-Miller]

Ok, no problem, just don't throw the baby with the bathwater, filewatch for file adding/removal works just fine

No throwing, the plan is to keep the status quo.

That said, in this case I think it's unfair to blame Apple because kqueue is a general BSD thing.

I'll direct some of my disappointment towards BSD then 🤷

[memeplex]

Ok, no problem, just don't throw the baby with the bathwater, filewatch for file adding/removal works just fine No throwing, the plan is to keep the status quo.

Ok, but please add a warning with red flashing lights in the README :)

[Alexander-Miller]

Good point. I'll add proper warnings tomorrow.

[Alexander-Miller]

Warnings are added now.

[survived]

Here's a little workaround that forcely refreshes treemacs after each file save. Dirty and inefficient, but kinda works. Should be placed in config.el.

(after! treemacs
  (defun treemacs--force-refresh ()
    "Forcely refreshes all the projects"
    (treemacs-run-in-every-buffer
       (treemacs-save-position
        (dolist (project (treemacs-workspace->projects workspace))
          (treemacs-project->refresh! project)))))
  (add-hook 'after-save-hook 'treemacs--force-refresh))

[Alexander-Miller]

Updating every single project on every save is quite expensive and can lead to stutters if treemacs is showing a lot of files. The much cheaper option is to only update the file that was changed:

(defun treemacs--force-git-update-current-file ()
  (-let [file (treemacs-canonical-path buffer-file-name)]
    (treemacs-run-in-every-buffer
     (when (treemacs-is-path file :in-workspace)
       (treemacs-update-single-file-git-state file)))))