Add a Scratch Buffer?

[countvajhula]

One workflow that I like is the idea of a "scratch buffer." That is, a buffer that you can quickly visit, put some code in and evaluate to see what happens. Many IDEs support something like this, including Emacs (the usual scratch buffer), Dreampie for Python, Groovy Console for Groovy, and of course, DrRacket's definitions window, where you can enter a bunch of code and then hit Run.

I have a similar workflow configured in Emacs, where I open a scratch buffer named *scratch - Racket* with racket-mode as the major mode. I can evaluate individual expressions in this buffer if I happen to have a REPL open, but what I'd like to be able to do is run this buffer without needing to save it to disk (the same way as with DrRacket). That way, I could just clear the contents upon visiting the buffer, and enter some fresh code in and have it evaluated from scratch each time, and being able to easily edit errors and re-run as needed. I find this helpful for quickly validating assumptions or trying things or initial prototyping without the "commitment" of having a saved file yet, and where I can treat it as a cohesive program (and not just individual expressions to edit with varying dynamic state as in the REPL). But at the moment, attempting to run this makeshift scratch buffer prompts to save the file -- which makes sense usually, but not for the UX I'm going for here.

Would it make sense to add a Scratch Buffer to Racket Mode? If you think so, some of the links above could provide some design ideas for iterating on this feature in Racket Mode - Dreampie in particular has some good ideas.

[greghendershott]

A couple initial reactions:

• If I understand you correctly, my personal version of this is C-x C-f and enter /var/tmp/foo.rkt.

  Although it is saved to file, that doesn't seem to bother me because it's in a "temporary box" I could clean up anytime. In any case, it's not cluttering my home dir, or forcing me to make a project subdir.

  (Note: On my system, /var/tmp survives reboots, which I like -- but if I preferred super transient I could use /tmp/ instead?)

  I think, for me, that hits the key points you mentioned. But I might be overlooking or misunderstanding some?

• As an implementation detail, I have a nagging feeling some things in Emacs, Racket, and/or Racket Mode will break in the case where no file exists.

  I know DrRacket allows run without save, but I also know this complicates some things. I recall discussing with Sam T-H some feature that was trivial to do in Racket Mode, but "impossible" in DrRacket because it can't assume a file exists; unfortunately I can't recall more details right now. 😞

  So even if there were a "scratch buffer" feature, I'd be inclined to implement it using a temp file behind the scenes, anyway.

[countvajhula]

Using /var/tmp/ or /tmp/ is a good idea! I have a similar workflow too, a dedicated racket/scratch folder. At the moment it contains about 70 files, and tellingly, most of them are numbered, e.g. 1.rkt ... 45.rkt, while a few have names like quine.rkt and qi-hash.rkt. Just like in other IDEs, I find I look for zero friction initially, and then, if it becomes something I want to keep around, I might save it with a real name. Some of these named files probably began as [N].rkt.

Unfortunately though, re: friction, the need to name scratch buffers is enough of an impediment that I use the REPL even for cases where a scratch buffer would be more appropriate, and then "graduate" to a saved scratch buffer when it becomes too inefficient. Even in this case, when I want to keep the file around, there's extra effort involved in renaming the N.rkt file to something else, and in most cases I end up not doing that even though that would be better (so I can find it later by scanning the filenames). I feel there is an in-between space here that would be ideally served by a runnable scratch buffer that could later be saved.

I know DrRacket allows run without save, but I also know this complicates some things. I recall discussing with Sam T-H some feature that was trivial to do in Racket Mode, but "impossible" in DrRacket because it can't assume a file exists; unfortunately I can't recall more details right now. 😞
So even if there were a "scratch buffer" feature, I'd be inclined to implement it using a temp file behind the scenes, anyway.

Interesting, yeah that makes sense. I suppose as long as the user can just start coding without needing to save a file and manage the names, that would still serve the purpose, if this feature were to be added. It would be ideal if the name of the file were completely abstracted somehow, though, as opposed to displaying as /var/tmp/cdef01a63.....rkt, but I wouldn't say that's crucial.

[greghendershott]

OK, I'll mull it over some more.

It would be ideal if the name of the file were completely abstracted somehow, though, as opposed to displaying as /var/tmp/cdef01a63.....rkt, but I wouldn't say that's crucial.

That shouldn't be a problem, in Emacs buffer-name and buffer-file-name may differ (and almost always do, trivially, e.g. the buffer name is just the base name not the full path name).

So I mean, I think you could sketch this out yourself (if you wanted) by (a) creating your *scratch - Racket* buffer as you already do, but also (b) setting its file name to some temp file somewhere, (c) maybe saving it even if empty to avoid some initial prompt about that, and ... just using that; I think that's it? If that's close to the workflow you want, then we could discuss any fit and finish details, and maybe making it official?

[countvajhula]

Sounds good, I'll take a crack at it and share when I have something worth looking at, and we can see from there.

[greghendershott]

One possible fit and finish detail: After creating such a buffer -- whose buffer-name is *scratch - Racket* and whose buffer-file-name is some temp file -- you could also in that buffer (setq-local racket-repl-buffer-name "*scratch - Racket REPL*"). In other words, you could have a REPL dedicated to the scratchpad.

This would be independent of whatever the user has chosen for racket-buffer-repl-name-function. That is, even if they've chosen to share one REPL among all buffers, or share one REPL per project, this scratch buffer would still get its own REPL.

Although I'm not sure if that's part of your vision of the workflow, I wanted to point out how to do it if you want.

(In my /var/tmp/foo.rkt style, and given my REPL-per-project setting, I like that it's a distinct REPL. Even if the emphasis is on running the file, and not so much on entering expressions at the REPL prompt, the REPL is where output goes, and I tend to like having a distinct REPL for that.)

[greghendershott]

Here's a draft implementation making some decisions (not necessarily ideal) about some of the details.

;; TODO: Change to `defucstom'.
(defvar racket-scratch-file-name "~/scratch.rkt")

(defun racket-scratch-buffer ()
  "Visit `racket-scratch-file-name' in a buffer named \"*scratch - Racket*\"."
  (interactive)
  (find-file racket-scratch-file-name)
  (rename-buffer "*scratch - Racket*")
  ;; When buffer is empty because file was empty or didn't exist,
  ;; insert a reasonable initial default. (Thereafter, the user's
  ;; edits to the file are preserved, e.g. they can change the #lang
  ;; and other contents as they wish.)
  (when (= (point-min) (point-max))
    (insert "#lang racket")
    (newline)
    (newline))
  (save-buffer)
  ;; Probably `racket-mode' due to .rkt extension, but to be sure:
  (racket-mode)
  ;; Ignore whatever `racket-repl-buffer-name-function' just did to
  ;; set `racket-repl-buffer-name' and give this its own REPL.
  (setq-local racket-repl-buffer-name "*scratch - Racket REPL*"))

[countvajhula]

@greghendershott Cool, thank you for the example! I did start on a prototype but got interrupted - will get back to it and try to post something, hopefully later today or tomorrow 🙂

[greghendershott]

Oh, yeah, it wasn't supposed to be a nudge. 😄 No rush from my POV. Just had some time to think about it today and wanted to share in case helpful.

[countvajhula]

Hey!
Okay, I felt inspired 😄 Here's what I have:

https://github.com/countvajhula/racket-scratch

This includes the following features:

• a Racket Mode scratch buffer
• user-definable templates, and the ability to create a new scratch buffer from any template (default #lang racket template included). Templates are just files in a designated (defcustom) folder.
• A notion of "sessions" - a session is a sequence of scratch buffer states (each reified as a file on disk - invisibly), demarcated by user invocation of "new" scratch buffer (which prompts for template to use)
• a model of scratch buffer "iteration" -- which occurs implicitly on each invocation of racket-run so all run-worthy states are navigable as a history, and also occurs explicitly on "clearing" the scratch buffer
• ability to save a scratch buffer to a (non-tmp) file
• ability to save a session to a (non-tmp) directory

WIP and planned features:

• ability to navigate sessions (e.g. previous/next scratch buffer state - these are implicitly demarcated by executing racket-run, but also explicitly demarcated by calling "Clear" which restores the scratch buffer to its initial template)
• ability to load sessions -- in principle the scratch buffer mechanisms are almost entirely filesystem-based so it should be easy to have them operate on any folder the user designates, which could be a saved session from an earlier time
• could it make sense to identify and model "sessions" as distinct git repos? That would give us versioning for free, with a thin scratch-buffer-level API wrapping Git facilities.

I haven't had a chance to thoroughly test it but I thought I'd share what I have for now. To use it, evaluate the racket-scratch.el buffer, and then (rackscratch-initialize) to set up the advice for racket-run-based implicit iteration. You should be able to do M-x rackscratch-new which would create the scratch buffer. Thereafter, running the buffer invisibly iterates the file name (in principle supporting navigation of the history, and the ability to modify any prior state of the scratch buffer at "natural" bounding points (i.e. racket-run invocations) - but this kind of navigation isn't fully implemented yet). There are other interactive commands that you could try, M-x rackscratch- is the prefix used.

So there you go! I realize there's a lot here, how would you prefer to proceed on this? Seems like we could either:

• work it into Racket Mode
• keep it as a separate "add-on" repo so it can be independently maintained and improved

Of these, integrating it into Racket Mode would have the advantage of it being immediately available to all Racket Mode users. Arguably, this is a drawback as well, in case not every user wants this feature. Benefit of the second approach is that improvements could be done in a decoupled way and there would be a little more flexibility for experimentation and possibly adding support for other Lisps. On the other hand, it could be less discoverable by not being bundled. A third and more speculative but possibly flexible option could be to structure racket-mode itself in a kind of lib/test/doc manner, where there's a racket-mode-core and other components and addons that, together, are provided in a distinct aggregate racket-mode package (which would correspond to the current Racket mode, except constituted from multiple decoupled packages, under the hood, and which could include the scratch buffer package as one of its components). Apologies for rambling, hope this makes some kind of sense. It's late here and I'm not sure I can tell!

What do you think?

[greghendershott]

This -- the combination of templates and versioning -- looks very cool!

One initial reaction is that it's not limited to Racket or Racket Mode. Conceptually. And also, implementation-wise: The code has almost no reference to racket-xxx functions or vars (modulo the racket-repl-buffer-name detail). Even the main "driver" to make new versions happens via you advising a command, which could be any command, not just racket-run.

It looks like you're already thinking this way, what with the major mode to use being a var?

So that leads me to think this should be its own Emacs package? Of utility to people using various programming languages? Or even authoring various non-program material, generally?

That's not my final answer, just my dominating initial thought.

[Even in that scenario, I'd be happy to help contributing to that package, if you wanted. And of course if you need some hook or API in Racket Mode, or the reverse (for Racket Mode to look for and use some hook in your thing), or whatever, would be happy to do that. And add it to "suggested packages for Racket Mode". And so on.]

[countvajhula]

That's a great point - I hadn't thought about it as being usable even for authoring in non-programming settings. That would probably take some additional design input to generalize it to that use case, but it's a good one to plan for, even if it starts out as no more and no less than a Racket scratch buffer 😄

I think I agree with you that it might be better as a separate package for this reason. But yes, with use with Racket Mode being its primary usecase, would be great to ensure that it's convenient to set up for Racket users, and of course I'd be glad to have contributions from you, if you feel so moved!

For now I'll plan to continue adding polish on an MVP for this and will touch base for any input or if any hooks are needed from Racket mode.

[countvajhula]

Hey! Just an update here - I think the package should be in working shape, if you're curious to try it. I've renamed it to "mindstream" (after a similar concept from buddhist philosophy) and added straight.el instructions in the README:

https://github.com/countvajhula/mindstream

There's one big change that's needed before I'd call it ready, though, and that's to base the versioning on git instead of managing it at the Emacs + naive filesystem level. I'm gonna have to take a break from this to do a Qi release though, so I'll probably return to it in a week or so 🙂

[countvajhula]

Hey @greghendershott !
I'm pleased to report that this is close to ready at last. Can't believe it's been almost two years since we first discussed it!

I took your advice to generalize it beyond Racket to any authoring setting -- text, markdown, scribble, whatever -- keeping it pretty lightweight so that most of the configuration would be done against the relevant major mode rather than for Mindstream specifically.

I think it's shaping up to be quite a handy little tool. I find myself using it all the time to write meeting notes each week for the Qi project, to try out code samples and iterate on small implementations, even to draft emails, and, in fact, this message!

If you have some time to try it out, I'd love to hear your feedback, and especially whether you think anything further could be done for it to integrate better with Racket Mode!

I've made a note about having a dedicated REPL for the session buffer in the README -- please let me know if you feel that doesn't cover it.

If we don't find anything terribly wrong at this point, I think we should be able to put it up on MELPA soon.

[greghendershott]

Nice!

I'll try to find some time to use it hands-on soon.

Meanwhile, quick comments from reading the README:

0. I love how you start off explaining why -- the motivation and use case. (I wish people did this more often.)

1. I wonder if a mindstream/templates subdir of user-emacs-directory would be a better default home for templates?

2. Likewise mindstream/tmp (or mindstream/anon or whatever) for anonymous sessions? Especially this would avoid the unknown whether the tmp dir persists across restarts.

[countvajhula]

Thank you, I've adopted those suggestions. One difficulty with the anonymous session path being in the Emacs folder is that it could become part of the containing .emacs.d repo, and it's a bit messy to add it to Gitignore and whatnot. That's still arguably better than /var/tmp as a default given the reliability concern. In any case, I've documented these considerations in the README. Let me know what you think.

[bestlem]

There are packages that allow you to have multiple scratch buffers and persist them between sessions e.g. persistent-scratch with examples of how to extend it in https://umarahmad.xyz/blog/quick-scratch-buffers/

[greghendershott]

@countvajhula The new README looks good.

As for the TODO comment about being platform independent, I think maybe you want to use expand-file-name instead of concat, and maybe just need to nest those??

[countvajhula]

@greghendershott Nice, looks like that'll do it!!

[countvajhula]

@bestlem Thanks for the pointer. Looking into that, I found a bunch of other packages related to scratch buffers!

Although there is some overlap, I would say that Mindstream is quite different from these packages.

Crucially, it's versioned, meaning that your entire history of changes is preserved. This gives rise to the organizing concept of a session. A session is a record of work on some topic or a snapshot of how you were thinking about a project at a certain time. For instance, a session could be you writing a blog post, or learning about a Racket library like megaparsack, or learning about macros, or solving a programming challenge. In addition to sessions providing features designed to help you be more efficient (like "Live mode"), over time, you'd amass a collection of such sessions that tend to be valuable as records, like old notebooks.

Now, the versioning provided by Mindstream isn't implemented in the package but is actually provided by Git! This does two things: (1) it keeps mindstream lightweight, and (2) it opens up a whole world of possibilities for harnessing the Git ecosystem and features, and plugins that already exist and which may be written in the future (e.g. Magit and Git-timemachine).

These things together make the package pretty handy for basic scratch buffers, but mean that there is potential for a lot more, and I feel we've only scratched the surface, so to speak :). If you have a moment, please try it out and let us know what you think.

[greghendershott]

As you know I've opened a couple issues on its repo. Will do that going forward.

As for this issue here: I think the actionable item with which to close it, would be to update any Racket Mode docs that discuss "recommended other packages". I'm happy to go ahead and do that pointing to your GitHub repo. Or, if you prefer I can wait and do this after it's available on MELPA. Just let me know what you prefer; not urgent.

[countvajhula]

Thank you so much for the thoughtful feedback so far! All that sounds good. I'll go ahead and submit a PR to add it to MELPA in the coming days and will keep you posted.

[countvajhula]

One last thing before I put it up on MELPA. I've been considering the name "freewrite" for the package. Mindstream is a cool name, but I wonder if "freewrite" conveys more specifically what you would use it for, without having to read the docs. This name perhaps loses the "session" aspect encoded in the present name, but arguably it's only present in retrospect rather than when you first encounter the name... if that makes any sense.

Googling "freewrite," I found a kickstarter for a device that has the same name. Quite a different thing and I'm not sure if that is a well-known device, but I wouldn't want users to feel that this package has something to do with that. Probably not something to worry about. But what do you think? Do you have any preferences on the name?

[greghendershott]

Like they say, naming things is hard.

I hear there's a new package to defer naming files:

Mindstream saves you the trouble of coming up with extraneous names (e.g. draft1.tex, draft2.tex, ..., draft_final2.tex, ...) and allows you to focus on the task at hand.

If only such a thing existed for naming packages. 😛

---

Seriously I'm of the school that a name needn't mean much. Almost any name is OK if most people can remember, spell, and pronounce it.

If the product/project succeeds, people will accept any name. If it doesn't succeed, the name won't matter.

Not to date myself but: "VisiCalc" was a descriptive name for a spreadsheet. Later, "Lotus 1-2-3" was a weird, no-meaning name -- and the product was a huge hit, and no one cared about the name. (Later they didn't care because they'd switched to "Excel".)

IMHO "Mindstream" is good. So would be the name of a pet, or a science fiction character, or anything really. If they need/like the project, people won't even "hear" the name after the first 5 minutes, it's just another label.

[greghendershott]

p.s. That's not to say I think there's anything wrong with "freewrite"; it's good, too!

Also I'm not dissing a creative pun, or use of Latin or French (e.g. some riff on "déjà vu"? like instead of "already seen", "already named/saved/xxx"). But in this case I'm just saying I'm not clever or patient enough to do so. :) Also some of those might run afoul of "tricky to spell and Google" for some of us.

[countvajhula]

Alright, "Mindstream" it is then, and I've submitted it to MELPA! Look at that, PR number 9000. Maybe we should call it "Mindstream 9000" 🤔 😆

[countvajhula]

Hey @greghendershott ! Any chance you can attend the Racket meetup tomorrow? I'm planning to give a short demo of Mindstream. Would be great to see you there!

[greghendershott]

Unfortunately I'm not free then. 😞

I hope the demo goes well! I'd love to watch a video of it later, if available.

Also look forward to meeting someday IRL or virtually.

[countvajhula]

@greghendershott Thank you, and yes that would be great!! I might make a trip out to Boston sometime, or at least, to Seattle for RacketCon this year. Come to RacketCon 😄

[countvajhula]

Did the demo today and it went well, I think! There was an inquiry about whether Mindstream could be used in other editors besides Emacs. Have you ever thought about writing cross-editor extensions? Not sure how feasible it would be.

[countvajhula]

Update: the MELPA PR was merged so this is "officially" out 😄