Atomic Data

[fricklerhandwerk]

Atomic Data is a modular specification for sharing, modifying and modeling graph data. It combines the ease of use of JSON, the connectivity of RDF (linked data) and the reliability of type-safety.

• NLnet project
  • source code:
    • https://github.com/atomicdata-dev/atomic-server
    • https://github.com/atomicdata-dev/atomic-data-browser

This project is actively developed, and the lead is interested in collaboration. We will figure out the best way to approach this, and it may involve some sort of onboarding if an upstream Nix setup is desired.

[joepio]

Hi! Awesome that you want to help out :)

I'll first make sure both Atomic-Server and Atomic-Data-Browser codebases are combined into one monorepo.

[fricklerhandwerk]

We can start with packaging the Rust backend, and then continue with the Typescript frontend when it's merged. Then testing both components in integration as well as building a module to run the service on NixOS would be possible.

[joepio]

It's merged! It's now one monorepo. Check the dockerfile for how it's currently built. Let me know if I can help in any way!

Thank you :)

[albertchae]

@ngi-nix/moss is claiming this. We will ask you about any questions we have about the project as we learn more about it @joepio !

[albertchae]

Brainstorming a plan of action (not necessarily in order):

• get a running instance of atomic data locally outside of nix – this was very helpful for debugging flarum issues
• potentially packaging pnpm with nix is difficult, @Chickensoupwithrice had some experience with this
• look into packaging rust separately and first (as suggested in Atomic Data #15 (comment)), hopefully it is decoupled enough
  • Looking through the repo, we think packaging https://github.com/atomicdata-dev/atomic-server/tree/develop/lib / https://crates.io/crates/atomic_lib is the first step

[albertchae]

Summarizing where we are right now:
https://github.com/atomicdata-dev/atomic-server has 5 components

• lib is a Rust library so will be compiled from source by components that need it, so it doesn't need to be packaged separately
• server and cli have been packaged in Init: {atomic-server, atomic-cli} at 0.34.5 #62
• We are currently working on browser (and desktop depends on this via tauri).

The current challenge with browser is that dependencies are packaged via pnpm, which has poor support for Nix at the moment (https://github.com/nix-community/pnpm2nix is old and outdated, at minimum we need to use this fork for compatibility with the latest version of pnpm nix-community/pnpm2nix#26). One of the issues with pnpm2nix is that it expects a version key in package.json. We have had trouble trying to patch this while building in Nix, so we have submitted a PR to add this in atomicdata-dev/atomic-server#673

Another approach we are exploring is converting pnpm-lock.yaml into an equivalent package-lock.json (npm) or yarn.lock.

• In theory, pnpm, npm, yarn etc all have custom lock file formats, but produce compatible node_modules directory structures. So one approach was to use pnpm to install a valid set of packages, then reverse engineer a yarn or npm lock file out of that.
  • https://github.com/orgs/pnpm/discussions/3367#discussioncomment-2919171
    • This suggests using npm i --package-lock-only, but apparently this is incompatible with pnpm workspaces

    • npm ERR! code EUNSUPPORTEDPROTOCOL
      npm ERR! Unsupported URL Type "workspace:": workspace:*
      

  • Similar discussion in yarn
    • Allow generating a lock file from an existing node_modules yarnpkg/yarn#682
    • Bootstrap lockfile with current modules. yarnpkg/yarn#1265
    • Add import command yarnpkg/yarn#2580
    • https://classic.yarnpkg.com/en/docs/cli/import
    • Unfortunately, running yarn import gives us

      yarn import v1.22.19
      info creating yarn.lock from local node_modules folder
      error "eslint-plugin-import#array-includes" not installed
      ... several similar errors
      

• While searching through nixpkgs for pnpm references, we came across https://github.com/adamcstephens/pnpm-lock-export which is packaged in https://github.com/NixOS/nixpkgs/tree/master/pkgs/development/web/pnpm-lock-export. This tool converts a pnpm-lock.yaml to either package-lock.json or yarn.lock directly without actually installing the packages into node_modules first.
  • Ran nix run nixpkgs#pnpm-lock-export -- --schema yarn.lock@v1 in the browser directory of atomic-server to get a yarn.lock. We are still learning how to get this to work with the mkYarnPackage builder in nixpkgs, but this seems promising so far.

@joepio we also noticed there is a bun.lockb checked in. Is bun an "officially" supported way to install dependencies for browser too and is it kept in sync with pnpm-lock.yaml? Because according to https://bun.sh/docs/install/lockfile, we can have bun additionally export a yarn.lock using bun install --yarn, which could be easier than using the pnpm-lock-export tool.

[albertchae]

@joepio another question, for running browser, do we only need the artifacts in data-browser/dist or do we need to copy over various node_modules as well? I couldn't tell if dependencies were getting bundled into the artifacts of data-browser/dist, but based on the Dockerfile of the current monorepo AND the build script from the old repo, it looks like that's all we need.

[joepio]

@joepio another question, for running browser, do we only need the artifacts in data-browser/dist or do we need to copy over various node_modules as well?

yes! Only the dist folder is needed to compile atomic-server

Is bun an "officially" supported way to install dependencies for browser too and is it kept in sync with pnpm-lock.yaml?

Hmm... No, not yet, but as soon as Bun Test supports the features I need (e.g .toHaveBeenCalledWith) I'll probably migrate. I don't expect this to take very long. So I don't know if it makes more sense for you to assume the bun scenario.

Also, perhaps interesting - I spent some time last weekend setting up an earthfile. Could be useful inspiration?

[jleightcap]

Summarizing some recent progress:

• we managed to compile the pnpm lockfiles into a yarn.lock we have committed.
• we can verify that Nix pulls in these dependencies in an "offline cache"
• when building the three sub-components in turn (lib > react > data-browser), we've noticed that the yarn.lock we've generated is not the union of all required packages.
• the top-level package.json does not contain all dependencies declared in lib package.json (e.g. whatwg-fetch, cross-fetch).
• the first error hit is the tsc binary not being in scope, which means we're successfully working around not having a pnpm builder.
• however, we believe that we need to generate a lockfile per-subdependency, not just a single lockfile from the main package.json.

[fricklerhandwerk]

@joepio is migrating to Bun still an option for you? This would likely make it easier on our end to finish packaging, otherwise we'll have to make a detour of creating tooling to deal with pnpm.

[joepio]

@fricklerhandwerk Hmm looks like bun now has support for the testing methods so maybe it can replce pnpm! I'll work on it today.

Related

[joepio]

WIP PR

[albertchae]

Reviving this issue!

Recap

@ngi-nix/moss picked this up towards the end of Summer of Nix 2023 and we were not able to finish packaging it by the end of the program. Sorry for not posting a final status update at the end.

We were still stuck on figuring out how to get pnpm + nix working. We wanted to consider bun as an alternative but it looks like it was rejected in the upstream project atomicdata-dev/atomic-server#806

Now

@Chickensoupwithrice and I picked this issue back up as part of our extension work.

First we updated the existing packaging to the latest releases from the upstream project
#162
#163

There seems to have been more activity with getting pnpm to work with nix and we should be able to use it directly instead of transforming it to npm or yarn as we were investigating before .

nix-community/pnpm2nix#26 (comment) leads us to

• https://github.com/nzbr/pnpm2nix-nzbr
• Add nixpkgs tooling for pnpm-lock.yaml NixOS/nixpkgs#231513
• https://github.com/NixOS/nixpkgs/blob/6b6cabaffd3f322466d45f08707b3cc05398e3f3/pkgs/applications/misc/pot/default.nix
• pnpm.fetchDeps: init NixOS/nixpkgs#290715

We decided to explore using https://github.com/nzbr/pnpm2nix-nzbr. We are making good progress so far and hope to report more soon.