Skip to content

Commit

Permalink
Complete overhaul (separated Dockerfiles, Nuxt3, ...) (#29)
Browse files Browse the repository at this point in the history
* fix: package-lock.json

* feat: add permissions test

+ move start servers into index.js

* fix: re-added missing functions/logic
- added back things that went missing during the migration
- the docker-compose logs are now saved
- reordered variables, imports, ...
- specify the source of $fetch calls
- fixed the default 8px margin, see https://www.joshwcomeau.com/css/custom-css-reset/ for more CSS hacks
- tried to fix the emits, subject to change
- VJSF migration not 100% done yet
- probably fixed some ws bugs
- moved sources/ to ui/assets/sources/
- removed every require left
- ui/assets/format.js seems unused
- ui build now deletes the right folder
- fixed the middleware
- fixed some event bus oversights
- store update: added error(), changed the login (! this can be the source of some errors)

* fix: tests

+ bug : /api/v1/runs never return 403 error
+ bug : processings tests doesn't work

* fix: permission reQSession -> req

+ add logs

* fix: temporarily disabled the middleware (/admin/plugins error)

* fix: test 'create a new processing'

* fix: finished the migration ?
- the middleware now works
- fixed some migration issues
- uses navigateTo instead of router.push
- the error layout is now in ui/
- properly set the HEAD metadata
- moved all filters in 1 file
- _id.vue is now [id].vue
- properly instantiate the store

* fix: test config dir

* Merge nuxt3 into livrables

* fix: css

* feat: use @df/testSpies

* fix: config dir

* fix: test SIGTERM kill and stop worker at the end

* fix: remove hooks in worker server

* chore: remove eslint config from package.json

* fix: removed the middleware, better requests body, better error layout, some fixes in the store

* fix: package-lock.json

* feat: notification event hook -> test:spy

* fix: vuetify styles are now properly applied

* fix: removed @data-fair/sd-vue

* feat: remove ./test

* fix: eslint config

- fix some eslint errors
- create a gitignore for ui
- better nuxt.config.js

* fix: better card styles

* fix: uses the compat function from VJSF

* fix: ui Dockerfile

* fix: mongo url in prod

- When testing the API Docker image, connection to the database was not working.

* fix: worker Dockerfile

* fix: worker Dockerfile

* fix: small UI tweaks + fixed saving the state of a plugin (public or not + define access)

* feat: clean ui config

* fix: the forms are displayed
- fixed the vjsf usage
- added the missing @koumoul/vjsf-markdown
- simplified the npm run i command
- when you create a processing, the create button is disabled (spam-clicking it would create a ton of them) and displays a progress bar
- /admin now redirects to /admin/plugins
- probably fixed the v2compat usage
- the vjsf form is now compact by default
- the 'show all processings' button now have a way better style

* feat: add ts-config

* fix: no .cjs, the breadcrumbs now displays

* fix: better runs handling

- Co-authored-by: BatLeDev

* fix: the notifications card now shows properly on dark theme

* fix: the breadcrumbs style is now properly applied

* feat: better Vuetify instantiation
- removed the manual css import from jsdelivr
- changed the way the theme is accessed/edited on the store
- you can now switch theme without triggering a reload
- added vuetify, vite-plugin-vuetify
- removed vuetify-nuxt-module
- separated the vuetify instantiation from the session one in ui/plugins
- the locale messages should be now properly interpreted

* chore: added contributing guidelines, edited the readme, added some comments for non-trivial code and made the ui/sources files ESM-compatible

* fix: the error page now works fine

* chore: Small mistake in Docker instructions

* fix: better menus
- the menus now appears when the window is smaller, with a better color
- fixed a bug where the icons would be dull
- fixed all #activator templates
- simplified icons in most v-btn
- added the Show all processings in the small menu

* fix: observer and tmpDir config

* fix: config schema

* fix: pass breadcrumbs to parent, better processing card styles, docker compose's data-fair config changed

* feat: use data-fair/lib session and vuetify

* fix: publicUrl

* fix: small UI tweaks + vjsf 3.0.0-beta.5

* fix: main.scss

* fix: some types

* feat: add schema for processing

* fix: processings run action should be triggered + vjsf update

* feat: add typescript annotation

* fix: runs actually works
- fixed an issue with the api
- added a progressbar when the processing config is edited. so you know why you can't (yet) run it
- the webhook link is in code tags
- running and deleting a processing have a progressbar as well and disables their respective buttons
- fixed the Run processing button not emitting its trigger
- fixed the duration method
- the timestamps in runs are correctly displayed (sticked to the right)
- removed notify from the websocket

* chore: dev navbar

* fix: errors correctly redirect to the error page + user check in processings

* fix: add some types in api

* feat: add type for permission

* fix: some types errors

* fix: correctly passes values to $fetch and better plugins page
- update icon if the plugin can be updated
- better display of the version
- remove installed plugins from the available plugins list
- sorts plugins by alphabetical order

* fix: don't display an empty space under the plugins if there's no description

* fix: better plugins deletion
- don't prompt to run a processing with a deleted plugin
- show a confirmation prompt before deleting a plugin

* fix: processing menu now appears correctly on small screens + bump dayjs from 1.11.10 to 2.0.0-alpha.4
- see iamkun/dayjs#1281

* fix: websocket does not send data

* fix: loading bar and apiKey

* fix: better v-icon, better design on a processing's runs list, fixed build error and display the navbar only if we're not in an iframe

* fix: the loading circle on processing's page is have a correct padding and better text

* fix: task never stop

* feat: show a status icon on each run step, creating a plugin now allows typing its name

* fix: kill task with ui

* feat: plugin input is clearable, display the number of installed/available plugins

* fix: test failed

* fix: worker dependencies version

* feat: search processings
- fixed 'mobile' menu for creating a processing
- display in the processings list if one have been/is being killed

* fix: consistent style on search elements + fixed the processing search looking like a D when filled

* fix: type error

* fix: some types

* feat: display processings by run status

* feat: plugin & processings search doesn't require to select elements anymore
- processing's search & status works together
- fixed an issue with vjsf validate

* feat: add custom plugin name config

+ fix types in plugin and plugins-registry

* chore: clean plugin router

* fix: add types on create and get processings

* fix: patch processing

* chore: add comments is processings router

* fix: validate the form, debounce the input

* fix: avoid exceeding call stack

* fix: run processing with delay

+ add JSDoc ts annotation
+ add shared to Dockerfile

* feat: API documentation

* fix: notify now works again

* feat: complete plugins & limits doc

* style: add summary on api doc

* feat: complete processings part of API doc

* fix: update version show when the disTag is different

+ fix axios moderate dependency warning
+ fix name of plugin in plugin page

* fix: re-add privateDaraFairUrl & privateDirectoryUrl

* fix: dataFairUrl & directoryUrl

* fix: drop next run when scheduling type is 'trigger'

+ move createNext function to shared
+ add ts annotation

* fix: add ts anotation

+ add 404 error when plugin not found
+ merge master in nuxt3

* feat: sort processings by planned, loading indicator when installing a plugin

* fix: avoid concurrent patch

* fix: end task properly

* fix: avoid concurrent patch (actually)

* feat: sort processings by plugin used (compatible with custom names and version agnostic)

* chore : rename reqSession by sessionState

+ add department support on sd

* chore: nuxt update to v3.11
- also updated all UI dependencies
- uses the proper Nuxt3 components naming convention
- uses the new Client Component feature from Nuxt 3.11.0
- deleted unused components
- stops explicitely importing component and instead rely on Nuxt's Components dynamic import

* fix: readd secondaryHost

* fix: better processings sort by plugin
- now also works when a plugin have been deleted
- stops renaming the customName (made at first to remove the version but this also would remove everything in parenthesis on custom names)

* chore: add better types for mongodb

* fix: mongo db starting error

* chore: added types annotations to all components

* feat: add a query param into plugin-registry

This param 'showAll' allow to get all plugins and their versions (distTag)

* fix: req.param -> req.query

+ better tests for plugin-registry

* feat: implement b50459f on UI's side

* fix: test permissions, admin1 doesn't exist

* chore: add better types in worker

* fix: processing's config timezone now works as expected
- killing a run now works instantly
- ui's build noww works again
- the show all option on plugins page have a more logical placement
- processing's config now validates properly again

* fix: test build images

* fix: better patch with vjsf beta 9

* feat: added skeleton loaders
- also fixed the Notifications snackbar

* fix: data folder on build docker images

* fix: runs correctly on build + notifications fix
- added NODE_ENV as an ARG so we can provide it on build
- removed USER node, as it won't work with volumes (created with root, containers then got no access)
- the notifications component is now readable on mobile
- fixed the skeleton loader having the wrong check on processings page

* chore: removed pinia leftovers

* feat: refacto of plugin.vue

Visual Changes:
- Refined the styling in index.vue for processings: the sidebars are now more compact.
- Enhanced the appearance of search bars in both processings and plugins views to be more square-shaped, similar to the dataset page.
- Improved the display of Skeleton loaders in plugins.vue.

Functional Changes:
- Enabled the ability to install multiple plugins simultaneously. (Very convenient 😍)
- Enhanced the v-menu display for deletion

Pending Fixes and Enhancements:
- Fix the issue with the 'No' button not dismissing the v-menu.
- Add try-catch blocks for both installation and deletion to handle errors gracefully.
- Implement notifications in the catch blocks.

* fix: hide delete plugin menu, mobile button is fixed and the owner avatar shows

* chore: added NPM_HTTPS_PROXY env var related stuff from master

* fix: readd v-iframe plugin and add better v-card style

* chore: change plugin filter by composable date-format

* chore: text update

* feat: get processings list with facets (statuses) and filter by statuses

* fix: readded skeleton loaders on processings list, removed the progress bar, fixed the patch

* fix: correct height in data-fair iframe

* feat: add facets and filters on plugins in processings endpoint

* fix: sort filters

* feat: filters displays in the URL

* Added new params to the API doc

* chore: small menu issue

* feat: better aggregation in plugin endpoint for facets

* fix: better minimum iframe height and passes url values to the mobile menu

* fix: better skeleton loader display on light theme

* feat: add facets count nb processings per plugins in plugin endpoint

* feat: can't uninstall a plugin if it's used by > 0 processings, displays that info too

* chore: update the API docs (facets in GET /plugins)

* fix: permissions on department

* chore: better description in PrivateAccess

* fix: renamed showAll to showAllPlugins on the plugins page to avoid conflict with showAll on processings page

* chore: add mongodb indexes

* chore: make tests work

* chore: configure husky

* fix: types of plugins-registry.js

* chore: use jsdoc linter + small implementation details

* fix: wrong root routing

* refactor: work on many small things

* fix: missing prefix path in tests

* chore: work on CI

* chore: default config values

* chore: docker on UI dockerfile

* chore: work on dockerfiles

* chore: some dependency management

* chore: some dependency management

* chore: some dependency management

* chore: some dependency management

* chore: some dependency management

* fix: broken use of tmp-promise

* fix: some regression due to dependency upgrades

* chore: some dependency management

* chore: some dependency management

* chore: some dependency management

* chore: better error display

* fix: missing error management in UI

* chore: better run requests in sequence

* chore: send notif to parent iframe

* fix: wrong var name

* fix: error when access file is missing

* fix: some reactivity problems

* fix: wrong data dir in dev env

* chore: various small changes

* fix: a errors with session and navigation

* fix: missing dep

* fix: missing dep

* fix: prevent null reference error

* chore: use privateNotifyUrl config variable

* fix: some vjsf error management problems

* fix: remove additional properties in config

* fix: regression in using secretKeys

* chore: remove file that should be ignored

* chore: ignore local configuration files

* fix: broken unset logic

* fix: configuring iframe-resizer doesn't work as a nuxt plugin

* fix: try another way to configure iframe-resizer

* fix: try another way to configure iframe-resizer

* fix: try another way to configure iframe-resizer

* fix: problem in vjsf wait for correct form validity

* fix: missing customName in older plugins

* fix: pull fixes from vjsf

* fix: pull fixes from vjsf

* fix: labels of buttons on permissions list

* fix: broken contrib access

* fix: missing python to install native deps

* chore: small forms improvements

* chore: small forms improvements

* chore: small forms improvements

* 2.0.0

* chore: temporarily activate quality checks on commit to test workflow

* chore: add build-types script to CI

* Revert "chore: temporarily activate quality checks on commit to test workflow"

This reverts commit c3febff.

* 2.0.1

* fix: npm config cannot run inside a workspace

* 2.0.2

* fix: broken us of reactive search params

* refactor: work on reactivity of processings page

* chore: adapt to reactive search params new composables

* chore: small changes

* fix: disable autocomplet of password fields

* fix: missing config for tmp dir

* fix: broken breadcrumb

* chore: use axios retry for more resilient http requests

* feat: add tippecanoe to worker image

* fix: some problems in debug mode and error management

* 2.0.3

* fix: prevent null reference exception

* 2.0.4

* fix: axios error interceptor incompatible with axios retry

* fix: axios error interceptor incompatible with axios retry

* fix: missing axios-retry in worker image

* 2.0.5

* fix: wrong mails transport config structure

* fix: upgrade vjsf

* fix: upgrade vjsf

* fix: broken edition and application of timezones

* fix: upgrade vjsf

* fix: missing native deps in worker image

* refactor: use locks from @data-fair/lib

---------

Co-authored-by: BatLeDev <[email protected]>
Co-authored-by: Alban Mouton <[email protected]>
  • Loading branch information
3 people authored Jun 17, 2024
1 parent a857827 commit a061639
Show file tree
Hide file tree
Showing 204 changed files with 21,426 additions and 26,134 deletions.
3 changes: 2 additions & 1 deletion .dockerignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
data/
node_modules
.nuxt
.output
data
2 changes: 2 additions & 0 deletions .eslintignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
node_modules
.type
18 changes: 18 additions & 0 deletions .eslintrc.cjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
module.exports = {
root: true,
env: {
node: true
},
extends: [
'standard',
'plugin:jsdoc/recommended-typescript-flavor-error'
],
plugins: ['jsdoc'],
rules: {
'jsdoc/require-param-description': 0,
'jsdoc/require-property-description': 0,
'jsdoc/require-jsdoc': 0,
'jsdoc/require-returns': 0,
'jsdoc/require-returns-description': 0
}
}
76 changes: 0 additions & 76 deletions .github/workflows/build.yml

This file was deleted.

23 changes: 23 additions & 0 deletions .github/workflows/commits.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# no need to run quality checks on every small commit, husky already did it on the dev computer
# in this case we simply build the docker image

name: Build simple commit

on:
push:
branches: ['*']
tags-ignore: [ 'v*.*.*' ]

jobs:
build:
uses: ./.github/workflows/reuse-build.yml

deploy:
needs: build
runs-on: ubuntu-latest
steps:
- name: Send Keel notifications to staging-koumoul.com
run: |
curl -s --fail --show-error -X POST https://keel.admin.staging-koumoul.com/v1/webhooks/native -u ${{ secrets.KEEL_STAGING_USER }}:${{ secrets.KEEL_STAGING_PASSWORD }} -d '{"name": "ghcr.io/${{ github.repository }}/api", "tag": "${{ github.ref_name }}"}'
curl -s --fail --show-error -X POST https://keel.admin.staging-koumoul.com/v1/webhooks/native -u ${{ secrets.KEEL_STAGING_USER }}:${{ secrets.KEEL_STAGING_PASSWORD }} -d '{"name": "ghcr.io/${{ github.repository }}/ui", "tag": "${{ github.ref_name }}"}'
curl -s --fail --show-error -X POST https://keel.admin.staging-koumoul.com/v1/webhooks/native -u ${{ secrets.KEEL_STAGING_USER }}:${{ secrets.KEEL_STAGING_PASSWORD }} -d '{"name": "ghcr.io/${{ github.repository }}/worker", "tag": "${{ github.ref_name }}"}'
9 changes: 9 additions & 0 deletions .github/workflows/pull-requests.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# for pull requests we do not build a docker image, we simply run quality checks

name: Check pull requests

on: pull_request

jobs:
quality:
uses: ./.github/workflows/reuse-quality.yml
29 changes: 29 additions & 0 deletions .github/workflows/releases.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# for releases we run in sequence the quality checks and the build

name: Build release

on:
push:
tags: [ 'v*.*.*' ]

jobs:
quality:
uses: ./.github/workflows/reuse-quality.yml

build:
needs: quality
uses: ./.github/workflows/reuse-build.yml

deploy:
needs: build
runs-on: ubuntu-latest
steps:
- id: get_version
uses: battila7/get-version-action@v2

- name: Send Keel notifications to koumoul.com
run: |
curl -s --fail --show-error -X POST https://keel.admin.koumoul.com/v1/webhooks/native -u ${{ secrets.KEEL_PROD_USER }}:${{ secrets.KEEL_PROD_PASSWORD }} -d '{"name": "ghcr.io/${{ github.repository }}/api", "tag": "${{ steps.get_version.outputs.major }}"}'
curl -s --fail --show-error -X POST https://keel.admin.koumoul.com/v1/webhooks/native -u ${{ secrets.KEEL_PROD_USER }}:${{ secrets.KEEL_PROD_PASSWORD }} -d '{"name": "ghcr.io/${{ github.repository }}/ui", "tag": "${{ steps.get_version.outputs.major }}"}'
curl -s --fail --show-error -X POST https://keel.admin.koumoul.com/v1/webhooks/native -u ${{ secrets.KEEL_PROD_USER }}:${{ secrets.KEEL_PROD_PASSWORD }} -d '{"name": "ghcr.io/${{ github.repository }}/worker", "tag": "${{ steps.get_version.outputs.major }}"}'
58 changes: 58 additions & 0 deletions .github/workflows/reuse-build-image.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
name: Build

on:
workflow_call:
inputs:
workspace:
required: true
type: string

jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Checkout git repository
uses: actions/checkout@v4

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3

- name: Login to Github container registry
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Prepare BUILD.json
run: |
echo '{"version": "${{github.ref_name}}", "repository": "${{github.server_url}}/${{github.repository}}", "commit": "${{github.sha}}"}' > BUILD.json
cat BUILD.json
- name: Prepare docker image tags
id: docker_meta
uses: docker/metadata-action@v5
with:
images: ghcr.io/${{ github.repository }}/${{ inputs.workspace}}
tags: |
type=ref,event=branch
type=ref,event=pr
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}}
type=semver,pattern={{major}}
- name: Build and push api docker images
id: docker_build
uses: docker/build-push-action@v5
with:
context: .
file: ${{ inputs.workspace}}/Dockerfile
push: ${{ github.event_name != 'pull_request' }}
tags: ${{ steps.docker_meta.outputs.tags }}
platforms: linux/amd64
labels: ${{ steps.docker_meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max

- name: Image digest
run: echo ${{ steps.docker_build.outputs.digest }}
20 changes: 20 additions & 0 deletions .github/workflows/reuse-build.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
name: Build

on:
workflow_call:

jobs:
build-api:
uses: ./.github/workflows/reuse-build-image.yml
with:
workspace: api

build-worker:
uses: ./.github/workflows/reuse-build-image.yml
with:
workspace: worker

build-ui:
uses: ./.github/workflows/reuse-build-image.yml
with:
workspace: ui
25 changes: 25 additions & 0 deletions .github/workflows/reuse-quality.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
name: Quality checks

on:
workflow_call:

jobs:
quality:
runs-on: ubuntu-latest
steps:
- name: Checkout git repository
uses: actions/checkout@v4

- uses: actions/setup-node@v3
with:
node-version: 20
cache: 'npm'

- name: Install dependencies
run: npm ci

- name: Install dependencies
run: npm run build-types

- name: Run quality checks
run: npm run quality
12 changes: 5 additions & 7 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,9 +1,7 @@
data/
node_modules/
.nuxt
config/local-*
/nuxt-dist
/dist

public/assets/css/
public/assets/fonts/
worker/config/local-*
api/config/local-*
.config/
.type/
/test-*
4 changes: 4 additions & 0 deletions .husky/commit-msg
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
#!/usr/bin/env sh
. "$(dirname -- "$0")/_/husky.sh"

npx --no-install commitlint --edit ""
4 changes: 4 additions & 0 deletions .husky/pre-commit
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
#!/usr/bin/env sh
. "$(dirname -- "$0")/_/husky.sh"

npm run lint
4 changes: 4 additions & 0 deletions .husky/pre-push
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
#!/usr/bin/env sh
. "$(dirname -- "$0")/_/husky.sh"

npm run quality
2 changes: 1 addition & 1 deletion .nvmrc
Original file line number Diff line number Diff line change
@@ -1 +1 @@
v20
20
31 changes: 31 additions & 0 deletions .zellij.kdl
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
layout {
pane {
split_direction "vertical"
pane name="processings" borderless=true {
command "${DEV_SHELL}"
args "-ic" "nvm use > /dev/null 2>&1 && ${DEV_SHELL}"
}
pane name="deps" {
command "${DEV_SHELL}"
args "-ic" "nvm use > /dev/null 2>&1 && npm run dev-deps && mkdir -p data; touch data/zellij-deps-ready && docker compose logs -f"
}
}
pane {
split_direction "vertical"
pane name="ui" {
command "${DEV_SHELL}"
args "-ic" "nvm use > /dev/null 2>&1 && echo 'waiting for deps to start...' && while [ ! -f data/zellij-deps-ready ]; do sleep 1; done && npm -w ui run dev"
}
pane name="api" {
command "${DEV_SHELL}"
args "-ic" "nvm use > /dev/null 2>&1 && echo 'waiting for deps to start...' && while [ ! -f data/zellij-deps-ready ]; do sleep 1; done && npm -w api run dev"
}
pane name="worker" {
command "${DEV_SHELL}"
args "-ic" "nvm use > /dev/null 2>&1 && echo 'waiting for deps to start...' && while [ ! -f data/zellij-deps-ready ]; do sleep 1; done && npm -w worker run dev"
}
}
pane size=2 borderless=true {
plugin location="zellij:status-bar"
}
}
Loading

0 comments on commit a061639

Please sign in to comment.