-
Notifications
You must be signed in to change notification settings - Fork 47
Development
This may need revision, ping me if you need clarification as I wrote it quickly.
- To work on Lute v3, you'll need at least Python 3.8 and pip. You'll probably want to use some kind of virtual environments; I use venv and so will write that out here.
- Note that GitHub CI tests Python versions 3.8 through 3.11, as we can't be sure what version of Python users have, so stay away from newer language features.
- Install MeCab according to your platform as described on the User Installation Page. This is required for passing unit tests irrespective of whether you are developing features relevant to the Japanese language.
- Clone as usual, then
# Check out the current development branch
git checkout develop
# Pull in submodules
git submodule init
git submodule update
Submodules:
- currently, only the language definitions, https://github.com/LuteOrg/lute-language-defs
- set up your virtual environment, install all dev dependencies from requirements-dev.txt, activate it:
python3.8 -m venv .venv
source .venv/bin/activate
# verify version
python --version
# Install dev and prod requirements using flit
pip install flit
flit install --only-deps --deps develop
# Install pre-commit hooks (optional, but recommended):
pre-commit install
deactivate
- Copy
lute/config/config.yml.example
tolute/config/config.yml
, making changes as you see fit.
If you're going to work on Lute, you're going to want to run unit tests. The unit tests are destructive, in that they wipe and reset the configured database.
To guard against mistakes, the DBNAME
in your config.yml must start with test_
, DATAPATH
must be set, and the ENV
must be dev
. This ensures that you won't accidentally run the tests against your real Lute data. I work with this by having two completely separate environments: one for dev work, and one for real Lute usage. My prod data (actual data) stays in the latter.
- Start lute up, ensure it's configured correctly
source .venv/bin/activate # if necessary
python -m lute.main
# Open web browser to http://localhost:5000
# ... work work work ...
# When done, Ctl-C then
deactivate
- Do initial run of all tests
Shut down your dev instance of Lute if it's running, and then run
inv full
to do a full pylint, test, acceptance, and playwright test run. This should complete without errors, as lute master and develop branch are always kept passing in CI.
You may/may not find the overview docs of Lute's architecture useful ... let me know.
Pre-commit hooks are installed with the pre-commit install
step, and are run on every commit. I find this useful, as it stops me from having to go back and clean up, but YMMV. You can skip a step, e.g.: SKIP=pylint git commit -m "Some non-lint-compliant commit."
Testing is done with pytest and pytest-bdd. Run them as usual:
pytest
pytest path/to/file.py
pytest -k filter_string
pytest -m somemark
pytest -s
pytest -vv
Lute3 uses Invoke to run tasks. Tasks are in tasks.py
. See inv --list
for commands.
Some useful tasks:
task | desc |
---|---|
inv start | start the app on a development Flask server in dev/debug mode |
inv lint | lint |
inv accept | start a running instance of the app server if needed, and run acceptance tests |
inv playwright | start a running instance of the app server if needed, and run playwright tests |
Database migrations are managed automatically when Lute starts up. See the DB schema README for notes about how the schema changes are managed.
DB migrations are stored in /lute/db/schema/migrations
. To create a script, run inv db.newscript <somesuffix>
, and edit the file to create a Sqlite-compliant change script. See the existing scripts for examples.
When doing releases, the baseline database gets recreated as described in Releases -- if you add a migration, you don't need to recreate the baseline.
Dependencies are managed with the pyproject.toml
file, and installations of the dependencies are done using flit
:
flit install --only-deps --deps develop
IMPORTANT: Make sure that your dependencies don't introduce other unexpected dependencies (e.g. that aren't under MIT licensing, that need compilers or build tools, etc), or other huge libraries!
Todos are in the code as comments, e.g. # TODO [<group name>:] detail
, <!-- TODO ... -->
.
inv todos
collects all of these in a simple report.
The "group name" is arbitrary: it's used to group things together, which is handy when several locations need to be changed as a single unit of work.
Notes for building and running a Docker container are at ../docker/README.com.
As of v3.4.0, Lute supports a simple "language parser plugin" capability. Set up your dev environment as explained on this page, and then head over to Developing language parser plugins.
This is much tougher than it needs to be ...
To find the correct path, first build and run the container, then connect to it, and find libmecab.so.2 like this:
$ docker exec -it lute_v3-lute-1 bash
root@cid/# which mecab
/usr/bin/mecab
root@cid:/# ldd /usr/bin/mecab
...
libmecab.so.2 => /lib/aarch64-linux-gnu/libmecab.so.2 (0x0000ffff9b540000)
Different platform architectures have this in different locations. :-/
Datatables is used for the term and book listings. See lute/static/vendor/datatables/README.md for notes about getting and updating it.
It appears that killing acceptance tests mid-run results in a zombie (?) python process that keeps a handle on the db, causing it to get locked in read-only mode.
I couldn't find a better way to kill this process than do a full machine restart. Sledgehammer approach that works.
Worning during run of tests with inv accept --exitfail
:
WARNING selenium.webdriver.common.selenium_manager:selenium_manager.py:139 The chromedriver version (118.0.5993.70) detected in PATH at /opt/homebrew/bin/chromedriver might not be compatible with the detected chrome version (119.0.6045.105); currently, chromedriver 119.0.6045.105 is recommended for chrome 119.*, so it is advised to delete the driver in PATH and retry
brew upgrade chromedriver
Then, on a Mac, have to "allow" it:
/opt/homebrew/bin/chromedriver --version
Will show message: "“chromedriver” can’t be opened because Apple cannot check it for malicious software." Click "Show in Finder", then in Finder, click "Open" and say "OK" when it can't be verified. Yes, this is a security risk.
This wiki is for developer documentation. User documentation is in the user manual
Thanks!