fs2 update (#47)

* fs2 update * mutox with new fs2 * ff * sonar setup * black * lint version * ff * py3.10 in lint * ff * updated version * ff * ff * ff * fairseq2 0.3.0rc1 * fairseq2n source maybe? * move to uv for deps on fairseq2 + lint/format * pin torchaudio * update README * update mypy ci * fix mypy2 * mypy 3 * fix mypy --------- Co-authored-by: artyomko <[email protected]> Co-authored-by: Pierre Andrews <[email protected]>
facebookresearch · Dec 11, 2024 · 549d287 · 549d287
1 parent a914bbd
commit 549d287
Show file tree

Hide file tree

Showing 46 changed files with 3,675 additions and 612 deletions.
diff --git a/.github/actions/setup/action.yaml b/.github/actions/setup/action.yaml
@@ -0,0 +1,11 @@
+runs:
+  using: composite
+  steps:
+    - name: "Install UV"
+      shell: bash
+      run: |
+        curl -LsSf https://astral.sh/uv/install.sh | sh
+    - name: "Install libsndfile"
+      shell: bash
+      run: |
+        sudo apt-get install libsndfile1 python3-distutils
diff --git a/.github/workflows/lint_and_test.yml b/.github/workflows/lint_and_test.yml
@@ -1,123 +1,52 @@
-# This workflow will install Python dependencies, run tests and lint with a variety of Python versions
-# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python
-
-name: Python package
+name: Lint Python Code
 
 on:
+  # Trigger the workflow on push to master or any pull request
   push:
-    branches: ["main"]
   pull_request:
-    branches: ["main"]
+    branches:
+      - main
 
 jobs:
-  linter:
+  lock_file:
     runs-on: ubuntu-latest
-    strategy:
-      fail-fast: false
-      matrix:
-        python-version: ["3.8"]
-
     steps:
-      - uses: actions/checkout@v3
-      - name: Set up Python ${{ matrix.python-version }}
-        uses: actions/setup-python@v3
-        with:
-          python-version: ${{ matrix.python-version }}
-          cache: "pip"
-      - name: Install dependencies
-        run: |
-          sudo apt-get install libsndfile1
-          python -m pip install --upgrade pip
-          pip install -r requirements.txt
-          pip install -r requirements-dev.txt
-      - name: Lint with flake8
-        run: |
-          # stop the build if there are Python syntax errors or undefined names
-          flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
-          # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
-          flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
-      - name: isort
-        run: isort --check --diff .
-      - name: black
-        run: black --check --diff .
-
-  mypy:
+      - uses: actions/checkout@v4
+      - uses: ./.github/actions/setup
+      - run: uv lock --locked
+  linting:
     runs-on: ubuntu-latest
-    strategy:
-      fail-fast: false
-      matrix:
-        python-version: ["3.8"]
+    needs: [lock_file]
     steps:
-      - uses: actions/checkout@v3
-      - name: Set up Python ${{ matrix.python-version }}
-        uses: actions/setup-python@v3
-        with:
-          python-version: ${{ matrix.python-version }}
-          cache: "pip"
-      - name: Install dependencies
-        run: |
-          sudo apt-get install libsndfile1
-          python -m pip install --upgrade pip
-          pip install -r requirements.txt
-          pip install -r requirements-dev.txt
-      - name: mypy
-        run: mypy --install-types --non-interactive ./ --cache-dir=.mypy_cache/
-
-  unit_test:
+      - uses: actions/checkout@v4
+      - uses: ./.github/actions/setup
+      - run: uvx ruff check .
+  formatting:
     runs-on: ubuntu-latest
-    timeout-minutes: 20
-    strategy:
-      fail-fast: false
-      matrix:
-        python-version: ["3.8"]
+    needs: [lock_file]
     steps:
-      - uses: actions/checkout@v3
-      - name: Set up Python ${{ matrix.python-version }}
-        uses: actions/setup-python@v3
-        with:
-          python-version: ${{ matrix.python-version }}
-          cache: "pip"
-      - name: Install dependencies
-        run: |
-          sudo apt-get install libsndfile1
-          pip install -r requirements.txt
-          pip install -r requirements-dev.txt
-      - name: pytest_unit
-        run: pytest -s -v tests/unit_tests/
-
-  integration_test:
+      - uses: actions/checkout@v4
+      - uses: ./.github/actions/setup
+      - run: uvx ruff check --select I .
+      - run: uvx ruff format --check .
+  type_consistency:
     runs-on: ubuntu-latest
-    strategy:
-      fail-fast: false
-      matrix:
-        python-version: ["3.8"]
-    timeout-minutes: 40
+    needs: [lock_file]
     steps:
-      - uses: actions/checkout@v3
-      - name: Set up Python ${{ matrix.python-version }}
-        uses: actions/setup-python@v3
-        with:
-          python-version: ${{ matrix.python-version }}
-          cache: "pip"
-      - name: Install dependencies
-        run: |
-          sudo apt-get install libsndfile1
-          python -m pip install --upgrade pip
-          pip install -r requirements.txt
-          pip install -r requirements-dev.txt
-      - name: Free Disk Space (Ubuntu)
-        uses: jlumbroso/free-disk-space@main
-        with:
-          # this might remove tools that are actually needed,
-          # if set to "true" but frees about 6 GB
-          tool-cache: false
-
-          # all of these default to true, but feel free to set to
-          # "false" if necessary for your workflow
-          android: true
-          dotnet: true
-          haskell: true
-          large-packages: true
-          docker-images: true
-      - name: pytest_integration
-        run: pytest -s -v tests/integration_tests/  --ignore-glob='*sonar*.py'
+      - uses: actions/checkout@v4
+      - uses: ./.github/actions/setup
+      - run: uvx mypy --install-types --non-interactive --ignore-missing-imports . 
+  tests:
+    runs-on: ubuntu-latest
+    needs: [lock_file]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: ./.github/actions/setup
+      - run: uv run --extra cpu pytest -s -v tests/unit_tests/
+  build:
+    runs-on: ubuntu-latest
+    needs: [lock_file, linting, formatting, type_consistency, tests]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: ./.github/actions/setup
+      - run: uv build
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -0,0 +1,16 @@
+repos:
+  - repo: https://github.com/astral-sh/uv-pre-commit
+    rev: 0.5.7
+    hooks:
+      - id: uv-lock
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.8.2
+    hooks:
+      # Lint
+      - id: ruff
+        args: [ --fix ]
+      # sort imports
+      - id: ruff
+        args: ["check", "--select", "I", "--fix"]
+      # format
+      - id: ruff-format
diff --git a/README.md b/README.md
@@ -31,14 +31,11 @@ The full list of supported languages (along with download links) can be found he
 
 You can install SONAR with `pip install sonar-space`. Note that there is another `sonar` package on pip that IS NOT this project, make sure to use `sonar-space` in your dependencies.
 
-If you want to install SONAR manually, you can install it localy. SONAR depends mainly on [Fairseq2](https://github.com/facebookresearch/fairseq2) and can be installed using (tested with `python=3.8`)
+SONAR depends on fairseq2 and torch/torchaudio, you will have to install the variant (cpu/cuda/...) that works for your environment manully.
 
-```bash
-pip install --upgrade pip
-pip install -e .
-```
+Check [fairseq2 variants](https://github.com/facebookresearch/fairseq2?tab=readme-ov-file#variants) for possible variants. Note that SONAR currently relies on the release candidate for fairseq2 0.3.0 rc1.
 
-If fairseq2 does not provide a build for your machine, check the readme of that project to build it locally.
+If [fairseq2](https://github.com/facebookresearch/fairseq2) does not provide a build for your machine, check the readme of that project to build it locally.
 
 ## Usage
 fairseq2 will automatically download models into your `$TORCH_HOME/hub` directory upon using the commands below.