Skip to content

Maintainer Info

Jump to bottom Edit New page
Jack Gerrits edited this page Dec 28, 2022 · 35 revisions

Semantic PR title types

See here: https://github.com/commitizen/conventional-commit-types/blob/master/index.json

Issue filters:

Nix

Setup Nix with flakes support

Linux, MacOS, WSL

  1. Install Nix
  2. Enable flakes:
    • Permanently (recommended): 
      mkdir -p ~/.config/nix
echo 'experimental-features = nix-command flakes' >> ~/.config/nix/nix.conf
    • Or, per command. Add: --extra-experimental-features nix-command --extra-experimental-features flakes

Windows

Docker can be used on Windows to provide a nix environment like so:

docker run -v %cd%:/src -it nixos/nix nix --extra-experimental-features nix-command --extra-experimental-features flakes develop /src

Clang format

To format the differences between master and HEAD:

git diff master -U0 --no-color | nix develop -c clang-format-diff -p1 -i

To format the entire codebase:

nix develop -c ./utl/clang-format.sh fix

Clang tidy

To check the changes between your branch and master:

git diff master -U0 --no-color | nix develop -c vw-clang-tidy-diff

To run clang-tidy on all files, as is done in CI you can run:

nix develop -c vw-clang-tidy

To just check specific files you can pass them as args:

nix develop -c vw-clang-tidy vowpalwabbit/core/src/kskip_ngram_transformer.cc

To apply fixes use -fix:

nix develop -c vw-clang-tidy vowpalwabbit/core/src/kskip_ngram_transformer.cc -fix

All args are passed through to the underlying run-clang-format call.

Using from Windows:

  • Using WSL - commands are identical as above
  • Using docker 
    docker run -v %cd%:/src -it nixos/nix nix --extra-experimental-features nix-command --extra-experimental-features flakes develop /src -c vw-clang-tidy

Running unit tests

Unit tests are defined per lib. Each lib (optionally) has a tests directory containing GoogleTest based tests. These are named as vw_LIB_test where LIB is the name of the library the tests are for. For example, vw_core_test.

The following assumes build is the CMake build directory used.

All tests across the project can be run using ctest like so:

ctest --test-dir build --output-on-failure --no-tests=error --label-regex VWTestList

ctest also supports parallel execution (-j):

ctest --test-dir build --output-on-failure --no-tests=error --label-regex VWTestList --parallel 4

Note: Lib tests are added to the VWTestList label to help avoid ctest running any subproject tests (zlib is the main culprit here).

To exclude the tests that use iterations and are generally slower you can use a regex exclude:

ctest --test-dir build --output-on-failure --no-tests=error --exclude-regex w_iterations --label-regex VWTestList

An individual lib's tests can be run from the build directory by running the built executable. For example:

./build/vowpalwabbit/core/vw_core_test

Valgrind

Unit tests can be run with valgrind like so:

ctest --output-on-failure --label-regex VWTestList -T memcheck --overwrite MemoryCheckCommandOptions="--leak-check=full --error-exitcode=100"

GDB - Pretty printing

The pretty printing script allows GDB to visualize types that it cannot by default, such as v_array. This works both with command line and using GDB in VSCode. The same thing is done in Visual Studio on Windows using the natvis file, which is included in the project and automatically loaded.

In order to use the GDB pretty printers, they need to be loaded. This can either be done automatically when GDB is started or each time.

To load the script into a running gdb session you can run:

(gdb) python exec(open('path/to/vowpalwabbit/gdb_pretty_printers.py').read())

To automatically load it, create the file if it does not exist ~/.gdbinit, and add the script load into that file:

python exec(open('path/to/vowpalwabbit/gdb_pretty_printers.py').read())

If using VSCode, ensure the launch target contains the following properties:

MIMode": "gdb",
"setupCommands": [
    {
        "description": "Enable pretty-printing for gdb",
        "text": "-enable-pretty-printing",
        "ignoreFailures": true
    }
]

Python tools

Installation

pip3 install yapf mypy pylint --user

Tools

  • yapf - Code formatter
    • Usage: python3 -m yapf -i python/vowpalwabbit/*.py
    • By default uses PEP8 style
    • Any other formatter can be used
  • mypy - Static type checking
    • Usage: python3 -m mypy vowpalwabbit/*.py --ignore-missing-imports
      • --ignore-missing-imports is required as the native extension, `pylibvw, does not have type stubs.
    • mypy supports mixed dynamic and static typed Python, so the code can be annotated one function at a time
  • pylint - Python code linting
    • Usage: python3 -m pylint vowpalwabbit/*.py

Obtaining new CMake version

Ubuntu

Instructions from: https://apt.kitware.com/

sudo apt-get update
sudo apt-get install apt-transport-https ca-certificates gnupg software-properties-common wget
wget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc 2>/dev/null | sudo apt-key add -

# 16.04
sudo apt-add-repository 'deb https://apt.kitware.com/ubuntu/ xenial main'

# 18.04
sudo apt-add-repository 'deb https://apt.kitware.com/ubuntu/ bionic main'

# 20.04
sudo apt-add-repository 'deb https://apt.kitware.com/ubuntu/ focal main'

sudo apt-get update

# Optional, ensure keyring stays up to date as it is rotated.
sudo apt-get install kitware-archive-keyring
sudo apt-key --keyring /etc/apt/trusted.gpg del C1F34CDD40CD72DA

sudo apt-get install cmake

Windows

Download and install from: https://cmake.org/download/

Manual Benchmark CI

  1. Go here: https://github.com/VowpalWabbit/vowpal_wabbit/actions/workflows/run_benchmarks_manual.yml
  2. Click "Run workflow" and input the refs you wish to test.

Generating Linux release artifacts

export VW_REPO_DIR=...
docker pull ubuntu:16.04
docker run -it --rm -v $VW_REPO_DIR:/vw ubuntu:16.04 /bin/bash

Inside the docker container:

apt update
apt install -y g++ wget dpkg-dev

# Install CMake
# Look here: https://github.com/VowpalWabbit/vowpal_wabbit/wiki/Maintainer-Info#obtaining-new-cmake-version

# Install zlib
wget -O zlib.tar.gz 'https://zlib.net/fossils/zlib-1.2.8.tar.gz' \
   && tar xvzf zlib.tar.gz \
   && cd zlib-1.2.8 \
   && ./configure --static --archs=-fPIC \
   && make -j$(nproc) \
   && make install \
   && cd .. && rm -rf zlib*

# Install boost 
wget -O boost.tar.gz 'https://sourceforge.net/projects/boost/files/boost/1.70.0/boost_1_70_0.tar.gz/download' \
   && tar -xvzf boost.tar.gz \
   && mkdir boost_output \
   && cd boost_1_70_0 \
   && ./bootstrap.sh --prefix=/boost_output --with-libraries=program_options,system,thread,test,chrono,date_time,atomic \
   && ./bjam -j$(nproc) cxxflags=-fPIC cflags=-fPIC -a install \
   && cd .. && rm -rf boost_1_70_0 boost.tar.gz

cd /vw
mkdir build
cd build
cmake .. -DBUILD_SHARED_LIBS=Off -DSTATIC_LINK_VW_JAVA=On -DBUILD_TESTS=Off -DBUILD_JAVA=Off -DBUILD_PYTHON=Off
make all -j$(nproc)

cpack -G DEB 
cpack -G TGZ

Build with vendored dependencies

To use vendored dependencies the following can be used:

cmake -S . -B build -G Ninja \
    -DCMAKE_BUILD_TYPE:STRING="Release" \
    -DFMT_SYS_DEP:BOOL="OFF" \
    -DRAPIDJSON_SYS_DEP:BOOL="OFF" \
    -DSPDLOG_SYS_DEP:BOOL="OFF" \
    -DVW_BOOST_MATH_SYS_DEP:BOOL="OFF" \
    -DVW_GTEST_SYS_DEP:BOOL="OFF" \
    -DVW_ZLIB_SYS_DEP:BOOL="OFF" \
    -DBUILD_TESTING:BOOL="OFF"
cmake --build build

Old Nix info

Nix allows for declarative package management (among other things) and reproducible builds across systems. To obtain a standard environment to build vw in you can simply run nix-shell --pure in the same direct as the following file. This will drop you into a new shell with all of the dependencies installed.

with import <nixpkgs> { };
{
  mystdenv ? pkgs.stdenv,
  myboost ? pkgs.boost174
 }:
mystdenv.mkDerivation {
  name = "vw-build-nix-shell";
  buildInputs = [
    cmake
    myboost
    zlib
    ninja
    flatbuffers
  ];
}

To override the compiler version:

# GCC - standard version
nix-shell --pure --arg mystdenv '(import <nixpkgs> {}).gccStdenv'
# GCC - specific version
nix-shell --pure --arg mystdenv '(import <nixpkgs> {}).gcc8Stdenv'

# Clang - standard version
nix-shell --pure --arg mystdenv '(import <nixpkgs> {}).clangStdenv'
# Clang - specific version
nix-shell --pure --arg mystdenv '(import <nixpkgs> {}).llvmPackages_9.stdenv'

To specify which Boost version to use:

nix-shell --pure --arg myboost '(import <nixpkgs> {}).boost174'
Add a custom footer
Clone this wiki locally