Tariff Management Tool

The Tariff Management Tool (TaMaTo) is a web application that enables Tariff Managers to browse and make changes to the UK Global Tariff, and submit these changes to HMRC.

The tool is available at https://www.manage-trade-tariffs.trade.gov.uk/

Development environment setup

Prerequisites

The following dependencies are required to run this app:

• Python 3.12.x
• Node.js 20.10.0 (LTS)
• PostgreSQL 12
• Redis 5.x

The app requires an AWS S3 bucket or a compatible implementation, such as MinIO

If using MacOS then libmagic is required:

$ brew install libmagic

Database

Create a database instance and user:

$ sudo su - postgres
postgres $ createdb tamato
postgres $ psql -c "CREATE USER <username> SUPERUSER PASSWORD '<password>';"

Make a note of the <username> and <password> for use in the :envvar:`DATABASE_URL` environment variable.

Import from a dump of the database:

postgres $ psql -d tamato -f /tmp/tamato-db-dump.sql

To get a database dump, please contact the TAP team.

Installing

$ git clone [email protected]:uktrade/tamato
$ cd tamato
$ python -m venv venv
$ source venv/bin/activate
$ pip install -U pip
$ pip install wheel -r requirements-dev.txt
$ npm install
$ npm run build

Running

Create a .env file containing :doc:`environment variables <envvars>`

$ cp sample.env .env

Open another terminal and Compile SCSS and Javascripts:

$ python manage.py collectstatic

To be able to login to the app, you will first need to create a Django user with superuser access:

$ python manage.py createsuperuser

Then run the app:

$ python manage.py runserver

Then you can browse to http://localhost:8000/ to view the app. To access the Django admin page, browse to http://localhost:8000/admin/.

In order to define or override developer-specific Django settings in a local developement environment, then you may wish to create a settings/dev_override.py file.

Testing

To run tests use the following command:

$ python manage.py test

For more detailed information on running tests, see :doc:`testing`

Pre-commit hooks

This project uses pre-commit hooks to update formatting and identify potential sensitive data before it is committed to the public repo.

note: The python package pre-commit is a requirement within requirements-dev.txt and should be installed to meet development requirements

Install

To initially setup the pre-commit hooks you can run the following command.

$ pre-commit install

Once installed, when committing it will first run all the predefined processes to clean up code formatting and notify about any detected sensitive strings found that are not in pii exclude files.

Note: the first commit or run of the pre-commit hooks after installing may take a few minutes for setup the dependent packages for the first time. This is normal, and will be faster on subsequent commits.

Update

The packages used to perform the pre-commit process are regularly updated. Periodically its advised you run the following command to keep the dependencies updated.

$ pre-commit autoupdate

This will verify that the dependencies are updated based on requirements.

Uninstall

The pre-commit hooks can be uninstalled with the following command

$ pre-commit uninstall

Run the hooks without committing

You may at times want to run the pre-commit hooks before committing. This can be done with the following command. This command will run the hooks on all changed files.

$ pre-commit run

If you would like to run the hooks over all files you can run the following command

$ pre-commit run -a

or

$ pre-commit run --all-files

Troubleshooting

If you encounter issues with the pre-commit hooks there are a number of things you can clear the cached pre-committed files using this command:

$ pre-commit clean

If that fails you can try updating the dependencies for the hooks

If the above fails, uninstall and then install again.

Dockerisation

Fully dockerised service

Prerequisites:

• A local instance of the tool can be run using Docker.
• A database dump - contact the TAP team for a database snapshot.

Guidance for running tamato via docker in Pycharm (follow initial set up below first) `Docker in PyCharm https://www.jetbrains.com/help/pycharm/using-docker-as-a-remote-interpreter.html#run`__.

https://testdriven.io/blog/django-debugging-pycharm/

Download the codebase:

$ git clone [email protected]:uktrade/tamato
$ cd tamato

Build and Run for the first time:

$ cp sample.env .env
    # Not used will be used for specific local docker stuff
    # cp docker-compose.override.yml.example docker-compose.override.yml

# to overwrite default db dump name pass in DB_DUMP=tamato_db.sql
$ make docker-first-use
    # take a tea break to import the db dump then
    # enter super user details when prompted
    # and visit localhost:8000/ when the containers are up

Run the tamato app every other time:

$ make docker-build
$ make docker-up

Go to http://localhost:8000/ in your web browser to view the app

Import from a dump of the database:

# to overwrite defaults
#   DB_DUMP=tamato_db.sql
#   DB_NAME=tamato
#   DB_USER=postgres
#   TEMPLATE_NAME={DB_NAME}_{DATE}
# this overwrites the default file set in the makefile variable
# docker-import-new-db will create a new template with the provided DB dump
# can override the name of the template at TEMPLATE_NAME
$ make docker-import-new-db

# Will restore the db DB_NAME with the provided TEMPLATE_NAME
$ make docker-restore-db

Sometimes docker gets clogged up and we need to clean it:

# cleans up images & volumes
$ make docker-clean
# cleans up everything including the cache which can get filled up because of db dumps
$ make docker-deep-clean

Run database migrations:

$ make docker-migrate

Create a superuser, to enable logging in to the app:

$ make docker-superuser

Run tests from within a docker container:

$ make docker-test
$ make docker-test-fast

DOCKER_RUN=run --rm by default but can be set to exec if you have containers up and running General commands:

$ make docker-down # brings down containers
$ make docker-up-db # brings up db in the background
$ make docker-makemigrations # runs django makemigrations
$ make docker-checkmigrations # runs django checkmigrations
$ make docker-bash # bash shell in tamato container
$ make docker-django-shell # django shell in tamato container

Hybrid host + container approach

You may prefer a hybrid approach to running Tamato, say, executing the Redis service and Celery workers in containers and the remaining services in the host environment. To do so, create a docker-compose.override.yml file to allow loading environment settings that are specific to this configuration:

version: '3'

services:
celery:
    env_file:
    - .env
    - settings/envs/docker.env
    - settings/envs/docker.override.env

rule-check-celery:
    env_file:
    - .env
    - settings/envs/docker.env
    - settings/envs/docker.override.env

Create a docker.override.env file:

# Point containerised services at the host environment hosted DB.
DATABASE_URL=postgres://host.docker.internal:5432/tamato

Now start dockerised instances of Redis and the Celery worker services:

$ docker-compose up -d celery-redis celery rule-check-celery

Using the importer

The Tariff Management Tool (TaMaTo) needs to import TARIC3 XML data from both the EU (for historical data) and from HMRC (for VAT measures).

TaMaTo provides an import which parses TARIC3 XML and inserts the data into the TAMATO database.

Run the script to see the command line arguments:

$ python manage.py import_taric --help

This command is broken into two stages:

1. Chunking the file and loading into the DB. If a file is greater than 50MB it is broken into chunks and those chunks saved into the database. This can be run in isolation using the command

   $ python manage.py chunk_taric

2. Passing the chunks through the importer system into TrackedModels. This can be run in isolation using the command

   $ python manage.py run_import_batch

Using the TARIC parser (currently referenced importer v2)

There are no command line tools available for this tool.

This tool is available as an importer alternative found within the web front end in the footer menu under "New TARIC parser".

This tool addresses several short falls that the current importer has.

Reference document data import

WARNING: this feature is in alpha : do not use in production until this feature has been fully tested.

In order to populate the reference document data from extracted data from an external tool you can use the management command ref_doc_csv_importer

Example:

$ python manage.py ref_doc_csv_importer "/absolute/path/to/duties.csv" "/absolute/path/to/quotas.csv"

Using the exporter

The Tariff Management Tool (TaMaTo) exports data to HMRC.

During normal operation uploads trigger the upload_transactions task which uploads transactions as XML to the HMRC bucket.

Running the exporter

The exporter pushes data to a queue, which one or more asynchronous worker processes monitor and perform the upload to S3, so as not to block the web server.

To run the exporter queue process, run the following command:

celery -A common.celery beat --loglevel=info

Open another terminal and start a Celery worker:

celery -A common.celery worker --loglevel=info -Q standard,rule-check
# The celery worker can be run as two workers for each queue
celery -A common.celery worker --loglevel=info -Q standard
celery -A common.celery worker --loglevel=info -Q rule-check

To monitor celery workers or individual tasks run:

celery flower

See flower docs for more details

Manually trigger the upload to s3

$ celery -A common.celery call exporter.tasks.upload_transactions

The celery job UUID is output and the command quits. To see output switch to the celery workers console. A more ergonomic way of launching the celery job is to launch the management command:

$ python manage.py upload_transactions

Dump transactions

Transactions waiting to be uploaded to the HMRC S3 bucket can be saved to a file or output to stdout using a management command:

$ python manage.py dump_transactions [-o filename]

Output defaults to stdout if filename is - or is not supplied.

Mocking s3 upload with minio

1. Follow instructions to install minio server
2. Export MINIO_ROOT_USER and MINIO_ROOT_PASSWORD variables of your choice
3. Run server with:

minio server --quiet --address 0.0.0.0:9003 ~/data

4. Navigate to http://localhost:9003/ and login using root user and password credentials just created. Create a bucket and an access key via the console.

5. Export environment variables for any storages you wish to dummy (e.g. for sqlite dump export this will be SQLITE_STORAGE_BUCKET_NAME, SQLITE_S3_ACCESS_KEY_ID, SQLITE_S3_SECRET_ACCESS_KEY, SQLITE_S3_ENDPOINT_URL, and SQLITE_STORAGE_DIRECTORY), setting s3 endpoint url to http://localhost:9003/

6. Alternatively, export all environment variables temporarily to an environment such as Bash (useful when running a local development instance of a Celery worker):

   set -a && source .env && set +a

Virus Scan and running locally

We use a shared service accross the department for virus scanning to run locally set up the following: 1. Follow set up instructions and run it 2. set SKIP_CLAM_AV_FILE_UPLOAD to False and CLAM_USE_HTTP True 3. add CLAM_AV_DOMAIN without http(s):// 4. set CLAM_AV_USERNAME,CLAM_AV_PASSWORD as the username and password found in the config.py in the dit-clamav-rest project

Application maintenance mode

The application can be put into a "maintenance mode" type of operation. By doing so, all user web access is routed to a maintenance view and the default database route removes the application's access to the database. This prevents inadvertent changes by users, via the application UI, to application data while in maintenance mode. Note, however, that this would not restrict other forms of data update, such as active Celery tasks - Celery and other similar processes need to be scaled down separately.

The process for transitioning the application into and back out of maintenance mode is as follows:

1. Set the application’s MAINTENANCE_MODE environment variable to True.
2. Restart the application so that it picks up the new value of MAINTENANCE_MODE.
3. Complete maintenance activities.
4. Set the value of the MAINTENANCE_MODE environment variable to False.
5. Restart the application.

How to contribute

See :ref:`contributing`

How to deploy

The app is deployed with Jenkins via the Tariffs/TaMaTo job. The master branch may be deployed to development, staging, uat, training or production environments by selecting the environment name from the ENV dropdown on the Build with Parameters page.

Accessing databases in GOV.UK PaaS

To access databases hosted in GOV.UK PaaS directly, you will need a PaaS login and the cf CLI tool.

You will need to install the conduit plugin:

cf install-plugin conduit

Then you need to login to the DIT GOV.UK PaaS:

cf login --sso -s <space>

Where <space> is one of tariffs-dev, tariffs-staging, tariffs-training or tariffs-uat.

Once you are logged in, you can list the services hosted in the space with

cf services

You can access postgres services with the following command:

cf conduit <name> -- psql

So if you are logged in to the tariffs-dev space, you could access the dev environment database with cf conduit tamato-dev-db -- psql.