Skip to main content

Development

Run the frontend of Data Workspace locally to develop features or evaluate if it’s suitable for your use case

Prerequisites

To run the frontend of Data Workspace locally, you must have these tools installed:

You should also have familiarity with the command line, and editing text files. If you plan to make changes to the Data Workspace source code, you should also have familiarity with Python.

Cloning source code

To run the frontend of Data Workspace locally, you must have its source code. This is stored in the data-workspace-frontend GitHub repository, and the process of copying this code so it’s available locally is known as cloning.

  1. If you don’t already have a GitHub account, create a GitHub account.

  2. Setup an SSH key and associate it with your GitHub account.

  3. Create a new fork of the Data Workspace repository. Make a note of the owner you choose to fork to. This is usually your GitHub username. There is more documentation on forking at GitHub’s guide on contributing to projects.

    If you’re a member if the uktrade GitHub organisation you should skip this step and not fork. If you’re not planning on contributing changes, you can also skip forking.

  4. Clone the repository by running the following command, replacing owner with the owner that you forked to in step 3. If you skipped forking, owner should be uktrade:

    git clone git@github.com:owner/data-workspace-frontend.git

    This will create a new directory containing a copy of the Data Workspace source code, data-workspace.

  5. Change to the data-workspace directory:

    cd data-workspace

Creating domains

In order to be able to properly test cookies that are shared with subdomains, localhost is not used for local development. Instead, by default the dataworkspace.test domain is used. For this to work, you will need the below in your /etc/hosts file:

127.0.0.1       dataworkspace.test
127.0.0.1       data-workspace-localstack
127.0.0.1       data-workspace-sso.test
127.0.0.1       superset-admin.dataworkspace.test
127.0.0.1       superset-edit.dataworkspace.test

To run tool and visualisation-related code, you will need subdomains in your /etc/hosts file, such as:

127.0.0.1       visualisation-a.dataworkspace.test

If you are using an Amazon WorkSpaces VDE, you should configure static forwarding for the necessary ports. In your local ~/.ssh/config add the following lines in the Host vde-workspace entry:

Host vde-workspace
    # ...
    LocalForward 8000 127.0.0.1:8000
    LocalForward 8081 127.0.0.1:8081

Starting the application

Set the required variables:

cp .envs/sample.env .envs/dev.env
cp .envs/superset-sample.env .envs/superset.dev.env

Start the application:

UID=$(id -u) GID=$(id -g) docker compose --env-file .envs/dev.env up --build

The application should then be visible at http://dataworkspace.localhost.uktrade.io:8000/.

You will notice that this address does not resolve to localhost. This is normal; *.localhost.uktrade.io is a DNS A record that points back to localhost. This is helpful for anyone who cannot edit their /etc/hosts file.

Running Superset locally

Then run docker compose using the superset profile:

docker compose --profile superset up

You can then visit http://superset-edit.dataworkspace.test:8000/ or http://superset-admin.dataworkspace.test:8000/.

Install pre-commit hooks

The data-workspace-frontend repository uses pre-commit hooks for linting and security scans to detect potential secrets or personal information such as email addresses.

These pre-commit hooks run before a commit is created, after a commit message has been written, and also during the GitHub Actions CI process.

For more information about the department’s security scan pre-commit hooks, please see github-standards.

  1. Create and activate a virtual environment
  2. Install pre-commit (via requirements-dev.txt)
pip install -r requirements.dev.txt
  1. Install the pre-commit hooks
pre-commit install --install-hooks --overwrite
  1. Verify that the pre-commit hooks are running
pre-commit run --all-files

Using a virtual environment

Although the application should be run using docker, there are benefits when developing to installing dataworkspace into a local virtual environment. The main benefit is that VS Code can highlight import issues, and the Go To Definition works as expected.

  1. Make sure you have python version 3.11 available locally, as this is the version the virtual environment will use
  2. Run the command python -m venv venv, this will create a new folder called venv on your machine
  3. Activate the venv by using the command source venv/bin/activate
  4. Run pip install --upgrade pip wheel pip-tools
  5. Run pip install -r requirements-dev.txt

Installing pre-commit hooks

We use pre-commit as a dev dependency to install and configure pre-commit hooks.

The data-workspace-frontend repository uses pre-commit hooks for linting and security scans to detect potential secrets or personal information such as email addresses.

These pre-commit hooks run before a commit is created, after a commit message has been written, and also during the GitHub Actions CI process.

For more information about the department’s security scan pre-commit hooks, please see github-standards.

  1. Create and activate a virtual environment (see above)
  2. Install requirements-dev.txt if you haven’t already done so (see above)
  3. Install the pre-commit hooks
pre-commit install --install-hooks --overwrite
  1. Verify that the pre-commit hooks are running.
pre-commit run --all-files

If the pre-commit hooks are running as expected, you should see a terminal output that looks like this

autopep8 wrapper.........................................................Passed
Flake8...................................................................Passed
Check python ast.........................................................Passed
Check for case conflicts.................................................Passed
Debug Statements (Python)................................................Passed
Fix End of Files.........................................................Passed
pylint...................................................................Passed
Run a security scan......................................................Passed

Front end static assets

We use node-sass to build the front end css and include the GOVUK Front End styles.

To build this locally requires NodeJS. Ideally installed via nvm https://github.com/nvm-sh/nvm:

  # this will configure node from .nvmrc or prompt you to install
  nvm use
  npm install
  npm run build:css

React apps

We’re set up to use django-webpack-loader for hotloading the React app while developing.

You can get it running by starting the dev server:

docker compose up

and in a separate terminal changing to the js app directory and running the webpack hotloader:

cd dataworkspace/dataworkspace/static/js/react/
npm run dev

For production usage we use pre-built JavaScript bundles to reduce the pain of having to build npm modules at deployment.

If you make any changes to the React apps you will need to rebuild and commit the bundles.
This will create the relevant js files in /static/js/bundles/ directory:

cd dataworkspace/dataworkspace/static/js/react/
# this may about 10 minutes to install all dependencies
npm install
npm run build
git add ../bundles/*.js ../stats/react_apps-stats.json

Issues on Apple Silicon

If you have issues building the containers try the following:

DOCKER_DEFAULT_PLATFORM=linux/amd64 docker compose up --build