Right now, integration tests exist on a cluster.
We run them from a master.
This means that to run a changed integration test (e.g. during development), one must either:
- Wait on CI (slow)
- Move the integration test file from the host laptop to the right place on a node (complex)
- Edit the file on a master node (complex, not my local dev environment)
This workflow is unusual and it is a barrier to entry to the integration test suite. This has been brought up internally by e.g. Ken Sipe.
We also therefore have to:
- Have all integration test requirements as DC/OS requirements
- Be prudent with requirements (because of the above)
- Use the same Python version in our tests as DC/OS uses
- Duplicate requirements between requirements.txt and pkgpanda - we have requirements.txt for tox linting
This issue proposes that we:
1. Make it possible to run integration tests from a host laptop against an existing cluster
2. Consider removing the integration tests from DC/OS (and make a follow-up issue if needed)
To have the integration tests running locally, I should follow instructions which tell me to:
- cd to some directory
- Create a virtualenv with a particular Python
- pip install -r requirements.txt
- Set some environment variables
- Run pytest
This might also be possible from tox.
Installing the requirements from a file already works.
The main issue is that some tests will fail as they need to be run from a cluster.
For example, those tests which read a file - that command should be run remotely on a node.
Instead, we should run a command remotely on the node to read the file (or download it and then read it).
We will find that some tests will not run in this way, for example probably those which include:
Let's enumerate those and then decide what to do with them.
Options may include putting them into the test-e2e repository with some modification.
Another follow-up step will be to change CI to run in this way.