LLD:SpeculosTesting
- Introduction
- Testing strategy
- Configuration and execution
- Reporting
- Workflow
- Contribution
- Limitations
This project is maintained by the QAA squad. Feel free to reach out if you have questions, thanks in advance for your contribution!
To enhance our testing framework and better align with user experience, we decided to create a new test suite leveraging Speculos.
Speculos is an emulator for Nano apps, allowing interaction with the blockchain, including verifying addresses, sending, swapping, and more.
Currently, we maintain over 70 UI end-to-end test scenarios executed using Speculos in conjunction with Playwright.
Speculos is a launcher. In the context of e2e tests on Ledger Live, our goal is to instantiate it with the correct libraries that will be used later on during the test scenario. In the context of swaps, up to 2 libraries will be needed to perform the operations (app to swap from and app to swap to). If a swap is to be performed from ETH to USDT, only one library will be needed (ETH one as USDT is an erc20). To make it work seamlessly, we add the app candidates (based on the device and last apps versions) to a map to make sure every lib is present no more than once. The swap dependencies are loaded at the beginning of a scenario, within the exchange app.
Spawning Speculos: For each cryptocurrency (coin), a dedicated Speculos instance is spawned. This instance emulates the specific Nano app needed for the tests.
Running Tests: The test scenarios are then executed, interacting with the Speculos instance to simulate real blockchain operations.
Terminating Speculos: Once the tests are completed, the Speculos instance is terminated to free up resources.
This strategy allows us to remove mocks and conduct tests that closely mirror real user interactions, providing more reliable test results.
To optimize the test execution time, especially during CI runs, we employ a sharding strategy. This involves splitting the test suite into smaller shards, which are run in parallel across multiple instances. This approach reduces the overall execution time and ensures faster feedback. Today we are using 3 shards.
A nightly run of the full test suite is scheduled to ensure ongoing stability. The result of this run is visible in the #live-repo-health slack channel.
You can also execute Speculos tests in the CI using the @Desktop - UI e2e - Test App workflow.
The workflow use specific inputs that can be configured before execution:
Input 1: Send notification on slack channel
Input 2: Send results to Xray:
Imput 3: Enable Broadcast input:
In case of test failures, the following steps should be taken to investigate:
Check Screenshots on allure report: Screenshots are automatically captured at the point of failure and can provide immediate insight into the issue.
Review Logs: Logs generated during test execution can help identify the root cause of the failure.
Rerun the Test: If needed, rerun the specific failed test in isolation to reproduce and analyze the issue.
All test results are collected and reported through Allure. The Allure dashboard provides detailed reports on each test case, including steps executed, screenshots, and logs. This enables easy tracking of test performance and quick identification of issues.
Test results are also integrated with Slack for real-time notifications. The results are posted in the #live-repo-health channel, including:
A direct link to the Allure report for detailed insights.
A link to the CI run where the tests were executed.
This ensures that the members of the channel are immediately informed of any issues that arise during test execution.
To send test results to Xray, ensure that the "Send tests results to Xray" option is checked in the workflow input fields. Additionally, you must provide a valid Jira Test Execution ticket ID in the "Test Execution ticket ID" field.
We welcome contributions to this project. Whether it's adding new test scenarios, or improving existing ones, your contributions are highly appreciated. Please follow the standard contribution guidelines and ensure all changes are tested thoroughly before submitting a pull request.
While this testing strategy offers significant benefits, there are some limitations to be aware of:
The primary limitation is that Speculos can only emulate the applications (apps) running on Nano devices, not the underlying operating system (OS) of the devices themselves. As a result, certain functionalities of Ledger Live cannot be fully covered by Speculos-based tests. For example:
- App Installation: It is not possible to simulate the downloading and installation of apps onto the Nano device.
- My Ledger Section: Operations that involve interacting with the "My Ledger" section, such as managing apps on the device, cannot be tested using Speculos.
We continue to explore ways to mitigate these limitations and enhance the overall coverage and reliability of our test suite.