CI observability concepts
Overview
To make the most our of Shipfox CI observability, it's useful to understand the main concepts how the product is structured.
Resources
Shipfox CI visibility handles 6 type of resources.
| Resource type | Description |
|---|---|
| CI pipeline | Group of CI jobs that are related and triggered together. Workflow in GitHub, Pipeline in GitLab. |
| CI job | CI task that runs continuously on a single machine. Job in GitHub and GitLab. |
| CI step | Single action within a CI job, such as running a command. |
| Test run | Ensemble of tests that run at the same time. |
| Test suite | Related tests located in the same file. |
| Test case | Single test. |
Flakiness
A flaky execution yields both passing and failing results despite zero changes to the code, test, or CI script. In other words, flaky executions fail to produce the same outcome with each individual run. The nondeterministic nature of flaky executions makes debugging extremely difficult for developers and can translate to issues for your team and end users.
Shipfox CI observability considers a resource execution to be flaky, if the execution failed and that there is another execution of the same resource with a successful status on the same commit. We also consider flaky executions for which there has been automatic retries to produce a successful result (typically the case for tests).