Details
-
Task
-
Resolution: Unresolved
-
L3 - Default
-
None
-
None
-
None
Description
Context
Umbrella ticket to improve the performance of the CI.
Proposed Approach
- Establish data to understand CI performance
- Where does a pipeline run spend its time?
- How many pipelines/stages were in parallel execution or scheduled over time?
- Establish continuous monitoring of performance data
- Define performance goal per pipeline
- Identify root causes
- Choose and implement tasks that fix the root causes
Hints
- Work iteratively, it is very easy to sink time here without any benefit. Carefully consider which next step to take and if it is worth it
- Resulting tasks may be for us or Infra
Potential tasks
- Don't use the Jenkins master for stashing; instead, use Google Bucket (CAM-14537)
- Improve performance of cambpmLogParser (CAM-13010)
- Avoid parallel pipeline runs: if a new run is triggered, the previous run is canceled (cf. [1])
- One downside could be that it is harder to understand what commit causes a CI failure; however, finding this out doesn't often happen these days since we have the PR-based CI and don't "hot merge" anymore.
- Detect release commits and run only assembly and distro-ee stages
[1] https://stackoverflow.com/questions/40760716/jenkins-abort-running-build-if-new-one-is-started
