Jenkins Pipeline as YAML: experimental plugin

Project goal: Add out-of-the-box support of Jenkins Pipeline definitions in YAML

Skills to study/improve: Java, Jenkins Pipeline, Domain Specific Languages, YAML, Jenkins X

Details

Jenkins Pipeline is a widely used way to define jobs in Jenkins. There are two different syntaxes: Declarative and Scripted Pipeline. Both of them are based on Groovy DSL, and there is interest in a declarative YAML syntax expressed by users. YAML definitions are widely used in CI/CD systems, including Jenkins X, Tekton, and many popular CI/CD services (see below). Having YAML support in Jenkins would make it more approachable for people who don’t know a lot about low-level Pipeline scripting.

For Jenkins Pipeline there are many Pipeline libraries which add support for YAML definitions: Modular Pipeline Library, POET, etc. Even Jenkins project uses YAML definitions for jobs, e.g. in the essentialsTest() step.

During GSoC 2018 we implemented a Simple Pull Request Job Plugin which allows defining Pipelines as YAML which is close to the Declarative Pipeline syntax-wise. See the alpha release announcement here. We have never finalized it, but it could be used as a base for a new project. Similarly, there are plugins like Travis YML which implement YAML definition support for Pipeline.

In this project idea we propose to continue the Pipeline-as-YAML experiment. It can be done by a student by continuing work on the existing plugin, by creating a new plugin or by contributing to the Pipeline: Declarative Plugin. Choice of the approach is up to an applicant. Taking the current state of the CI/CD ecosystem and user demands, we would be interested in the following deliverables (not strict requirements!):

  • New Pipeline syntax which is close to the Jenkins X Pipeline. It is not possible to provide full syntax compatibility between Jenkins and Jenkins X due to different use-cases, but having a close syntax is important for interoperability of projects.

  • Support of the new Pipeline type in Pipeline jobs, Multi-Branch Pipeline and organization folders.

  • Support of the Pipeline YAML definition in SCM (e.g. jenkinsfile.yml in the repo root)

  • Support of key Declarative Pipeline features: steps, agent definitions, matrix configurations, and post-build actions

  • YAML Schema or YAML schema generator for the newly introduced syntax

  • Sample projects which demonstrate usage of the new Pipeline definitions

Students are also welcome to come up with original ideas about Jenkins Pipeline related to declarative definitions, e.g. Declarative Pipeline improvements or support of other syntaxes in Jenkins (e.g. Tekton).

Quickstart

Further discovery

  • If you are familiar with Kubernetes, explore Jenkins X and specifically Jenkins X Pipelines. Also, you can explore the Tekton project which also implements Pipelines

  • Explore other CI/CD tools which use YAML build definitions: Travis CI, Appveyor, Gitlab CI, GitHub Actions, etc. All these tools have demo projects and numerous real-life examples in open-source projects, and you can use them as additional source of inspiration.

Newbie-friendly issues

Right now there are no specific newbie-friendly issues created for the project, but there is a lot of small improvements which could be done in the Simple Pull Request Job Plugin plugin. Also, you can submit patches to existing Pipeline libraries which introduce support of YAML definitions.

References

Sample Pipeline declarations in YAML

Potential Mentors

Project Links

Organization Links

> Go back to other GSoC 2020 project ideas