Linux installation packages migrated from System V init to systemd

Users who run Jenkins 2.335 or later and 2.332.1 LTS or later through the official Debian, Red Hat, or openSUSE packages are affected by this migration.

The System V init(8) packages delivered the following configuration files:

Each configuration file used a unique format, with the Debian version being radically different from the Red Hat and openSUSE versions. Furthermore, manual edits to these files would frequently conflict with updated upstream versions, triggering the package manager to flag the conflict and requiring manual intervention on upgrade.

New versions of Jenkins still ship these files and support Linux distributions that do not use systemd(1). At the time, no actively-supported distributions based on Red Hat or openSUSE are known to use System V init(8). On such systems, the /etc/sysconfig/jenkins file is left in place but unused at runtime. However, the Debian package can still be installed on distributions such as Devuan GNU+Linux, which is a fork of Debian without systemd(1). On such systems, the /etc/default/jenkins file will still be used at runtime.

When installed on a modern Linux distribution running systemd(1), the systemd(1) service unit is delivered to:

The main service unit is read-only and not intended to be edited manually. It contains a large notice at the top of the file stating as much. The canonical way to customize the systemd(1) service is to run systemctl edit jenkins, which creates a new drop-in unit at /etc/systemd/system/jenkins.service.d/override.conf. On a clean install, this drop-in unit does not exist. When a user first customizes a clean install with systemctl edit jenkins, systemd(1) creates the drop-in unit and allows the user to customize it. Note that such customizations must be done in a [Service] section in order to take effect.

One benefit of the drop-in unit is that it unifies configuration across all three distributions: Debian, Red Hat, and openSUSE. Gone are the days of maintaining distribution-specific configuration logic.

Also note that the drop-in unit is not overwritten on upgrades. Gone are the days of getting conflicts in /etc/{default,sysconfig}/jenkins on upgrades, at least after the upgrade to a systemd(1)-based package is completed.

Editing the drop-in unit with systemctl edit jenkins will automatically reload the systemd(1) configuration. The settings will take effect the next time Jenkins is restarted. If you edit the drop-in unit without systemctl(1), you need to run systemctl daemon-reload for the changes to take effect.

A final point to mention about the service unit is its use of specifiers, which may be unfamiliar to some users. The drop-in unit does not perform shell expansion. Specifiers can insert contextual information (like system hostname, unit name, and operating system kernel release) into the drop-in unit. The systemd(1) documentation contains a table of specifiers available in unit files.

The Jenkins project ships logic to automatically migrate the System V init(8) configuration file to the new systemd(1) override.conf format. This migration logic does nothing if an override.conf file already exists, which would be an indication that the migration script already ran or that the user has made their own customizations that should be preserved. If override.conf does not exist, package installation migrates the old System V init(8) configuration file to override.conf.

The systemd(1) package also uses systemd-journald(8) for logging by default. Rather than creating a log file in /var/log/jenkins/jenkins.log, Jenkins now logs to the system’s journal. Log entries may be viewed with journalctl -u jenkins. This is perhaps the most noticeable user-visible change in this migration.

See the DigitalOcean log management tutorial for more detailed information.

The System V init(8) logic was asynchronous; i.e., running /etc/init.d/jenkins start would return prior to the completion of Jenkins startup. The systemd(1) logic is synchronous; i.e., running systemctl start jenkins will block until Jenkins signals successful startup. This allows system administrators to write automation to programmatically deploy Jenkins using modern tools like Ansible.

Jenkins previously restarted itself after upgrading plugins or via the /restart or /safeRestart endpoints by calling exec(2). This was fragile and exposed users to a variety of bugs. The systemd(1) implementation allows the main process to exit normally before starting it again from scratch. In addition to eliminating a category of bugs, this also provides more visibility into service startup progress. Some examples are shown below.

From systemctl status jenkins after upgrading plugins:

As Jenkins is being brought down:

As Jenkins is starting up:

After successful startup:

If Jenkins does not signal startup completion within a configured time, the service will be considered failed and will be shut down again. As each initialization milestone (i.e., "Started initialization", "Listed all plugins", "Prepared all plugins", "Started all plugins", "Augmented all extensions", "System config loaded", "System config adapted", "Loaded all jobs", "Configuration for all jobs updated", and "Completed initialization") is attained, the timeout is extended by the value of the jenkins.model.Jenkins.extendTimeoutSeconds system property (by default, 15 seconds). The timeout can be configured with the TimeoutStartSec directive in the service unit.