我在使用 jenkins kubernetes pod 时遇到问题，它在最后一次 pod 重启(由 kubernetes 完成)后停止工作。
所以，我的日志中出现了这样的错误:
2021-02-05 11:00:55.856+0000 [id=27] INFO jenkins.InitReactorRunner$1#onAttained: Listed all plugins
2021-02-05 11:00:56.883+0000 [id=30] SEVERE jenkins.InitReactorRunner$1#onTaskFailed: Failed Loading plugin Pipeline: Multibranch v2.22 (workflow-multibranch)
java.io.IOException: Failed to load: Pipeline: Multibranch (2.22)
- Update required: Pipeline: Job (2.36) to be updated to 2.39 or higher
at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:952)
at hudson.PluginManager$2$1$1.run(PluginManager.java:549)
at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
at jenkins.model.Jenkins$5.runTask(Jenkins.java:1131)
at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
我看到我应该将 workflow-job 插件的版本升级到 2.39。
Jenkins 由 helm 管理。如果我从 S3 下载最新的 helm chart，我可以看到其中有一个文件夹 jenkins，我可以在其中看到 jenkins/values.yaml，如下所示:
# Default values for jenkins.
# This is a YAML-formatted file.
# Declare name/value pairs to be passed into your templates.
# name: value
## Overrides for generated resource names
# See templates/_helpers.tpl
# nameOverride:
# fullnameOverride:
Master:
Name: jenkins-master
Image: "jenkins/jenkins"
ImageTag: "lts"
ImagePullPolicy: "Always"
# ImagePullSecret: jenkins
Component: "jenkins-master"
NumExecutors: 0
# configAutoReload requires UseSecurity is set to true:
UseSecurity: true
# SecurityRealm:
# Optionally configure a different AuthorizationStrategy using Jenkins XML
# AuthorizationStrategy: |-
# <authorizationStrategy class="hudson.security.FullControlOnceLoggedInAuthorizationStrategy">
# <denyAnonymousReadAccess>true</denyAnonymousReadAccess>
# </authorizationStrategy>
HostNetworking: false
# When enabling LDAP or another non-Jenkins identity source, the built-in admin account will no longer exist.
# Since the AdminUser is used by configAutoReload, in order to use configAutoReload you must change the
# .Master.AdminUser to a valid username on your LDAP (or other) server. This user does not need
# to have administrator rights in Jenkins (the default Overall:Read is sufficient) nor will it be granted any
# additional rights. Failure to do this will cause the sidecar container to fail to authenticate via SSH and enter
# a restart loop. Likewise if you disable the non-Jenkins identity store and instead use the Jenkins internal one,
# you should revert Master.AdminUser to your preferred admin user:
AdminUser: admin
# AdminPassword: <defaults to random>
OwnSshKey: false
# If CasC auto-reload is enabled, an SSH (RSA) keypair is needed. Can either provide your own, or leave unconfigured\false to allow a random key to be auto-generated.
# If you choose to use your own, you must upload your decrypted RSA private key (not the public key above) to a Kubernetes secret using the following command:
# kubectl -n <namespace> create secret generic <helm_release_name> --dry-run --from-file=jenkins-admin-private-key=~/.ssh/id_rsa -o yaml |kubectl -n <namespace> apply -f -
# Replace ~/.ssh/id_rsa in the above command with the path to your private key file and the <helm_release_name> and <namespace> placeholders to suit.
RollingUpdate: {}
# Ignored if Persistence is enabled
# maxSurge: 1
# maxUnavailable: 25%
resources:
requests:
cpu: "50m"
memory: "256Mi"
limits:
cpu: "2000m"
memory: "4096Mi"
# Environment variables that get added to the init container (useful for e.g. http_proxy)
# InitContainerEnv:
# - name: http_proxy
# value: "http://192.168.64.1:3128"
# ContainerEnv:
# - name: http_proxy
# value: "http://192.168.64.1:3128"
# Set min/max heap here if needed with:
# JavaOpts: "-Xms512m -Xmx512m"
# JenkinsOpts: ""
# JenkinsUrl: ""
# If you set this prefix and use ingress controller then you might want to set the ingress path below
# JenkinsUriPrefix: "/jenkins"
# Enable pod security context (must be `true` if RunAsUser or FsGroup are set)
UsePodSecurityContext: true
# Set RunAsUser to 1000 to let Jenkins run as non-root user 'jenkins' which exists in 'jenkins/jenkins' docker image.
# When setting RunAsUser to a different value than 0 also set FsGroup to the same value:
# RunAsUser: <defaults to 0>
# FsGroup: <will be omitted in deployment if RunAsUser is 0>
ServicePort: 8080
# For minikube, set this to NodePort, elsewhere use LoadBalancer
# Use ClusterIP if your setup includes ingress controller
ServiceType: LoadBalancer
# Master Service annotations
ServiceAnnotations: {}
# Master Custom Labels
DeploymentLabels: {}
# foo: bar
# bar: foo
# Master Service Labels
ServiceLabels: {}
# service.beta.kubernetes.io/aws-load-balancer-backend-protocol: https
# Put labels on jeknins-master pod
PodLabels: {}
# Used to create Ingress record (should used with ServiceType: ClusterIP)
# HostName: jenkins.cluster.local
# NodePort: <to set explicitly, choose port between 30000-32767
# Enable Kubernetes Liveness and Readiness Probes
# ~ 2 minutes to allow Jenkins to restart when upgrading plugins. Set ReadinessTimeout to be shorter than LivenessTimeout.
HealthProbes: true
HealthProbesLivenessTimeout: 90
HealthProbesReadinessTimeout: 60
HealthProbeReadinessPeriodSeconds: 10
HealthProbeLivenessFailureThreshold: 12
SlaveListenerPort: 50000
# SlaveHostPort: 50000
DisabledAgentProtocols:
- JNLP-connect
- JNLP2-connect
CSRF:
DefaultCrumbIssuer:
Enabled: true
ProxyCompatability: true
CLI: false
# Kubernetes service type for the JNLP slave service
# SlaveListenerServiceType is the Kubernetes Service type for the JNLP slave service,
# either 'LoadBalancer', 'NodePort', or 'ClusterIP'
# Note if you set this to 'LoadBalancer', you *must* define annotations to secure it. By default
# this will be an external load balancer and allowing inbound 0.0.0.0/0, a HUGE
# security risk: https://github.com/kubernetes/charts/issues/1341
SlaveListenerServiceType: ClusterIP
SlaveListenerServiceAnnotations: {}
# Example of 'LoadBalancer' type of slave listener with annotations securing it
# SlaveListenerServiceType: LoadBalancer
# SlaveListenerServiceAnnotations:
# service.beta.kubernetes.io/aws-load-balancer-internal: "True"
# service.beta.kubernetes.io/load-balancer-source-ranges: "172.0.0.0/8, 10.0.0.0/8"
# LoadBalancerSourcesRange is a list of allowed CIDR values, which are combined with ServicePort to
# set allowed inbound rules on the security group assigned to the master load balancer
LoadBalancerSourceRanges:
- 0.0.0.0/0
# Optionally assign a known public LB IP
# LoadBalancerIP: 1.2.3.4
# Optionally configure a JMX port
# requires additional JavaOpts, ie
# JavaOpts: >
# -Dcom.sun.management.jmxremote.port=4000
# -Dcom.sun.management.jmxremote.authenticate=false
# -Dcom.sun.management.jmxremote.ssl=false
# JMXPort: 4000
# Optionally configure other ports to expose in the Master container
ExtraPorts:
# - name: BuildInfoProxy
# port: 9000
# List of plugins to be install during Jenkins master start
OverwritePlugins: true
InstallPlugins:
- kubernetes:1.16.1
- workflow-job:2.39
- workflow-aggregator:2.6
- workflow-basic-steps:2.18
- credentials-binding:1.23
- job-dsl:1.76
- git:4.2.2
- parameterized-trigger:2.35.2
- slack:2.34
- global-slack-notifier:1.5
- ansicolor:0.6.2
- simple-theme-plugin:0.5.1
- aws-bucket-credentials:1.0.0
- aws-credentials:1.28
- ssh-agent:1.17
# - blueocean:1.21.0
- basic-branch-build-strategies:1.3.2
- buildtriggerbadge:2.10
- rebuild:1.31
- ghprb:1.42.0
- antisamy-markup-formatter:1.5
- github-oauth:0.31
- role-strategy:2.15
# Enable to always override the installed plugins with the values of 'Master.InstallPlugins' on upgrade or redeployment.
# OverwritePlugins: true
# Enable HTML parsing using OWASP Markup Formatter Plugin (antisamy-markup-formatter), useful with ghprb plugin.
# The plugin is not installed by default, please update Master.InstallPlugins.
# EnableRawHtmlMarkupFormatter: true
# Used to approve a list of groovy functions in pipelines used the script-security plugin. Can be viewed under /scriptApproval
# ScriptApproval:
# - "method groovy.json.JsonSlurperClassic parseText java.lang.String"
# - "new groovy.json.JsonSlurperClassic"
# List of groovy init scripts to be executed during Jenkins master start
InitScripts:
# - |
# print 'adding global pipeline libraries, register properties, bootstrap jobs...'
# Kubernetes secret that contains a 'credentials.xml' for Jenkins
# CredentialsXmlSecret: jenkins-credentials
# Kubernetes secret that contains files to be put in the Jenkins 'secrets' directory,
# useful to manage encryption keys used for credentials.xml for instance (such as
# master.key and hudson.util.Secret)
# SecretsFilesSecret: jenkins-secrets
# Jenkins XML job configs to provision
# Jobs:
# test: |-
# <<xml here>>
# Below is the implementation of Jenkins Configuration as Code. Add a key under ConfigScripts for each configuration area,
# where each corresponds to a plugin or section of the UI. Each key (prior to | character) is just a label, and can be any value.
# Keys are only used to give the section a meaningful name. The only restriction is they may only contain RFC 1123 \ DNS label
# characters: lowercase letters, numbers, and hyphens. The keys become the name of a configuration yaml file on the master in
# /var/jenkins_home/casc_configs (by default) and will be processed by the Configuration as Code Plugin. The lines after each |
# become the content of the configuration yaml file. The first line after this is a JCasC root element, eg jenkins, credentials,
# etc. Best reference is https://<jenkins_url>/configuration-as-code/reference. The example below creates a welcome message:
JCasC:
enabled: false
PluginVersion: 1.5
SupportPluginVersion: 1.5
ConfigScripts:
welcome-message: |
jenkins:
systemMessage: Welcome to our CI\CD server. This Jenkins is configured and managed 'as code'.
Sidecars:
configAutoReload:
# If enabled: true, Jenkins Configuration as Code will be reloaded on-the-fly without a reboot. If false or not-specified,
# jcasc changes will cause a reboot and will only be applied at the subsequent start-up. Auto-reload uses the Jenkins CLI
# over SSH to reapply config when changes to the ConfigScripts are detected. The admin user (or account you specify in
# Master.AdminUser) will have a random SSH private key (RSA 4096) assigned unless you specify OwnSshKey: true. This will be saved to a k8s secret.
enabled: false
image: shadwell/k8s-sidecar:0.0.2
imagePullPolicy: IfNotPresent
resources:
# limits:
# cpu: 100m
# memory: 100Mi
# requests:
# cpu: 50m
# memory: 50Mi
# SSH port value can be set to any unused TCP port. The default, 1044, is a non-standard SSH port that has been chosen at random.
# Is only used to reload jcasc config from the sidecar container running in the Jenkins master pod.
# This TCP port will not be open in the pod (unless you specifically configure this), so Jenkins will not be
# accessible via SSH from outside of the pod. Note if you use non-root pod privileges (RunAsUser & FsGroup),
# this must be > 1024:
sshTcpPort: 1044
# label that the configmaps with dashboards are marked with:
label: jenkins_config
# folder in the pod that should hold the collected dashboards:
folder: /var/jenkins_home/casc_configs
# If specified, the sidecar will search for dashboard config-maps inside this namespace.
# Otherwise the namespace in which the sidecar is running will be used.
# It's also possible to specify ALL to search in all namespaces:
# searchNamespace:
# Allows you to inject additional/other sidecars
other:
## The example below runs the client for https://smee.io as sidecar container next to Jenkins,
## that allows to trigger build behind a secure firewall.
## https://jenkins.io/blog/2019/01/07/webhook-firewalls/#triggering-builds-with-webhooks-behind-a-secure-firewall
##
## Note: To use it you should go to https://smee.io/new and update the url to the generete one.
# - name: smee
# image: docker.io/twalter/smee-client:1.0.2
# args: ["--port", "{{ .Values.Master.ServicePort }}", "--path", "/github-webhook/", "--url", "https://smee.io/new"]
# resources:
# limits:
# cpu: 50m
# memory: 128Mi
# requests:
# cpu: 10m
# memory: 32Mi
# Node labels and tolerations for pod assignment
# ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector
# ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#taints-and-tolerations-beta-feature
NodeSelector: {}
Tolerations: {}
PodAnnotations: {}
# The below two configuration-related values are deprecated and replaced by Jenkins Configuration as Code (see above
# JCasC key). They will be deleted in an upcoming version.
CustomConfigMap: false
# By default, the configMap is only used to set the initial config the first time
# that the chart is installed. Setting `OverwriteConfig` to `true` will overwrite
# the jenkins config with the contents of the configMap every time the pod starts.
# This will also overwrite all init scripts
OverwriteConfig: false
ingress:
enabled: false
labels: {}
annotations: {}
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"
# Set this path to JenkinsUriPrefix above or use annotations to rewrite path
# path: "/jenkins"
tls:
# - secretName: jenkins.cluster.local
# hosts:
# - jenkins.cluster.local
AdditionalConfig: {}
# Master.HostAliases allows for adding entries to Pod /etc/hosts:
# https://kubernetes.io/docs/concepts/services-networking/add-entries-to-pod-etc-hosts-with-host-aliases/
HostAliases: []
# - ip: 192.168.50.50
# hostnames:
# - something.local
# - ip: 10.0.50.50
# hostnames:
# - other.local
Agent:
Enabled: true
Image: jenkins/jnlp-slave
ImageTag: 3.27-1
CustomJenkinsLabels: []
# ImagePullSecret: jenkins
Component: "jenkins-slave"
Privileged: false
resources:
requests:
cpu: "200m"
memory: "256Mi"
limits:
cpu: "200m"
memory: "256Mi"
# You may want to change this to true while testing a new image
AlwaysPullImage: false
# Controls how slave pods are retained after the Jenkins build completes
# Possible values: Always, Never, OnFailure
PodRetention: Never
# You can define the volumes that you want to mount for this container
# Allowed types are: ConfigMap, EmptyDir, HostPath, Nfs, Pod, Secret
# Configure the attributes as they appear in the corresponding Java class for that type
# https://github.com/jenkinsci/kubernetes-plugin/tree/master/src/main/java/org/csanchez/jenkins/plugins/kubernetes/volumes
# Pod-wide ennvironment, these vars are visible to any container in the slave pod
envVars:
# - name: PATH
# value: /usr/local/bin
volumes:
# - type: Secret
# secretName: mysecret
# mountPath: /var/myapp/mysecret
NodeSelector: {}
# Key Value selectors. Ex:
# jenkins-agent: v1
# Executed command when side container gets started
Command:
Args:
# Side container name
SideContainerName: jnlp
# Doesn't allocate pseudo TTY by default
TTYEnabled: false
# Max number of spawned agent
ContainerCap: 10
# Pod name
PodName: default
Persistence:
Enabled: true
## A manually managed Persistent Volume and Claim
## Requires Persistence.Enabled: true
## If defined, PVC must be created manually before volume will be bound
# ExistingClaim:
## jenkins data Persistent Volume Storage Class
## If defined, storageClassName: <storageClass>
## If set to "-", storageClassName: "", which disables dynamic provisioning
## If undefined (the default) or set to null, no storageClassName spec is
## set, choosing the default provisioner. (gp2 on AWS, standard on
## GKE, AWS & OpenStack)
##
# StorageClass: "-"
Annotations: {}
AccessMode: ReadWriteOnce
Size: 8Gi
volumes:
# - name: nothing
# emptyDir: {}
mounts:
# - mountPath: /var/nothing
# name: nothing
# readOnly: true
NetworkPolicy:
# Enable creation of NetworkPolicy resources.
Enabled: false
# For Kubernetes v1.4, v1.5 and v1.6, use 'extensions/v1beta1'
# For Kubernetes v1.7, use 'networking.k8s.io/v1'
ApiVersion: networking.k8s.io/v1
## Install Default RBAC roles and bindings
rbac:
install: false
serviceAccountName: default
# Role reference
roleRef: cluster-admin
# Role kind (Role or ClusterRole)
roleKind: ClusterRole
# Role binding kind (RoleBinding or ClusterRoleBinding)
roleBindingKind: ClusterRoleBinding
## Backup cronjob configuration
## Ref: https://github.com/nuvo/kube-tasks
backup:
# Backup must use RBAC
# So by enabling backup you are enabling RBAC specific for backup
enabled: false
# Schedule to run jobs. Must be in cron time format
# Ref: https://crontab.guru/
schedule: "0 2 * * *"
annotations:
# Example for authorization to AWS S3 using kube2iam
# Can also be done using environment variables
iam.amazonaws.com/role: jenkins
image:
repository: nuvo/kube-tasks
tag: 0.1.2
# Additional arguments for kube-tasks
# Ref: https://github.com/nuvo/kube-tasks#simple-backup
extraArgs: []
# Add additional environment variables
env:
# Example environment variable required for AWS credentials chain
- name: AWS_REGION
value: us-east-1
resources:
requests:
memory: 1Gi
cpu: 1
limits:
memory: 1Gi
cpu: 1
# Destination to store the backup artifacts
# Supported cloud storage services: AWS S3, Minio S3, Azure Blob Storage
# Additional support can added. Visit this repository for details
# Ref: https://github.com/nuvo/skbn
destination: s3://nuvo-jenkins-data/backup
我的 requirements.yaml 文件在图表中看起来像
dependencies:
- name: jenkins
version: 0.35.1
repository: https://kubernetes-charts.storage.googleapis.com/
- name: istio-components
version: 0.4.1
repository: s3://helm-chart-repository/
我尝试在 InstallPlugins 下更新插件版本并推送 helm chart 并列出它(我可以看到它已经更新)，但它对错误没有影响，就像插件没有更新在全部。
关于如何继续调试这个问题有什么建议吗？
最佳答案
我能够通过删除/var/jenkins_home 下的/plugins 文件夹然后重新启动我的 Jenkins pod 来解决这个问题。这创建了一个新的/plugins 文件夹，然后我不再有任何插件依赖性问题。
关于Jenkins helm chart 不更新插件，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/66062422/