释放双眼，带上耳机，听听看~！
当我们直接执行containerd的时候，提示下面的错误。并且会影响kubelet启动失败
[root@k8s-node2 ~]# containerd
INFO[2023-08-02T15:00:36.034761099+08:00] skip loading plugin "io.containerd.tracing.processor.v1.otlp"... error="no OpenTelemetry endpoint: skip plugin" type=io.containerd.tracing.processor.v1
INFO[2023-08-02T15:00:36.034777657+08:00] loading plugin "io.containerd.internal.v1.tracing"... type=io.containerd.internal.v1
ERRO[2023-08-02T15:00:36.034820837+08:00] failed to initialize a tracing processor "otlp" error="no OpenTelemetry endpoint: skip plugin"
INFO[2023-08-02T15:00:36.034870488+08:00] loading plugin "io.containerd.grpc.v1.cri"... type=io.containerd.grpc.v1
INFO[2023-08-02T15:00:36.035178573+08:00] Start cri plugin with config {PluginConfig:{ContainerdConfig:{Snapshotter:overlayfs DefaultRuntimeName:runc DefaultRuntime:{Type: Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[] PrivilegedWithoutHostDevices:false BaseRuntimeSpec: NetworkPluginConfDir: NetworkPluginMaxConfNum:0} UntrustedWorkloadRuntime:{Type: Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[] PrivilegedWithoutHostDevices:false BaseRuntimeSpec: NetworkPluginConfDir: NetworkPluginMaxConfNum:0} Runtimes:map[runc:{Type:io.containerd.runc.v2 Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[BinaryName: CriuImagePath: CriuPath: CriuWorkPath: IoGid:0 IoUid:0 NoNewKeyring:false NoPivotRoot:false Root: ShimCgroup: SystemdCgroup:true] PrivilegedWithoutHostDevices:false BaseRuntimeSpec: NetworkPluginConfDir: NetworkPluginMaxConfNum:0}] NoPivot:false DisableSnapshotAnnotations:true DiscardUnpackedLayers:false IgnoreRdtNotEnabledErrors:false} CniConfig:{NetworkPluginBinDir:/opt/cni/bin NetworkPluginConfDir:/etc/cni/net.d NetworkPluginMaxConfNum:1 NetworkPluginConfTemplate: IPPreference:} Registry:{ConfigPath: Mirrors:map[] Configs:map[] Auths:map[] Headers:map[]} ImageDecryption:{KeyModel:node} DisableTCPService:true StreamServerAddress:127.0.0.1 StreamServerPort:0 StreamIdleTimeout:4h0m0s EnableSelinux:false SelinuxCategoryRange:1024 SandboxImage:registry.cn-beijing.aliyuncs.com/abcdocker/pause:3.6 StatsCollectPeriod:10 SystemdCgroup:false EnableTLSStreaming:false X509KeyPairStreaming:{TLSCertFile: TLSKeyFile:} MaxContainerLogLineSize:16384 DisableCgroup:false DisableApparmor:false RestrictOOMScoreAdj:false MaxConcurrentDownloads:3 DisableProcMount:false UnsetSeccompProfile: TolerateMissingHugetlbController:true DisableHugetlbController:true DeviceOwnershipFromSecurityContext:false IgnoreImageDefinedVolumes:false NetNSMountsUnderStateDir:false EnableUnprivilegedPorts:false EnableUnprivilegedICMP:false} ContainerdRootDir:/var/lib/containerd ContainerdEndpoint:/run/containerd/containerd.sock RootDir:/var/lib/containerd/io.containerd.grpc.v1.cri StateDir:/run/containerd/io.containerd.grpc.v1.cri}
INFO[2023-08-02T15:00:36.035236257+08:00] Connect containerd service
INFO[2023-08-02T15:00:36.035283564+08:00] Get image filesystem path "/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs"
INFO[2023-08-02T15:00:36.036205794+08:00] Start subscribing containerd event
INFO[2023-08-02T15:00:36.036245684+08:00] serving... address=/run/containerd/containerd.sock.ttrpc
INFO[2023-08-02T15:00:36.036271301+08:00] Start recovering state
INFO[2023-08-02T15:00:36.036310941+08:00] serving... address=/run/containerd/containerd.sock
INFO[2023-08-02T15:00:36.036347849+08:00] containerd successfully booted in 0.030414s
FATA[2023-08-02T15:00:36.052281827+08:00] Failed to run CRI service error="failed to recover state: failed to reserve sandbox name \"loki-0_middleware_2ede5a97-2fdc-4cac-b670-7eec5309f771_1\": name \"loki-0_middleware_2ede5a97-2fdc-4cac-b670-7eec5309f771_1\" is reserved for \"34ce1e016c25483acc01d779a79188b50cb97f7ff253b064a2988a773f7c5d3d\""Kubelet错误日志如下:
Aug 02 14:18:03 k8s-node2 kubelet[30409]: QOSReserved=true|false (ALPHA - default=false)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: ReadWriteOncePod=true|false (ALPHA - default=false)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: RecoverVolumeExpansionFailure=true|false (ALPHA - default=false)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: RemainingItemCount=true|false (BETA - default=true)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: RotateKubeletServerCertificate=true|false (BETA - default=true)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: SeccompDefault=true|false (ALPHA - default=false)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: ServerSideFieldValidation=true|false (ALPHA - default=false)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: ServiceIPStaticSubrange=true|false (ALPHA - default=false)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: ServiceInternalTrafficPolicy=true|false (BETA - default=true)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: SizeMemoryBackedVolumes=true|false (BETA - default=true)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: StatefulSetAutoDeletePVC=true|false (ALPHA - default=false)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: StatefulSetMinReadySeconds=true|false (BETA - default=true)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: StorageVersionAPI=true|false (ALPHA - default=false)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: StorageVersionHash=true|false (BETA - default=true)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: TopologyAwareHints=true|false (BETA - default=true)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: TopologyManager=true|false (BETA - default=true)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: VolumeCapacityPriority=true|false (ALPHA - default=false)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: WinDSR=true|false (ALPHA - default=false)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: WinOverlay=true|false (BETA - default=true)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: WindowsHostProcessContainers=true|false (BETA - default=true) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --file-check-frequency duration Duration between checking config files for new data (default 20s) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --global-housekeeping-interval duration Interval between global housekeepings (default 1m0s) (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --hairpin-mode string How should the kubelet setup hairpin NAT. This allows endpoints of a Service to loadbalance back to themselves if they should try to access their own Service. Valid values are "promiscuous-bridge", "hairpin-veth" and "none". (default "promiscuous-bridge") (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --healthz-bind-address ip The IP address for the healthz server to serve on (set to '0.0.0.0' or '::' for listening in all interfaces and IP families) (default 127.0.0.1) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --healthz-port int32 The port of the localhost healthz endpoint (set to 0 to disable) (default 10248) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: -h, --help help for kubelet
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --hostname-override string If non-empty, will use this string as identification instead of the actual hostname. If --cloud-provider is set, the cloud provider determines the name of the node (consult cloud provider documentation to determine if and how the hostname is used).
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --housekeeping-interval duration Interval between container housekeepings (default 10s)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --http-check-frequency duration Duration between checking http for new data (default 20s) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --image-credential-provider-bin-dir string The path to the directory where credential provider plugin binaries are located.
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --image-credential-provider-config string The path to the credential provider plugin config file.
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --image-gc-high-threshold int32 The percent of disk usage after which image garbage collection is always run. Values must be within the range [0, 100], To disable image garbage collection, set to 100. (default 85) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --image-gc-low-threshold int32 The percent of disk usage before which image garbage collection is never run. Lowest disk usage to garbage collect to. Values must be within the range [0, 100] and should not be larger than that of --image-gc-high-threshold. (default 80) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --image-service-endpoint string The endpoint of remote image service. If not specified, it will be the same with --container-runtime-endpoint by default. Unix Domain Socket are supported on Linux, while npipe and tcp endpoints are supported on Windows. Examples:'unix:///path/to/runtime.sock', 'npipe:////./pipe/runtime'
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --iptables-drop-bit int32 The bit of the fwmark space to mark packets for dropping. Must be within the range [0, 31]. (default 15) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --iptables-masquerade-bit int32 The bit of the fwmark space to mark packets for SNAT. Must be within the range [0, 31]. Please match this parameter with corresponding parameter in kube-proxy. (default 14) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --keep-terminated-pod-volumes Keep terminated pod volumes mounted to the node after the pod terminates. Can be useful for debugging volume related issues. (DEPRECATED: will be removed in a future version)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --kernel-memcg-notification If enabled, the kubelet will integrate with the kernel memcg notification to determine if memory eviction thresholds are crossed rather than polling. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --kube-api-burst int32 Burst to use while talking with kubernetes apiserver. The number must be >= 0. If 0 will use DefaultBurst: 10. Doesn't cover events and node heartbeat apis which rate limiting is controlled by a different set of flags (default 10) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --kube-api-content-type string Content type of requests sent to apiserver. (default "application/vnd.kubernetes.protobuf") (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --kube-api-qps int32 QPS to use while talking with kubernetes apiserver. The number must be >= 0. If 0 will use DefaultQPS: 5. Doesn't cover events and node heartbeat apis which rate limiting is controlled by a different set of flags (default 5) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --kube-reserved mapStringString A set of ResourceName=ResourceQuantity (e.g. cpu=200m,memory=500Mi,ephemeral-storage=1Gi) pairs that describe resources reserved for kubernetes system components. Currently cpu, memory and local ephemeral storage for root file system are supported. See https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ for more detail. [default=none] (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --kube-reserved-cgroup string Absolute name of the top level cgroup that is used to manage kubernetes components for which compute resources were reserved via '--kube-reserved' flag. Ex. '/kube-reserved'. [default=''] (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --kubeconfig string Path to a kubeconfig file, specifying how to connect to the API server. Providing --kubeconfig enables API server mode, omitting --kubeconfig enables standalone mode.
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --kubelet-cgroups string Optional absolute name of cgroups to create and run the Kubelet in. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --lock-file string <Warning: Alpha feature> The path to file for kubelet to use as a lock file.
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --log-backtrace-at traceLocation when logging hits line file:N, emit a stack trace (default :0) (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --log-cadvisor-usage Whether to log the usage of the cAdvisor container (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --log-dir string If non-empty, write log files in this directory (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --log-file string If non-empty, use this log file (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --log-file-max-size uint Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800) (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --log-flush-frequency duration Maximum number of seconds between log flushes (default 5s)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --log-json-info-buffer-size quantity [Experimental] In JSON format with split output streams, the info messages can be buffered for a while to increase performance. The default value of zero bytes disables buffering. The size can be specified as number of bytes (512), multiples of 1000 (1K), multiples of 1024 (2Ki), or powers of those (3M, 4G, 5Mi, 6Gi). (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --log-json-split-stream [Experimental] In JSON format, write error messages to stderr and info messages to stdout. The default is to write a single stream to stdout. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --logging-format string Sets the log format. Permitted formats: "json", "text".
Aug 02 14:18:03 k8s-node2 kubelet[30409]: Non-default formats don't honor these flags: --add-dir-header, --alsologtostderr, --log-backtrace-at, --log-dir, --log-file, --log-file-max-size, --logtostderr, --one-output, --skip-headers, --skip-log-headers, --stderrthreshold, --vmodule.
Aug 02 14:18:03 k8s-node2 kubelet[30409]: Non-default choices are currently alpha and subject to change without warning. (default "text") (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --logtostderr log to standard error instead of files (default true) (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --machine-id-file string Comma-separated list of files to check for machine-id. Use the first one that exists. (default "/etc/machine-id,/var/lib/dbus/machine-id") (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --make-iptables-util-chains If true, kubelet will ensure iptables utility rules are present on host. (default true) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --manifest-url string URL for accessing additional Pod specifications to run (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --manifest-url-header colonSeparatedMultimapStringString Comma-separated list of HTTP headers to use when accessing the url provided to --manifest-url. Multiple headers with the same name will be added in the same order provided. This flag can be repeatedly invoked. For example: --manifest-url-header 'a:hello,b:again,c:world' --manifest-url-header 'b:beautiful' (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --master-service-namespace string The namespace from which the kubernetes master services should be injected into pods (default "default") (DEPRECATED: This flag will be removed in a future version.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --max-open-files int Number of files that can be opened by Kubelet process. (default 1000000) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --max-pods int32 Number of Pods that can run on this Kubelet. (default 110) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --maximum-dead-containers int32 Maximum number of old instances of containers to retain globally. Each container takes up some disk space. To disable, set to a negative number. (default -1) (DEPRECATED: Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --maximum-dead-containers-per-container int32 Maximum number of old instances to retain per container. Each container takes up some disk space. (default 1) (DEPRECATED: Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --memory-manager-policy string Memory Manager policy to use. Possible values: 'None', 'Static'. (default "None") (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --minimum-container-ttl-duration duration Minimum age for a finished container before it is garbage collected. Examples: '300ms', '10s' or '2h45m' (DEPRECATED: Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --minimum-image-ttl-duration duration Minimum age for an unused image before it is garbage collected. Examples: '300ms', '10s' or '2h45m'. (default 2m0s) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --node-ip string IP address (or comma-separated dual-stack IP addresses) of the node. If unset, kubelet will use the node's default IPv4 address, if any, or its default IPv6 address if it has no IPv4 addresses. You can pass '::' to make it prefer the default IPv6 address rather than the default IPv4 address.
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --node-labels mapStringString <Warning: Alpha feature> Labels to add when registering the node in the cluster. Labels must be key=value pairs separated by ','. Labels in the 'kubernetes.io' namespace must begin with an allowed prefix (kubelet.kubernetes.io, node.kubernetes.io) or be in the specifically allowed set (beta.kubernetes.io/arch, beta.kubernetes.io/instance-type, beta.kubernetes.io/os, failure-domain.beta.kubernetes.io/region, failure-domain.beta.kubernetes.io/zone, kubernetes.io/arch, kubernetes.io/hostname, kubernetes.io/os, node.kubernetes.io/instance-type, topology.kubernetes.io/region, topology.kubernetes.io/zone)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --node-status-max-images int32 The maximum number of images to report in Node.Status.Images. If -1 is specified, no cap will be applied. (default 50) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --node-status-update-frequency duration Specifies how often kubelet posts node status to master. Note: be cautious when changing the constant, it must work with nodeMonitorGracePeriod in nodecontroller. (default 10s) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --one-output If true, only write logs to their native severity level (vs also writing to each lower severity level) (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --oom-score-adj int32 The oom-score-adj value for kubelet process. Values must be within the range [-1000, 1000] (default -999) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --pod-cidr string The CIDR to use for pod IP addresses, only used in standalone mode. In cluster mode, this is obtained from the master. For IPv6, the maximum number of IP's allocated is 65536 (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --pod-infra-container-image string Specified image will not be pruned by the image garbage collector. CRI implementations have their own configuration to set this image. (default "k8s.gcr.io/pause:3.7") (DEPRECATED: will be removed in 1.27. Image garbage collector will get sandbox image information from CRI.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --pod-manifest-path string Path to the directory containing static pod files to run, or the path to a single static pod file. Files starting with dots will be ignored. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --pod-max-pids int Set the maximum number of processes per pod. If -1, the kubelet defaults to the node allocatable pid capacity. (default -1) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --pods-per-core int32 Number of Pods per core that can run on this Kubelet. The total number of Pods on this Kubelet cannot exceed max-pods, so max-pods will be used if this calculation results in a larger number of Pods allowed on the Kubelet. A value of 0 disables this limit. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --port int32 The port for the Kubelet to serve on. (default 10250) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --protect-kernel-defaults Default kubelet behaviour for kernel tuning. If set, kubelet errors if any of kernel tunables is different than kubelet defaults. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --provider-id string Unique identifier for identifying the node in a machine database, i.e cloudprovider (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --qos-reserved mapStringString <Warning: Alpha feature> A set of ResourceName=Percentage (e.g. memory=50%) pairs that describe how pod resource requests are reserved at the QoS level. Currently only memory is supported. Requires the QOSReserved feature gate to be enabled. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --read-only-port int32 The read-only port for the Kubelet to serve on with no authentication/authorization (set to 0 to disable) (default 10255) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --register-node Register the node with the apiserver. If --kubeconfig is not provided, this flag is irrelevant, as the Kubelet won't have an apiserver to register with. (default true) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --register-schedulable Register the node as schedulable. Won't have any effect if register-node is false. (default true) (DEPRECATED: will be removed in a future version)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --register-with-taints []v1.Taint Register the node with the given list of taints (comma separated "<key>=<value>:<effect>"). No-op if register-node is false. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --registry-burst int32 Maximum size of a bursty pulls, temporarily allows pulls to burst to this number, while still not exceeding registry-qps. Only used if --registry-qps > 0 (default 10) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --registry-qps int32 If > 0, limit registry pull QPS to this value. If 0, unlimited. (default 5) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --reserved-cpus string A comma-separated list of CPUs or CPU ranges that are reserved for system and kubernetes usage. This specific list will supersede cpu counts in --system-reserved and --kube-reserved. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --reserved-memory reserved-memory A comma separated list of memory reservations for NUMA nodes. (e.g. --reserved-memory 0:memory=1Gi,hugepages-1M=2Gi --reserved-memory 1:memory=2Gi). The total sum for each memory type should be equal to the sum of kube-reserved, system-reserved and eviction-threshold. See https://kubernetes.io/docs/tasks/administer-cluster/memory-manager/#reserved-memory-flag for more details. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --resolv-conf string Resolver configuration file used as the basis for the container DNS resolution configuration. (default "/etc/resolv.conf") (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --root-dir string Directory path for managing kubelet files (volume mounts,etc). (default "/var/lib/kubelet")
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --rotate-certificates <Warning: Beta feature> Auto rotate the kubelet client certificates by requesting new certificates from the kube-apiserver when the certificate expiration approaches. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --rotate-server-certificates Auto-request and rotate the kubelet serving certificates by requesting new certificates from the kube-apiserver when the certificate expiration approaches. Requires the RotateKubeletServerCertificate feature gate to be enabled, and approval of the submitted CertificateSigningRequest objects. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --runonce If true, exit after spawning pods from static pod files or remote urls. Exclusive with --enable-server (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --runtime-cgroups string Optional absolute name of cgroups to create and run the runtime in.
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --runtime-request-timeout duration Timeout of all runtime requests except long running request - pull, logs, exec and attach. When timeout exceeded, kubelet will cancel the request, throw out an error and retry later. (default 2m0s) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --seccomp-default RuntimeDefault <Warning: Alpha feature> Enable the use of RuntimeDefault as the default seccomp profile for all workloads. The SeccompDefault feature gate must be enabled to allow this flag, which is disabled per default.
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --serialize-image-pulls Pull images one at a time. We recommend *not* changing the default value on nodes that run docker daemon with version < 1.9 or an Aufs storage backend. Issue #10959 has more details. (default true) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --skip-headers If true, avoid header prefixes in the log messages (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --skip-log-headers If true, avoid headers when opening log files (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --stderrthreshold severity logs at or above this threshold go to stderr (default 2) (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --storage-driver-buffer-duration duration Writes in the storage driver will be buffered for this duration, and committed to the non memory backends as a single transaction (default 1m0s) (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --storage-driver-db string database name (default "cadvisor") (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --storage-driver-host string database host:port (default "localhost:8086") (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --storage-driver-password string database password (default "root") (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --storage-driver-secure use secure connection with database (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --storage-driver-table string table name (default "stats") (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --storage-driver-user string database username (default "root") (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --streaming-connection-idle-timeout duration Maximum time a streaming connection can be idle before the connection is automatically closed. 0 indicates no timeout. Example: '5m'. Note: All connections to the kubelet server have a maximum duration of 4 hours. (default 4h0m0s) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --sync-frequency duration Max period between synchronizing running containers and config (default 1m0s) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --system-cgroups string Optional absolute name of cgroups in which to place all non-kernel processes that are not already inside a cgroup under '/'. Empty for no container. Rolling back the flag requires a reboot. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --system-reserved mapStringString A set of ResourceName=ResourceQuantity (e.g. cpu=200m,memory=500Mi,ephemeral-storage=1Gi) pairs that describe resources reserved for non-kubernetes components. Currently only cpu and memory are supported. See https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ for more detail. [default=none] (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --system-reserved-cgroup string Absolute name of the top level cgroup that is used to manage non-kubernetes components for which compute resources were reserved via '--system-reserved' flag. Ex. '/system-reserved'. [default=''] (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --tls-cert-file string File containing x509 Certificate used for serving HTTPS (with intermediate certs, if any, concatenated after server cert). If --tls-cert-file and --tls-private-key-file are not provided, a self-signed certificate and key are generated for the public address and saved to the directory passed to --cert-dir. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --tls-cipher-suites strings Comma-separated list of cipher suites for the server. If omitted, the default Go cipher suites will be used.
Aug 02 14:18:03 k8s-node2 kubelet[30409]: Preferred values: TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384.
Aug 02 14:18:03 k8s-node2 kubelet[30409]: Insecure values: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_3DES_EDE_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_RC4_128_SHA. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --tls-min-version string Minimum TLS version supported. Possible values: VersionTLS10, VersionTLS11, VersionTLS12, VersionTLS13 (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --tls-private-key-file string File containing x509 private key matching --tls-cert-file. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --topology-manager-policy string Topology Manager policy to use. Possible values: 'none', 'best-effort', 'restricted', 'single-numa-node'. (default "none") (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --topology-manager-scope string Scope to which topology hints applied. Topology Manager collects hints from Hint Providers and applies them to defined scope to ensure the pod admission. Possible values: 'container', 'pod'. (default "container") (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: -v, --v Level number for the log level verbosity
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --version version[=true] Print version information and quit
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --vmodule pattern=N,... comma-separated list of pattern=N settings for file-filtered logging (only works for text log format)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --volume-plugin-dir string The full path of the directory in which to search for additional third party volume plugins (default "/usr/libexec/kubernetes/kubelet-plugins/volume/exec/") (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 02 14:18:03 k8s-node2 kubelet[30409]: --volume-stats-agg-period duration Specifies interval for kubelet to calculate and cache the volume disk usage for all pods and volumes. To disable volume calculations, set to a negative number. (default 1m0s) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
^C
[root@k8s-node2 kubernetes]#原因: containerd容器数据已损坏，因此解决此问题的方法是通过执行以下操作重新创建容器数据
[root@k8s-node2 ~]# ls cd /var/lib/containerd/
[root@k8s-node2 ~]# ls /var/lib/containerd/
io.containerd.content.v1.content io.containerd.metadata.v1.bolt io.containerd.runtime.v2.task io.containerd.snapshotter.v1.native old
io.containerd.grpc.v1.cri io.containerd.runtime.v1.linux io.containerd.snapshotter.v1.btrfs io.containerd.snapshotter.v1.overlayfs tmpmounts
[root@k8s-node2 ~]# mkdir old
[root@k8s-node2 ~]# ls mv * old/
[root@k8s-node2 ~]# reboot #当然也可以直接restart containerd此时我们node节点也正常恢复
[root@k8s-01 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-01 Ready control-plane 84d v1.24.0
k8s-02 Ready control-plane 84d v1.24.0
k8s-03 Ready control-plane 84d v1.24.0
k8s-node1 Ready <none> 84d v1.24.0
k8s-node2 Ready <none> 84d v1.24.0
