dockerd: failed to start daemon: Devices cgroup isn‘t mounted

问题复现

有个项目在往测试服务器发布时报错了，

[root@Container 001.docker-project]# docker container run -d --volume /etc/localtime:/etc/localtime:ro --restart always --user root --name FZ_CaseCenter_Web --hostname FZ_CaseCenter_Web -p 8004:80 hub.nercoa.com/case-center/web:v1.3.6
44bf499c7f8b948f9705024406c00cb46c151eb7134e45e2bc5b361e3b8d5d62
docker: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: mountpoint for devices not found: unknown.

applying cgroup configuration for process caused: mountpoint for devices not found: unknown.

问题分析

查看服务运行状态及信息

[root@Container 001.docker-project]# systemctl restart docker
Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.
[root@Container 001.docker-project]# journalctl -xe
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit docker.service has finished shutting down.
Aug 24 13:45:28 Container systemd[1]: Starting Docker Application Container Engine...
-- Subject: Unit docker.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit docker.service has begun starting up.
Aug 24 13:45:28 Container dockerd[52230]: time="2021-08-24T13:45:28.227759932+08:00" level=info msg="Starting up"
Aug 24 13:45:28 Container dockerd[52230]: time="2021-08-24T13:45:28.228470643+08:00" level=warning msg="insecure registry http://harbor.nercoa.com should not contain 'http:
Aug 24 13:45:28 Container dockerd[52230]: time="2021-08-24T13:45:28.229599674+08:00" level=info msg="parsed scheme: \"unix\"" module=grpc
Aug 24 13:45:28 Container dockerd[52230]: time="2021-08-24T13:45:28.229632777+08:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Aug 24 13:45:28 Container dockerd[52230]: time="2021-08-24T13:45:28.229678411+08:00" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/cont
Aug 24 13:45:28 Container dockerd[52230]: time="2021-08-24T13:45:28.229707725+08:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Aug 24 13:45:28 Container dockerd[52230]: time="2021-08-24T13:45:28.231722114+08:00" level=info msg="parsed scheme: \"unix\"" module=grpc
Aug 24 13:45:28 Container dockerd[52230]: time="2021-08-24T13:45:28.231776964+08:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Aug 24 13:45:28 Container dockerd[52230]: time="2021-08-24T13:45:28.231825087+08:00" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/cont
Aug 24 13:45:28 Container dockerd[52230]: time="2021-08-24T13:45:28.231883312+08:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Aug 24 13:45:28 Container dockerd[52230]: time="2021-08-24T13:45:28.251139635+08:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
Aug 24 13:45:28 Container dockerd[52230]: time="2021-08-24T13:45:28.453402896+08:00" level=warning msg="Your kernel does not support cgroup memory limit"
Aug 24 13:45:28 Container dockerd[52230]: time="2021-08-24T13:45:28.453457019+08:00" level=warning msg="Unable to find cpu cgroup in mounts"
Aug 24 13:45:28 Container dockerd[52230]: time="2021-08-24T13:45:28.453472712+08:00" level=warning msg="Unable to find blkio cgroup in mounts"
Aug 24 13:45:28 Container dockerd[52230]: time="2021-08-24T13:45:28.453485270+08:00" level=warning msg="Unable to find cpuset cgroup in mounts"
Aug 24 13:45:28 Container dockerd[52230]: time="2021-08-24T13:45:28.453498322+08:00" level=warning msg="Unable to find pids cgroup in mounts"
Aug 24 13:45:28 Container dockerd[52230]: failed to start daemon: Devices cgroup isn't mounted   **### 注意这条错误信息**
Aug 24 13:45:28 Container systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Aug 24 13:45:28 Container systemd[1]: Failed to start Docker Application Container Engine.
-- Subject: Unit docker.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit docker.service has failed.
-- 
-- The result is failed.
Aug 24 13:45:28 Container systemd[1]: Unit docker.service entered failed state.
Aug 24 13:45:28 Container systemd[1]: docker.service failed.
lines 1809-1846/1846 (END)

看到有一条错误信息为Aug 24 13:45:28 Container dockerd[52230]: failed to start daemon: Devices cgroup isn't mounted

查看存储挂载情况

cgroup已经是挂载状态了，但是Docker Overlay没有挂载上。

由于上午操作过umount -a && mount -a命令及修改过/etc/fstab文件，应该是个操作导致的Docker异常。

问题解决

由于是测试服务器，我直接重启了服务器即处理完成了。

可以看到Docker Overlay驱动都工作正常了。。

网上也找到一篇文章使用脚本处理

注释：
set -e : 在脚本的前面加上这条命令，表示如果任何语句的执行结果不是true则应该退出。
如果要增加可读性，可以使用set -o errexit，它的作用与set -e相同。
-e 表示文件存在
moutpoint：判断指定的目录是否是挂载点

set -e   
# 这句是告诉bash如何有任何语句执行结果不为ture，就应该退出。 

if grep -v '^#' /etc/fstab | grep -q cgroup; then
	echo 'cgroups mounted from fstab, not mounting /sys/fs/cgroup'
	exit 0
fi

# kernel provides cgroups?
if [ ! -e /proc/cgroups ]; then
	exit 0
fi

# 确保目录存在
if [ ! -d /sys/fs/cgroup ]; then
	exit 0
fi

# mount /sys/fs/cgroup if not already done
if ! mountpoint -q /sys/fs/cgroup; then
	mount -t tmpfs -o uid=0,gid=0,mode=0755 cgroup /sys/fs/cgroup
fi

cd /sys/fs/cgroup

# get/mount list of enabled cgroup controllers
for sys in $(awk '!/^#/ { if ($4 == 1) print $1 }' /proc/cgroups); do
	mkdir -p $sys
	if ! mountpoint -q $sys; then
		if ! mount -n -t cgroup -o $sys cgroup $sys; then
			rmdir $sys || true
		fi
	fi
done


exit 0