[Ceph] failed cephadm daemon(s) out of quorum

Symptom..

ceph -s 로 간단한 상태 확인

root@mon0:~# ceph -s
  cluster:
    id:     [fsid]
    health: HEALTH_WARN
            1 failed cephadm daemon(s)
            1/3 mons down
  
  services:
    mon: 3 daemon, quorum [mon0], [mon1], out of quorum: [mon2]

 

ceph health detail로 증상 확인

root@mon:~# ceph health detail
HEALTH_WARN 1 failed cephadm daemon(s);
[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s)
	daemon mon on [mon2] is in error state

 

mon서버 한대가 서비스 다운되어 warning이 뜸..

반응형

 

Trouble_shooting..

서비스 중지, 재시작 등을 해도 실패함

Failed to start Ceph mon.[mon2] for [fsid]
A start job for unit ceph-[fsid]@mon.[mon2].service has finished with a failure.

 

ceph 데몬을 삭제후 재시작 진행

root@mon:~# ceph orch daemon rm mon.[mon2] --force
Removed mon.[mon2] from host '[mon2]'

root@mon:~# ceph orch daemon start mon.[mon2]
Scheduled to start mon.[mon2] on host '[mon2]'

728x90

 

Verifying..

ceph -s 로 상태 확인

root@mon0:~# ceph -s
  cluster:
    id:     [fsid]
    health: HEALTH_OK
  
  services:
    mon: 3 daemon, quorum [mon0],[mon1],[mon2]

 

서비스도 정상 확인

root@mon0:~# systemctl status ceph-[fsid]@mon.[mon2].service
ceph-[fsid]@mon.[mon2].service
	Loaded: loaded
    Active: active (running)