배우면서 기록하기

Remove OSD/OSD-node from Ceph running in docker도커로 운영되는 Ceph에서 OSD/OSD노드를 제거하기Env.OS : RHEL 9 ~ / Ubuntu 20.04 ~Ceph : Octopus ~Ceph버전이 Octopus이상부터 docker방식으로 운영된다.* 위 버전은 운영해본 버전을 기재함.    Prep.운영중인 ceph-cluster의 영향이 없도록 불필요한 데이터 이동을 제한.root@mon0:~# ceph osd set nobackfillnobackfill is setroot@mon0:~# ceph osd set nooutnoout is setroot@mon0:~# ceph osd set norecovernorecover is set 제거할 OSD노드에서 서..

Symptoms..블록 스토리지를 삭제하는 과정에서 아래와 같은 에러가 발생.rbd: error: image still has watcherslibrbd::image::RemoveRequest: 0x55ce6659a5c0 check_image_watchers: image has watchers - not removingRemoving image: 0% complete...failed.rbd: error: image still has watchersThis means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to t..

Symptoms..ceph -s 로 확인하면.root@mon0:~# ceph -s cluster: id: [fsid] health: HEALTH_WARN 1 pgs not deep-scrubbed in time 여기서 좀 더 자세하게 확인하기 위해 ceph health detail 로 확인하면 아래와 같이 어떤 pg_id인지 확인 할 수 있다.root@mon0:~# ceph health detail HEALTH_WARN 1 pgs not deep-scrubbed in time[WRN] PG_NOT_DEEP_SCRUBBED: 1 pgs not deep-scrubbed in time pg [pg_id] not deep-scrubbed since [date]Toub..

Symptom..ceph -s 로 상태 확인root@mon0:~# ceph -s cluster: id: [fsid] health: HEALTH_WARN N daemons have recently crashed crash 확인root@mon0:~# ceph crash lsID ENTITY NEW...2024-11-22T10:30:06.857696Z_6667e9da-d97e-4d86-898e-2d1d2e771fc6 [mon] *...Touble_shooting..crash가 되었다는 알람을 확인하고 처리하는 명령어 archive를 사용root@mon0:~# ceph crash archive [ID]ex)ceph crash archive 20..

Symptom..ceph -s 로 간단한 상태 확인root@mon0:~# ceph -s cluster: id: [fsid] health: HEALTH_WARN 1 failed cephadm daemon(s) 1/3 mons down services: mon: 3 daemon, quorum [mon0], [mon1], out of quorum: [mon2] ceph health detail로 증상 확인root@mon:~# ceph health detailHEALTH_WARN 1 failed cephadm daemon(s);[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) daemon mo..

Symptom..ceph -s로 간단한 상태 확인root@mon0:~# ceph -s cluster: id: [fsid] health: HEALTH_WARN 4 hosts fail cephadm check ceph health detail로 증상 확인root@mon0:~# ceph health detailHEALTH_WARN 4 hosts fail cephadm check[WRN] CEPHADM_HOST_CHECK_FAILED: 4 hosts fail cephadm check host osd2 ([osd2_ip]) failed check: Can't communicate with remote host `[osd2_ip]`, possibly because pyt..