In the examples used for this procedure, “storage3” will be the targeted node for removal.
-
Find the connection between the bare metal server and the overcloud server. The output from the command below shows us that the serer we are looking for is “overcloud8st-cephstorageblue1-0”. This information will be used later in the procedure.
(undercloud) [stack@undercloud ~]$ openstack ccloud nodemap list
+---------------------------------+----------------+------------+----------------+
| Name | IP | Hypervisor | Hypervisor IP |
+---------------------------------+----------------+------------+----------------+
| overcloud8st-cc-2 | 192.168.213.54 | controler2 | 192.168.213.6 |
| overcloud8st-cc-1 | 192.168.213.51 | controler3 | 192.168.213.7 |
| overcloud8st-cc-0 | 192.168.213.55 | controler1 | 192.168.213.5 |
| overcloud8st-ca-1 | 192.168.213.72 | controler1 | 192.168.213.5 |
| overcloud8st-ca-0 | 192.168.213.60 | controler3 | 192.168.213.7 |
| overcloud8st-ca-2 | 192.168.213.53 | controler2 | 192.168.213.6 |
| overcloud8st-cadb-1 | 192.168.213.71 | controler1 | 192.168.213.5 |
| overcloud8st-afxctrl-0 | 192.168.213.69 | controler2 | 192.168.213.6 |
| overcloud8st-afxctrl-1 | 192.168.213.52 | controler3 | 192.168.213.7 |
| overcloud8st-afxctrl-2 | 192.168.213.58 | controler1 | 192.168.213.5 |
| overcloud8st-cadb-0 | 192.168.213.65 | controler2 | 192.168.213.6 |
| overcloud8st-ctrl-0 | 192.168.213.73 | controler2 | 192.168.213.6 |
| overcloud8st-ctrl-1 | 192.168.213.63 | controler1 | 192.168.213.5 |
| overcloud8st-ctrl-2 | 192.168.213.59 | controler3 | 192.168.213.7 |
| overcloud8st-cephstorageblue1-0 | 192.168.213.62 | storage3 | 192.168.213.62 |
| overcloud8st-compdpdk-0 | 192.168.213.56 | compute1 | 192.168.213.56 |
| overcloud8st-cephstorageblue2-0 | 192.168.213.61 | storage2 | 192.168.213.61 |
| overcloud8st-cephstorageblue2-1 | 192.168.213.80 | storage1 | 192.168.213.80 |
| overcloud8st-cadb-2 | 192.168.213.74 | controler3 | 192.168.213.7 |
+---------------------------------+----------------+------------+----------------+
-
Log in as root user to any of the openstack controllers and verify that the Ceph cluster is healthy:
[root@overcloud8st-ctrl-1 ~]# ceph -s
cluster:
id: a98b1580-bb97-11ea-9f2b-525400882160
health: HEALTH_OK
-
Find the OSDs that reside on the server to be removed (overcloud8st-cephstorageblue1-0). We identify osd.2, osd.3, osd.6, and osd.7 from the example below:
[root@overcloud8st-ctrl-1 ~]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 10.91638 root default
-3 3.63879 host overcloud8st-cephstorageblue1-0
2 hdd 0.90970 osd.2 up 1.00000 1.00000
3 hdd 0.90970 osd.3 up 1.00000 1.00000
6 hdd 0.90970 osd.6 up 1.00000 1.00000
7 hdd 0.90970 osd.7 up 1.00000 1.00000
-7 3.63879 host overcloud8st-cephstorageblue2-0
1 hdd 0.90970 osd.1 up 1.00000 1.00000
4 hdd 0.90970 osd.4 up 1.00000 1.00000
8 hdd 0.90970 osd.8 up 1.00000 1.00000
10 hdd 0.90970 osd.10 up 1.00000 1.00000
-5 3.63879 host overcloud8st-cephstorageblue2-1
0 hdd 0.90970 osd.0 up 1.00000 1.00000
5 hdd 0.90970 osd.5 up 1.00000 1.00000
9 hdd 0.90970 osd.9 up 1.00000 1.00000
11 hdd 0.90970 osd.11 up 1.00000 1.00000
-
Remove osd.2, osd.3, osd.6, and osd.7 while still logged in to the openstack controller:
[root@overcloud8st-ctrl-1 ~]# ceph osd out 2
marked out osd.2.
[root@overcloud8st-ctrl-1 ~]# ceph osd out 3
marked out osd.3.
[root@overcloud8st-ctrl-1 ~]# ceph osd out 6
marked out osd.6.
[root@overcloud8st-ctrl-1 ~]# ceph osd out 7
marked out osd.7.
-
Moving forward, check to see that the cluster health returns to “health_ok” state by using the ceph -s command. Check for cluster health after every operation.
-
Log in to the Ceph node (overcloud8st-cephstorageblue1-0) to stop the OSD services:
[root@overcloud8st-cephstorageblue1-0 ~]# systemctl stop ceph-osd@2.service
[root@overcloud8st-cephstorageblue1-0 ~]# systemctl stop ceph-osd@3.service
[root@overcloud8st-cephstorageblue1-0 ~]# systemctl stop ceph-osd@6.service
[root@overcloud8st-cephstorageblue1-0 ~]# systemctl stop ceph-osd@7.service
-
Log back into the controller and remove further information about the OSDs from overcloud8st-cephstorageblue1-0:
[root@overcloud8st-ctrl-1 ~]# ceph osd crush remove osd.2
removed item id 2 name 'osd.2' from crush map
[root@overcloud8st-ctrl-1 ~]# ceph osd crush remove osd.3
removed item id 3 name 'osd.3' from crush map
[root@overcloud8st-ctrl-1 ~]# ceph osd crush remove osd.6
removed item id 6 name 'osd.6' from crush map
[root@overcloud8st-ctrl-1 ~]# ceph osd crush remove osd.7
removed item id 7 name 'osd.7' from crush map
[root@overcloud8st-ctrl-1 ~]# ceph auth del osd.2
updated
[root@overcloud8st-ctrl-1 ~]# ceph auth del osd.3
updated
[root@overcloud8st-ctrl-1 ~]# ceph auth del osd.6
updated
[root@overcloud8st-ctrl-1 ~]# ceph auth del osd.7
updated
[root@overcloud8st-ctrl-1 ~]# ceph osd rm 2
removed osd.2
[root@overcloud8st-ctrl-1 ~]# ceph osd rm 3
removed osd.3
[root@overcloud8st-ctrl-1 ~]# ceph osd rm 6
removed osd.6
[root@overcloud8st-ctrl-1 ~]# ceph osd rm 7
removed osd.7
[root@overcloud8st-ctrl-1 ~]# ceph osd crush rm overcloud8st-cephstorageblue1-0
-
From the undercloud VM find the ID of the Ceph storage node:
(undercloud) [stack@undercloud ~]$ openstack server list | grep overcloud8st-cephstorageblue1-0
| 7ee9be4f-efda-4837-a597-a6554027d0c9 | overcloud8st-cephstorageblue1-0 | ACTIVE | ctlplane=192.168.213.62 | overcloud-full | CephStorageBlue1
-
Initiate a removal using the node ID from the previous step:
(undercloud) [stack@undercloud ~]$ openstack overcloud node delete --stack overcloud 7ee9be4f-efda-4837-a597-a6554027d0c9
-
Verify that the bare metal node is in a state of power off and available:
(undercloud) [stack@undercloud ~]$ openstack baremetal node list | grep storage3
| 05bbab4b-b968-4d1d-87bc-a26ac335303d | storage3 | None | power off | available | False |
-
From the jump host as the contrail user mark the storage node with ‘status: deleting’ so the Ceph profile will be removed from it. Do this by adding the ‘status: deleting’ to the storage-nodes.yml file for storage3 and then run the script storage-nodes-assign.sh.
[contrail@5a6s13-node1 contrail_cloud]$ cat config/storage-nodes.yml
storage_nodes:
- name: storage1
profile: blue2
- name: storage2
profile: blue2
- name: storage3
profile: blue1
status: deleting
[contrail@5a6s13-node1 contrail_cloud]$ ./scripts/storage-nodes-assign.sh
-
Run openstack-deploy.sh to regenerate the templates to reflect the current state:
[contrail@5a6s13-node1 contrail_cloud]$ ./scripts/openstack-deploy.sh
-
If the goal is to remove the bare metal node completely, the ‘status: deleting’ can be assigned in the inventory.yml file. Information about the bare metal will be removed from the storage-nodes.yml file:
[contrail@5a6s13-node1 contrail_cloud]$ cat config/inventory.yml
...
inventory_nodes:
- name: "storage3"
pm_addr: "10.84.129.184"
status: deleting
<<: *common
[contrail@5a6s13-node1 contrail_cloud]$ ./scripts/inventory-assign.sh
[contrail@5a6s13-node1 contrail_cloud]$ cat config/storage-nodes.yml
storage_nodes:
- name: storage1
profile: blue2
- name: storage2
profile: blue2