897241 - Move rack 101-20 minis (all) to B side July 31

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[u429623]

Attachment: hostnames of minis in this rack

A side work will be completed, need to move all to B side.

List of affected minis is attached.

Comment 1

[u429623]

Timetable:
 0600 PT - disable affected minis in slavealloc (script will be attached)
         - announce in #releng
 0830 PT - power down affected minis (script will be attached)
 0900 PT - verify all down, hand off to DCOPS for work
 0930 PT - (when DCOPS hands back) verify all minis back up
         - re-enable in slavealloc (see notes in scripts)
         - reboot all affected minis (to pick up slavealloc)
 1000 PT - announce in #releng

Comment 2

[Pete Moore [:pmoore][:pete]]

Minis disabled. Waiting for 0830 PT for next step.

Comment 3

[Pete Moore [:pmoore][:pete]]

Starting next phase:  0830 PT - power down affected minis (script will be attached)

Comment 4

[Pete Moore [:pmoore][:pete]]

Minis are in process of being shutdown

Comment 5

[Pete Moore [:pmoore][:pete]]

Power down complete!

Comment 6

[u429623]

Boxes all back online with exception of:
 talos-mtnlion-r5-043.test.releng.scl3.mozilla.com
which kept rebooting, so is being reimaged (bug to be filed)

Notes:
 servo-lion-r5-001.build.servo.releng.scl3.mozilla.com
 servo-lion-r5-002.build.servo.releng.scl3.mozilla.com
are not accessible from build lan, so must be managed from laptops

Thanks to all for the help

Comment 7

[Van Le [:van]]

all hosts should be up.

[[e-mail]3 ~]$ fping !$
fping talos-mtnlion-r5-043.test.releng.scl3.mozilla.com
talos-mtnlion-r5-043.test.releng.scl3.mozilla.com is alive

[[e-mail]3 ~]$ ssh !$
ssh talos-mtnlion-r5-043.test.releng.scl3.mozilla.com
The authenticity of host 'talos-mtnlion-r5-043.test.releng.scl3.mozilla.com (10.26.56.63)' can't be established.
RSA key fingerprint is 8f:32:a6:9d:7d:d7:04:f3:8e:66:d4:e3:3a:6c:56:a4.
Are you sure you want to continue connecting (yes/no)?