Oshino-san, DanChen-san, Washimi-san, Ikeda
Related to K-Log#37506
Below is a summary of the recovery work following the power outage on September 21.
DC Power Supplies
* All central DC power supplies for both the 24 V and 18 V systems had tripped.
* The DC power supplies at both end stations and on the second floor had not tripped and had recovered to a state in which power was being supplied.
* The front-panel power buttons on the V2 I/O chassis were OFF after power was restored. This is the expected behavior following a power outage.
FEPCs
* k1ix1, k1iy0, k1asc0, and k1test0 had restarted but stopped during the boot process because k1boot was unavailable and the required file systems could not be mounted.
* All other FEPCs were powered off.
The following recovery work was performed on the real-time systems:
1. After waiting for the air-conditioning system and k1boot to recover, we switched off the 18 V and 24 V power supplies. On the rack side, we also switched off the circuit breakers, chassis power supplies, and power strips.
2. We placed the control panels in operation mode.
3. For both the 24 V and 18 V systems, we performed the following sequence:
* Switched on the power supply
* Measured the voltage at the circuit breaker
* Switched on the circuit breaker
* Measured the voltage at the power strip
* Switched on the power strip
* Switched on the chassis power supply
4. After powering on all equipment in the central area, we followed the same procedure for the second floor of the central area and the first and second floors of both the X- and Y-end stations. We also powered on the workstations.
Some equipment, including the OMC PZT drivers, POS-related equipment, and several other devices, remains powered off.
The DC power supplies on the second floor of the central area and on the first and second floors of both end stations had remained on. However, all real-time PCs and I/O chassis had their power switches set to OFF, so we switched them on.
5. We restored the real-time computers in the central area.
Recovered:
* Without Dolphin:
k1mcf0, k1ix1, k1iy1, k1ex1, k1ey1, k1ex0, k1ey0, k1test0, k1iy0
* With Dolphin:
k1lsc0, k1asc0, k1als0, k1ioo, k1ioo1, k1imc0
Although some systems show drift in IRIGB_TIME, we confirmed that they have otherwise recovered and are operating normally.
Remaining systems:
* Without Dolphin:
k1px1 — The electronics power supply has not yet been checked.
* With Dolphin:
k1pr2, k1pr0, k1prm, k1bs, k1sr2, k1sr3, k1srm, k1omc0, k1omc1
The cards on k1pr2 are not being recognized. Due to time constraints, we will investigate the issue in detail tomorrow.
