In the afternoon of the last Friday, I turned off all the power for below items in the front room

all RT-PCs, all Dolphin switches, all GE RFM switches, 5 DAQ servers, 1 Boot server, 20TB storages, 1 client WS, 4 UPSs, DC power supplies for some circuits and timing system, DC power supply controller for field DC power,

and all DC power supplies and IO chassis for 2 field racks.

DMG PCs and Detchar PCs were turned off remotely in the morning of Friday by Kanda-san and Hayama-kun. Also the Control instruments rack was turned off by Aso-kun (I am not sure the day and time).

I left several network switches to ensure remote access, but one of the main switch was connected to UPS to communicate remaining PCs to be shutdown for unexpected power failure, so I took power for the switch from the normal AC power tap temporarily.


On Monday morning, I turned on all the power for things I turned off last week. Basically almost of everything was OK, but I need to note several things.

1. Order is important: it should be something like below;

FIeld rack DC power fuelies, IO chassis in field racks. UPSs, Boot server, client WS in the computer room, DC power for timing system, IRIG-B power disconnect and connect again, 20TB datastorages, k1dmg0, k1dmg1, k1dc0, k1fw0, k1fw1, k1nds0, k1nds1, IO chassis for RT-PCs in the computer room, Dolphin and GE RFM switch, all RT PCs.

2. IRIG-B module could not get the time even when DC power supply was on. It should be tuned off once more after the Fanout was in operation. This is the know problem.

3. Temperature in the rack was 21 degree and humidity was over 70% for both racks. For short term, it may be OK but not acceptable for long term.

4. Humidity in the computer room was 80%. Cooler was working when I checked the humidity but it was too cool because no PC was in operation.


We need check storages are mounted, RT-PCs are running well. I noticed several things;

* IRIG-B signal on k1imc is something wrong (known problem).
* k1dmg1(hyades-1) failed to be booted. it failed to recognize the HDD.
* one of the 20TB storage is something wrong. Web cannot be accessible. Some of the LEDs on front panel are orange. It may be related above trouble on k1dmg1
* k1fw1 took very long time to be booted. It may be related above trouble on the storage.

* Second trial of k1dmg1 succeeded.
* DAQ data is not being written by frame writer. fb/master was something wrong and daqd was not starting at k1dc0. After fixing master file, daw started anyway, but still DAQ data is not coming.

I gave up at this point and got out from mine.