I am very sorry for your inconvenient, especially TCam workers and real time model recovery workers.
But it is BAD to use the outlet inside the clean booth such a very important power supply.
Abstract
We shutdown IFO once to reboot IMC frotn-end.After rebooting IMC front-end (no trouble related to Dolphin), full lock came back.
Details
We recovered IFO with the following procedures.1. Request SAFE for all models.
2. Accept/revert all SDF differences (with taking snapshot).
3. Disable Dolphin for k1imc0
4. Reboot k1imc0 via BMC interface (ssh was also dead).
5. Request LOCK_ACQUISITION to all VIS guardians.
6. Request PROVIDING_STABLE_LIGHT to IO guardian.
7. Request PLL_LOCKED to ALS_PLLX and ALS_PLLY guardians.
8. Request RESET_ASC_FEEDBACK to LSC_LOCK guardian (GrY flash came back at this time).
9. Request PRFPMI_RF_LOCKED to LSC_LOCK guardian.
10. Request OBSERVATION to LSC_LOCK guardian (for CAL measurement not as a recovering work).
After then, we also noticed that timing of K1MCF0 became crazy (always reached 16us which is upper limit of IOP model).
Because it's not related to the Dolphin, we just reboot it.
Timing when K1IMC0 was dead and K1MCF0 became crazy was different as 1-2min.
But maybe this was also related to the works around MCF racks.
