[Miyakawa, Kokeyama]

We found that MICH_LOCK guardian had not distinguished the MICH lock status properly since the beginning of the test run phase II. There was a trivial bug in OPERATION state in the code, and it was not checking the lock status continuously. It checked the lock status only once before entering OPERATION status.

Why PMC, IMC couldn’t automatically recover last night (reported by Prof Kawamura) was because MICH guardian couldn’t notice MICH lock was lost and didn’t go to DOWN state. DOWN state is the only state when MICH_LOCK guardian request IMC locked (and IMC requests PMC locked; it’s a hierarchical structure).

This bug is now fixed, and I have confirmed PMC, IMC are automatically recovered when MICH guardian is executed. OPERATION state is valid ony from the run this morning, starting at 10:54:30 JST.

I also added a new state LOCKLOSS in MICH guardian. Guardian jumps to this state when the lock check (in the code, it’s called much_refl_quiet_check) fails. Then it goes to DOWN immidiately. This new state can be used for a lockloss analysis.