KAGRA Logbook

MIF (General)

shinji.miyoki - 21:48 Saturday 27 May 2023 (25409)

RESET_ASC_FEEDBACK(4) in the LSC_LOCK Guardian failed after PR3 alignment work

[Ushiba, Miyoki, Aoumi, Hirata, (Yokozawa, Oshino)]

Summary

RESET_ASC_FEEDBACK(4) in the LSC_LOCK Guardian seemed not to work and all ASC failed to set "one" before the HANDOVER process. Because all ASC gains were left to be ZERO, the HANDOVER process kept failing. After resetting these ASC gains to one, the HANDOVER smoothly proceeded and PRFPMI+DC was recovered.

What happened

Around 15:51, the IFO was unlocked as reported in the operator's summary.
Because the GRX trans was already below 0.28, the PR3 realignment was done by Oshino-kun and Yokozawa-kun. The GRX trans was recovered up to 0.4 or more.
After this PR3 realignment, the GRX lock became hard. TEM_0,high mode could be seen. Even if ALSX/Y lock was scarcely realized with TEM01 modes, the handover failed. Actually, only two handover trials were done during two hours after the realignment.
We recognized two problems: one was ALSX (GRX) lock trouble, and the other is the handover failures.
Finally, Ushiba-kun was called.

Investigation and Remedy

For the ALX lock recovery
- Because this phenomenon used to be expected because of the too much drift of PR3, Ushiba-kun instructed to change 1 urad in the Pitch of PR3 towards the good value. Then the GRX lock was recovered. Because too many changes such as ~3 urad and more would make the IR lock difficult, 1 urad was selected as a trial value.
For the handover recovery
- Even if ALSX/Y was recovered, the HANDOVER kept failing (~ 4 times or more).
- Ushiba-kun checked the LSC_LOCK Guardian step by step in the past processes, he found that all ASC gains were set to be zero, which should be set to be one before the HANDOVER. This means that the program at "### setting gain 1 for all optic" in "RESET_ASC_FEEDBACK(4)" of the LSC_LOCK Guardian failed to set one from zero.
- Anyway, Ushiba-kun instructed Aoumi-san to redo "RESET_ASC" manually(?). Then the handover was successfully done. Finally, the IFO could reach "OBSERVING".
- We could not understand what conditions occurred this RESET_ASC failure. Anyway, we asked Yuzurihara-kun to check the codes.

Cautions

This RESET_ASC failure was not improved yet. So, the same thing could happen again.

Comments to this report:

takaaki.yokozawa - 7:00 Sunday 28 May 2023 (25411)

Before operator call to Ushiba-san, I tried to perform some activities.

1. Set ETMX_P +1.5 urad by optic align
2. Set PR3_P +0.3 urad by optic align
But both work didn't affect the interferometer lock
After those work, I back to ETMX_P and PR3_P to original position.

3. Lock the ALS X/Y by my hand (Using manual mode of Guradians)
In this case, we can reach the state of handover, but as reported by Miyoki-san, handover process was failed.

At this timing, there are possibility to skip the
"### setting gain 1 for all optic" in RESET_ASC_FEEDBACK(4)
I will discuss with Yuzurihara-san to check them.

takahiro.yamamoto - 23:31 Sunday 28 May 2023 (25419)

> This RESET_ASC failure was not improved yet. So, the same thing could happen again.
This was not a mystery, or code bugs. Guardian codes worked well as we intended.
This was just a result of manipulation in an unrecommended manner of Guardian.

Figure 1 shows the abstract time line during trouble.
Around t=-4h10m (this is consistent with Miyoki-san's log = 15:51 JST), lockloss occurred (See orange curve or red curve).
After then (~5min?), DOWN was requested as shown in green curve.

This trouble caused by the manipulation in the MANUAL mode for LSC_LOCK guardian.
Figure 2 shows the same signals as Fig.1 focused on the trouble time.
At this time LSC_LOCK guardian was in the MANUAL mode, not the AUTO mode as shown in the bottom pannel.
(AUTO mode = 0, MANUAL mode = 2)
I think this is the most serious problem for the observation.
This mode should be used only for a code debug test or for escaping from an inescapable bug at least after code reading.

Anyway, at this time, LSC_LOCK guardian was in MANUAL mode.
RESET_ASC_FEEDBACK (4) was requested in MANUAL mode and ALS_XY_LOCKED (100) was requested 2-second later.
(See red or green curve in Fig.2.)
In the usual case, RESET_ASC_FEEDBACK needs ~10 seconds to finish its process as shown as the red curve in Fig.3.
I guessed that ALS_XY_LOCKED was requested between after ASC gain was changed 0 and before ASC gain was reverted as 1.
If LSC_LOCK guardian is in AUTO mode, it waits to finish the reset process (until True is returned) even if an other state is requested.
But, because it was in MANUAL mode unfortunately, LSC_LOCK moved from the RESET_ASC_FEEDBACK state immediately.

We need to take care about following things
- Don't use MANUAL mode during observing runs.
- Even if we are in the commissioning phase, use MANUAL mode after reading guardian codes and completely understanding what's happen.

Images attached to this comment