Abstract

Details

I added the SYS_LOCKLOSS guardian in the medm screen. See the bottom of the attached screenshot.

I updated k1grdconfig for adding necessary EPICS channels on SYS_LOCKLOSS guardian as shown in Fig.1 and Fig.2.
All added channels are StringIn blocks, so there is no change in the channel list for DAQ.

Compile check was already completed but it's not installed yet.
It will be installed on the next maintenance day.

A new function was deployed around 11:30 in LOCKLOSS guardian for the cases of lockloss due to ISS saturation reported in klog#31974.
So it can be automatically tagged from now.

-----
Figure 1 and Figure 2 show the lockloss cases due to non-ISS issue and ISS issue. In the case of non-ISS issue, ISS guardian goes to DONW after (or at the same time in 16Hz) LSC_LOCK guardian goes to LOCKLOSS. AOM feedback becomes also large value after the lockloss. On the other hand, we can see ISS down and oscillation of AOM output a few second before the lockloss in the case of ISS issue.
We could identify the most cases of known lockloss reasons by seeing proper signals 0~2 seconds before the lockloss time. But in the ISS case, problem occurs much earlier than the lockloss time and AOM output is already stopped by ISS guardian just before the lockloss. So checking longer duration must be needed to identify the ISS issue. Figure 3 is also ISS-related lockloss in another time. Relative down time between ISS and LSC_LOCK is different from the case of Fig.2. According to check several cases, this issue seems to be detected by seeing AOM output 0~4 seconds before the lockloss time.

Note for future maintenance.
When ISS goes to DOWN, BPC control doesn't work because BPC lines are hidden by intensity noise. So difference in the relative time between ISS and LSC_LOCK may depends on the speed of BPC control and we may need to modify the lockloss detection scheme when BPC control will be drastically changed.

Yuzurihara-kun requested to add a duration of staying in the last state to the lockloss list.
So I added a new function to compute a duration in the guardian code and added them in the lockloss table.
A duration for the past lockloss hasn't been computed.
If someone make a list of a duration for the past lockloss events, I can add them to the json file for making the table on the Web pages.

For this update, k1grdconfig was updated to add new channels as shown in Fig.1. All added channels are the EPICS StringIn. So the daqd was not restarted for this work because there was no change in the DAQ list. k1grdconfig is only a process which has been restarted.

-----
For computing an accurate duration, guardian may have to read so long time data when lockloss occurred after staying some state for a long time. Such an implementation is not realistic for guardian which is a semi-real-time process. For solving this issue, LOCKLOSS guardian saves a time stamp when LSC_LOCK guardian moves to states and a duration is computed as a difference between the lockloss time and the saved time stamp. For this reason, this duration doesn't have same accuracy with the DAQ time stamp. Please use it as a rough reference.

Thank you for updating the SYS_LOCKLOSS guardian.

It is possible to add the label indicating if someone was adding the excitation signal? In the lockloss of 2025-01-19 02:47:58 UTC, the commissioners were working on the measurement of the transfer function by using the excitation.
Ushiba-san commented that when the excitation is happening, any situations are possible. If you add such label, we can ignore the lockloss under the excitation, during the lockloss investigation.

If it's only referring the Excitation check guardian, it's easy and it might be a good practice to touch guardian. Note that the reason why lockloss check code is prepared as guardian is all on-site people can improve it by themselves (If I'm only people to edit it, I want to refactor it as a non-guardian code for flexibility).

What we must seriously consider is that excitation channels are often left with un-closed even if an actual excitation is finished. Unfortunately it seemed happen today as shown in Fig.1. Though some excitations from suspensions seemed to be presence in this 4hrs., it's hard to separate actual excitation from wrong flagged data due to un-closed excitations. It's a matter of a manner of user codes and activities, some people had already reported many times (so it's no longer possible for us to improve it?).

From the view point of observation, we can just eliminate the all suspicious data. It becomes just a reduction of the duty factor by our fault. On the other hand, it had better to avoid a false labeling for lockloss study because workers become not to check them. For this sense, labeling with a low reliable way is worse than non-labeling. Automatic labeling is for reducing our work, not for hiding check items it's really needed. We should decide to adding EXC flag now with consideration whether the most people prefer to increase labelled events even if reliability is low or to label only reliable events.

This is remaining work of klog#32380.

Lockloss guardian was updated on Jan17 to add the information about the duration staying in the last state. Though lockloss information files stored in /users/Commissioning/data/lockloss/yyyy/ has duration after this update, files before Jan16 didn't have it yet. So I re-analized all lockloss events (the 1st event by lockloss guardian is on Sep10, 2024) and re-provided lockloss information files from Sep10 2024 to Jan16 2025. I checked `diff` between all old and new files, so there should be no difference other than the addition of the duration.

They are now available /users/Commissioning/data/lockloss/yyyy/ and the lockloss table on the web is provided based on these new files. Old files are kept in /users/Commissioning/data/lockloss/yyyy/__old/2025_0126_before_add_duration/.

Related to klog32695,

Ushiba-san requested me to add the label to know if ISC watch dog was turned on before the lockloss. I implemented the function and and tested in the test guardian. The label name is ISC_WD_L.

If the LSC_LOCK guardian will be DOWN state tomorrow morning (such as 8:50~9:00), I will add this feature in SYS_LOCKLOSS guardian. I will replace the json files of past data (after 2/14) after that.

Note

• The working directory: /users/yuzu/work/20250217_lockloss_guardian/ISC_WD
• The condition to label LSC_WD_L is K1:VIS-ETMX_ISCWD_WD_AC_L_RMSMON > K1:VIS-ETMX_ISCWD_WD_AC_L_RMS_MAX.
  • Exampe of the label.
  • If the LSC_LOCK guardian entered lockloss state and the watch dog was turned on at the same time (Fig), we will not flag the label to avoid the false detection. (memo) at this time, the Ushiba-san's idea to flag this phenomena is to check if the RMSMON data just before the lockloss is approaching to the threshold or not. When the ISC_WD_P or Y is implemented, I will add this feature.
• This label will be checked only when the LSC_LOCK guardian is more than ENGAGE_ALS_DARM.
• This kind of watch dog will be implemented for pitch and yaw, and for other type-A suspensions. In future, it's possible to check 12 watch dogs. As the chat with Ushiba-san, we will implement three label (ISC_WD_L, ISC_WD_P, ISC_WD_Y), instead of preparing 12 labels

I finished to implement the function to label the ISC_WD_L in the SYS_LOCKLOSS guardian, at 8:50. At that time, the LSC_LOCK guardian was in the DOWN state.

I also updated the function to treat the case that the LSC_LOCK guardian entered lockloss state and the watch dog was turned on at the same time (Fig). I implemented the code to check if the average of RMSMON data just before the lockloss is approaching to the threshold or not. I set the threshold for the the average of RMSMON data as 0.9 * RMS_MAX.
For the lockloss events for 02/14, 15 and 17, this threshold (0.9 * RMS_MAX) works well. If necessary in future, I will tune it.
The related lockloss are

• 1423577533.0625 (Fig)
• 1423582334.9375 (Fig)
• 1423695458.3125 (Fig)

I replaced the json files for 02-14, 02-15, 02-17, 02-18, stored in /users/Commissioning/data/lockloss/2025. The previous files are backuped to /users/Commissioning/data/lockloss/2025/__old/2025_0219_before_add_ISC_WD_L/.

The lockloss due to the oscillation of IMC control could be seen in the recent lockloss investigation (klog33036, klog32903, klog32803). I implemented new label to tag the lockloss identify to the oscilation of IMC length control. The label name is IMCL.
In the some case, even though the 1Hz seismic motion is less than the threshold, the oscillation of the IMC control occured and it leads the lockloss. (Fig1) (Fig2) By adding this label, we can identify such loskloss.

I tested the function in local directory and by using the CAL_PROC guardian, Betweek 2024/09~2024/03/24, 323 lockloss were tagged with this label. I performed the eye scan for all the lockloss and confirmed that it works well. After the test, I added to SYS_LOCKLOSS guardian at 14:18.

I replaced the json files in /users/Commissioning/data/lockloss with the update json files. The old json files are stored in /users/Commissioning/data/lockloss/2025/__old/2025_0327_before_add_IMCL