I found that OMC_LSC guardian stops due to the camera errors.
I'm not so sure what happens but is it related to the maintenance work today?
This function makes a run-time error when the picture file cannot be found in some reason.
I have no idea now why the picture file couldn't be found.
When I started to investigate this issue (maybe around 1:30?), there was no image file in any directories.
Did you remove taken pictures by manually?
I want to know whether a picture was taken but couldn't be found or a picture wasn't taken at all.
Investigation memo
According to the guardian logs, following procedure was executed.
1. FM4 of OMC-LSC_FB_FLT was turned ON.
2. CAM-OMC_TRANS_ARCHIVE_INTERVAL was changed to 1.
3. It takes 2-second long as waiting time by 'OMC_snapshot_timer'.
4. CAM-OMC_TRANS_ARCHIVE_INTERVAL was reverted to 0.
5. The get_latest_file function was called and it made a run-time error.
This condition is satisfied by executing
933 elif ezca['OMC-TRANS_DC_SUM_OUTPUT'] >= ezca['OMC-LSC_TRIG_THRESHOLD_LO']:after then, executing
934 ezca['OMC-LSC_TRIG_THRESHOLD_LO'] = 1
935 ezca.switch('OMC-LSC_FB_FLT','FM4', 'ON')
936 #Check the mode shape
937 ezca['CAM-OMC_TRANS_ARCHIVE_INTERVAL'] = 1
938 self.timer['OMC_snapshot_timer'] = 2
939 self.taking_snapshot = True
940
941 pattern = '/opt/rtcds/userapps/trunk/omc/k1/guardian/sounds/' + ezca['SYS-SOUND_OMC_LOCK_GLOB']
942 # kagralib.play_random_wavefile(pattern)
891 if self.taking_snapshot:
892 if self.timer['OMC_snapshot_timer']:
893 ezca['CAM-OMC_TRANS_ARCHIVE_INTERVAL'] = 0
894 fname = get_latest_file('/kagra/camera/images/[0-9]*/[0-9]*/[0-9]*/OMC_TRANS*')
895 log(fname)
896 self.p = sp.Popen(['/opt/rtcds/userapps/trunk/cds/common/scripts/mode', fname],
897 stdout=sp.PIPE)
898 self.analyzing_mode = True
899 self.taking_snapshot = False
I simulated this condition as follows. But there was no problem to take a picture and to find it.
>>> from ezca import Ezca
>>> import glob
>>> ezca = Ezca()
>>> import time
>>> def func():
... ezca['CAM-OMC_TRANS_ARCHIVE_INTERVAL'] = 1
... time.sleep(2)
... ezca['CAM-OMC_TRANS_ARCHIVE_INTERVAL'] = 0
... print( glob.glob('/kagra/camera/images/[0-9]*/[0-9]*/[0-9]*/OMC_TRANS*') )
...
>>> print( glob.glob('/kagra/camera/images/[0-9]*/[0-9]*/[0-9]*/OMC_TRANS*') )
[]
>>> func()
K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 1
K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 0
['/kagra/camera/images/2026/02/06/OMC_TRANS_2026-02-06-17-04-53.tif']
Another possibility is very rare timing issue between two self.timer and a delay of subprocess.
According to the following logs, 'snapshot clean up timer' was activated just ~1.5s before taking picture.
2026-02-06_13:26:37.230904Z OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] timer['snapshot clean up timer'] done
2026-02-06_13:26:37.405732Z OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:OMC-LSC_FB_FLT => OFF: FM4
2026-02-06_13:26:37.406907Z OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:OMC-PZT_HV1_OFFSET => 4780.0
2026-02-06_13:26:37.473963Z OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] timer['snapshot clean up timer'] = 120
2026-02-06_13:26:37.726469Z OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:OMC-LSC_FB_FLT => OFF: FM4
2026-02-06_13:26:37.727160Z OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:OMC-PZT_HV1_OFFSET => 4790.0
2026-02-06_13:26:38.019507Z OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:OMC-LSC_FB_FLT => OFF: FM4
2026-02-06_13:26:38.021371Z OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:OMC-PZT_HV1_OFFSET => 4800.0
2026-02-06_13:26:38.285156Z OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:OMC-LSC_FB_FLT => OFF: FM4
2026-02-06_13:26:38.286160Z OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:OMC-PZT_HV1_OFFSET => 4810.0
2026-02-06_13:26:38.594727Z OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:OMC-LSC_FB_FLT => OFF: FM4
2026-02-06_13:26:38.600320Z OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:OMC-PZT_HV1_OFFSET => 4820.0
2026-02-06_13:26:38.646883Z OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:OMC-LSC_TRIG_THRESHOLD_LO => 1
2026-02-06_13:26:38.647217Z OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:OMC-LSC_FB_FLT_SW1 => 1024
2026-02-06_13:26:38.898631Z OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:OMC-LSC_FB_FLT => ON: FM4
2026-02-06_13:26:38.899212Z OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 1
2026-02-06_13:26:38.900160Z OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] timer['OMC_snapshot_timer'] = 2
2026-02-06_13:26:40.901117Z OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] timer['OMC_snapshot_timer'] done
2026-02-06_13:26:40.948043Z OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 0
2026-02-06_13:26:40.966347Z OMC_LSC W: Traceback (most recent call last):
2026-02-06_13:26:40.966347Z File "/usr/lib/python3/dist-packages/guardian/worker.py", line 494, in run
2026-02-06_13:26:40.966347Z retval = statefunc()
2026-02-06_13:26:40.966347Z File "/usr/lib/python3/dist-packages/guardian/state.py", line 246, in __call__
2026-02-06_13:26:40.966347Z main_return = self.func.__call__(state_obj, *args, **kwargs)
2026-02-06_13:26:40.966347Z File "/usr/lib/python3/dist-packages/guardian/state.py", line 246, in __call__
2026-02-06_13:26:40.966347Z main_return = self.func.__call__(state_obj, *args, **kwargs)
2026-02-06_13:26:40.966347Z File "/opt/rtcds/userapps/release/omc/k1/guardian/OMC_LSC.py", line 894, in run
2026-02-06_13:26:40.966347Z fname = get_latest_file('/kagra/camera/images/[0-9]*/[0-9]*/[0-9]*/OMC_TRANS*')
2026-02-06_13:26:40.966347Z File "/opt/rtcds/userapps/release/omc/k1/guardian/OMC_LSC.py", line 42, in get_latest_file
2026-02-06_13:26:40.966347Z return file_name
2026-02-06_13:26:40.966347Z UnboundLocalError: local variable 'file_name' referenced before assignment
This time launched the "rm" command by the background subprocess as follows.
886 elif self.timer['snapshot clean up timer']:If the subprocess is delayed by merely ~1 second in some reason, taken picture was removed by delayed subprocess before finding picture file by the get_latest_file function. (It is not surprising that there is a slight delay due to the process crossing multiple networks such as NFS.) It still remains imagination. But now, there is no enough information and reproducibility of this issue.
887 if not (self.analyzing_mode or self.taking_snapshot):
888 self.p2 = sp.Popen(['/bin/rm /kagra/camera/images/[0-9]*/[0-9]*/[0-9]*/OMC_TRANS*'], shell=True)
889 self.timer['snapshot clean up timer'] = 120
It is difficult to confirm the exact cause of user code errors, but it is very ikely that the problem was happened due to the very unfortune timing of the subprocess to remove the snapshot files.
To avoid this error, I deleted the function to remve the snapshot files during the OMC scan from the FIND_RESONANCE, FIND_RESONANCE_FOR_10W, and FIND_RESONANCE_FOR_IAL_WITH_RF states while adding this function at DOWN state.
Figure 1 shows the script I modified at DOWN state.
Figure 2 shows the script I commented out at FIND_RESONANCE state, which is same for the other FIND_RESONANCe states.
Following logs are around the error in this time. Guardian tried to take images several times with a few seconds cadence. But a same file was chosen as a latest image. When 2min.-long timer was executed, all image files were removed with leaving directories. After then guardian tries to take an image again, but a new image file wasn't created because it hadn't passed 1min. cadence yet which was a specification of pylon-camera-server. Probably, this is the full picture of what happened this time.
controls@k1grd0:~$ journalctl --user --since '2026-02-06 22:00:00' --until '2026-02-06 23:00:00' | grep OMC_LSC | grep 'INTERVAL\|tif$\|ERROR'
Feb 06 22:24:42 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 1
Feb 06 22:24:44 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 0
Feb 06 22:24:44 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] /kagra/camera/images/2026/02/06/OMC_TRANS_2026-02-06-13-24-42.tif
Feb 06 22:24:55 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 1
Feb 06 22:24:57 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 0
Feb 06 22:24:57 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] /kagra/camera/images/2026/02/06/OMC_TRANS_2026-02-06-13-24-42.tif
Feb 06 22:25:01 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 1
Feb 06 22:25:03 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 0
Feb 06 22:25:03 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] /kagra/camera/images/2026/02/06/OMC_TRANS_2026-02-06-13-24-42.tif
Feb 06 22:25:05 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 1
Feb 06 22:25:07 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 0
Feb 06 22:25:07 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] /kagra/camera/images/2026/02/06/OMC_TRANS_2026-02-06-13-24-42.tif
Feb 06 22:25:09 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 1
Feb 06 22:25:11 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 0
Feb 06 22:25:11 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] /kagra/camera/images/2026/02/06/OMC_TRANS_2026-02-06-13-24-42.tif
Feb 06 22:25:21 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 1
Feb 06 22:25:23 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 0
Feb 06 22:25:23 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] /kagra/camera/images/2026/02/06/OMC_TRANS_2026-02-06-13-24-42.tif
Feb 06 22:25:36 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 1
Feb 06 22:25:38 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 0
Feb 06 22:25:38 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] /kagra/camera/images/2026/02/06/OMC_TRANS_2026-02-06-13-24-42.tif
Feb 06 22:25:58 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 1
Feb 06 22:26:00 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 0
Feb 06 22:26:00 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] /kagra/camera/images/2026/02/06/OMC_TRANS_2026-02-06-13-25-58.tif
Feb 06 22:26:02 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 1
Feb 06 22:26:05 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 0
Feb 06 22:26:05 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] /kagra/camera/images/2026/02/06/OMC_TRANS_2026-02-06-13-25-58.tif
Feb 06 22:26:26 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 1
Feb 06 22:26:29 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 0
Feb 06 22:26:29 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] /kagra/camera/images/2026/02/06/OMC_TRANS_2026-02-06-13-25-58.tif
Feb 06 22:26:38 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 1
Feb 06 22:26:40 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 0
Feb 06 22:26:40 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] USERMSG 0: USER CODE ERROR (see log)
Feb 06 22:26:41 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC ERROR in state FIND_RESONANCE_FOR_IAL_WITH_RF: see log for more info (LOAD to reset)
I believe Ushiba-kun's update will allow us to avoid this error itself. But a taking image can still be done only with 1min. cadence. From the view point of a taken time for the lock acquisition, the combination of the pylon-camera-server and KAGRA's OMC lock scheme is unrealistic. At least OMC_TRANS camera may be better to be backed to the old camlan camera-server during IR1.
camlan camera-server was already expired 8 years ago and it doesn't work modern OS. So we need to decide toward O5 whether we will continue to maintain old OS and softwares, will develop a new camera-server application as a fork of pylon-camera-server (may finally be merged) to satisfy our purpose, or will change the lock aquisition scheme.
OMC_LSC guardian stops again because no snapshot file is found (fig1).
This time, FIND_RESONANCE state was once passed but the OMC lockloss happened soon after OMC lock and the guardian tried to lock OMC again within 1 minute.
To solve this problem, I think up two ideas:
1. Wait 60 seconds at the end of DOWN state. This will guarantee the last image taking is at least 1 minute before the previous image taking.
2. Move find image process into exception handling by using try-except.
Probably, it is better to use the 2nd idea but I modified the guardian based on the first idea at this moment (fig2) because I have no time to test the code very well.
In addition, I confirmed LSC_LOCK guardian stops at LOCK_PREP state as we expect because the one of latch switchfor OMC protection function is closed (fig3).
It's not deployed yet, but we can start to test soon.
-----
To proper management about a process of taking picture, self.taking_snapshot and self.analyzing_mode sub-states had to be divided two sub-states, respectively. For this reason, FIND_RESONANCE* states were drastically rewritten. We considered two approaches for backgrounding a taking picture process. One is to load camera_client as a python module and to call its function under a thread pool. Another one is to execute it via subprocess. Former one is better from the view point of avoiding zombie processes. But a code becomes complicated. Because OMC_LSC guardian must be understood by many people (most of commissioners), we decided to use subprocess method for easy code reading.
A new code and differences can be found as
- /opt/rtcds/userapps/release/omc/k1/guardian/OMC_LSC_new_camera_client.py
- /opt/rtcds/userapps/release/omc/k1/guardian/patches/OMC_LSC-new-camera-client_260217.patch
[Ushiba, YamaT]
We deployed the new OMC_LSC guardian code to use the new camera client script for taking OMC_TRANS images.
Guardian logs (Fig.1) shows that new code can take multiple images with a short interval as expected.
After running it for a few days without major issues, it should be fine to remove all temporary measures like the 60-second sleep (klog#36364).
At this time, we also made two minor improvements as follows
1. Mitigation of a sign flip issue of sweep direction when guardian load is done in FIND_RESONANCE (Fig.2)
Because the sign of sweep direction is defined as global variable, loading guardian code during a stay in the run function makes undesirable a sign flip. In normal operation, such code loading isn't done, so it's not an issue. But it sometimes occur during a code development to escape from the user code error. This issue makes it difficult to distinguish whether the issue lies with the modified code or simply with loading a code in the run function. This improvement avoids undesirable sign flips caused by loading code within the run function, which should make future code development easier.
2. Reduction of a possibility to grab a same resonance when it's a non-TEM00 (Fig.3)
When guardian detects non-TEM00 lock, it makes a jump of the offset value to try to get away from that wrong resonance. But this offset jump is made with shorter time (in same cycle) than the decay time of analogue electronics, so the offset jump on the digital system doesn't sometimes work well. As a result, it detects non-TEM00 lock once, wrong lock is often repeated. This improvement makes the offset jump on the digital system effective to grab TEM00 with a small number of trial.
The difference before and after these minor improvements are available in/opt/rtcds/userapps/release/omc/k1/guardian/patches/OMC_LSC-new-camera-client_260219.patch
