KAGRA Logbook

IOO (OMC)

takafumi.ushiba - 22:53 Friday 06 February 2026 (36311)

OMC_LSC guardian stops due to the camera error

I found that OMC_LSC guardian stops due to the camera errors.
I'm not so sure what happens but is it related to the maintenance work today?

Images attached to this report

36311_1770385874_34072_1770385777_Screenshot from 2026-02-06 22-47-51.png

Comments to this report:

takahiro.yamamoto - 3:04 Saturday 07 February 2026 (36312)

An error itself comes from an implementation of the get_latest_file function which was not changed in today's update.
This function makes a run-time error when the picture file cannot be found in some reason.

I have no idea now why the picture file couldn't be found.
When I started to investigate this issue (maybe around 1:30?), there was no image file in any directories.
Did you remove taken pictures by manually?
I want to know whether a picture was taken but couldn't be found or a picture wasn't taken at all.

Investigation memo
According to the guardian logs, following procedure was executed.
1. FM4 of OMC-LSC_FB_FLT was turned ON.
2. CAM-OMC_TRANS_ARCHIVE_INTERVAL was changed to 1.
3. It takes 2-second long as waiting time by 'OMC_snapshot_timer'.
4. CAM-OMC_TRANS_ARCHIVE_INTERVAL was reverted to 0.
5. The get_latest_file function was called and it made a run-time error.

This condition is satisfied by executing

 933         elif ezca['OMC-TRANS_DC_SUM_OUTPUT'] >= ezca['OMC-LSC_TRIG_THRESHOLD_LO']:
 934             ezca['OMC-LSC_TRIG_THRESHOLD_LO'] = 1
 935             ezca.switch('OMC-LSC_FB_FLT','FM4', 'ON')
 936             #Check the mode shape                                                                                                                                                                                                                                                                                                                                      
 937             ezca['CAM-OMC_TRANS_ARCHIVE_INTERVAL'] = 1
 938             self.timer['OMC_snapshot_timer'] = 2
 939             self.taking_snapshot = True
 940 
 941             pattern = '/opt/rtcds/userapps/trunk/omc/k1/guardian/sounds/' + ezca['SYS-SOUND_OMC_LOCK_GLOB']
 942             # kagralib.play_random_wavefile(pattern)

after then, executing

 891         if self.taking_snapshot:
 892             if self.timer['OMC_snapshot_timer']:
 893                 ezca['CAM-OMC_TRANS_ARCHIVE_INTERVAL'] = 0
 894                 fname = get_latest_file('/kagra/camera/images/[0-9]*/[0-9]*/[0-9]*/OMC_TRANS*')
 895                 log(fname)
 896                 self.p = sp.Popen(['/opt/rtcds/userapps/trunk/cds/common/scripts/mode', fname],
 897                             stdout=sp.PIPE)
 898                 self.analyzing_mode = True
 899                 self.taking_snapshot = False

I simulated this condition as follows. But there was no problem to take a picture and to find it.

>>> from ezca import Ezca                                                                                                                                                                                                                                                                                                                                                   
>>> import glob
>>> ezca = Ezca()
>>> import time
>>> def func():
...     ezca['CAM-OMC_TRANS_ARCHIVE_INTERVAL'] = 1
...     time.sleep(2)
...     ezca['CAM-OMC_TRANS_ARCHIVE_INTERVAL'] = 0
...     print( glob.glob('/kagra/camera/images/[0-9]*/[0-9]*/[0-9]*/OMC_TRANS*') )
...
>>> print( glob.glob('/kagra/camera/images/[0-9]*/[0-9]*/[0-9]*/OMC_TRANS*') )
[]
>>> func()
K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 1
K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 0
['/kagra/camera/images/2026/02/06/OMC_TRANS_2026-02-06-17-04-53.tif']

Another possibility is very rare timing issue between two self.timer and a delay of subprocess.
According to the following logs, 'snapshot clean up timer' was activated just ~1.5s before taking picture.

2026-02-06_13:26:37.230904Z OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] timer['snapshot clean up timer'] done
2026-02-06_13:26:37.405732Z OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:OMC-LSC_FB_FLT => OFF: FM4
2026-02-06_13:26:37.406907Z OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:OMC-PZT_HV1_OFFSET => 4780.0
2026-02-06_13:26:37.473963Z OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] timer['snapshot clean up timer'] = 120
2026-02-06_13:26:37.726469Z OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:OMC-LSC_FB_FLT => OFF: FM4
2026-02-06_13:26:37.727160Z OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:OMC-PZT_HV1_OFFSET => 4790.0
2026-02-06_13:26:38.019507Z OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:OMC-LSC_FB_FLT => OFF: FM4
2026-02-06_13:26:38.021371Z OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:OMC-PZT_HV1_OFFSET => 4800.0
2026-02-06_13:26:38.285156Z OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:OMC-LSC_FB_FLT => OFF: FM4
2026-02-06_13:26:38.286160Z OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:OMC-PZT_HV1_OFFSET => 4810.0
2026-02-06_13:26:38.594727Z OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:OMC-LSC_FB_FLT => OFF: FM4
2026-02-06_13:26:38.600320Z OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:OMC-PZT_HV1_OFFSET => 4820.0
2026-02-06_13:26:38.646883Z OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:OMC-LSC_TRIG_THRESHOLD_LO => 1
2026-02-06_13:26:38.647217Z OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:OMC-LSC_FB_FLT_SW1 => 1024
2026-02-06_13:26:38.898631Z OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:OMC-LSC_FB_FLT => ON: FM4
2026-02-06_13:26:38.899212Z OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 1
2026-02-06_13:26:38.900160Z OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] timer['OMC_snapshot_timer'] = 2
2026-02-06_13:26:40.901117Z OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] timer['OMC_snapshot_timer'] done
2026-02-06_13:26:40.948043Z OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 0
2026-02-06_13:26:40.966347Z OMC_LSC W: Traceback (most recent call last):
2026-02-06_13:26:40.966347Z   File "/usr/lib/python3/dist-packages/guardian/worker.py", line 494, in run
2026-02-06_13:26:40.966347Z     retval = statefunc()
2026-02-06_13:26:40.966347Z   File "/usr/lib/python3/dist-packages/guardian/state.py", line 246, in __call__
2026-02-06_13:26:40.966347Z     main_return = self.func.__call__(state_obj, *args, **kwargs)
2026-02-06_13:26:40.966347Z   File "/usr/lib/python3/dist-packages/guardian/state.py", line 246, in __call__
2026-02-06_13:26:40.966347Z     main_return = self.func.__call__(state_obj, *args, **kwargs)
2026-02-06_13:26:40.966347Z   File "/opt/rtcds/userapps/release/omc/k1/guardian/OMC_LSC.py", line 894, in run
2026-02-06_13:26:40.966347Z     fname = get_latest_file('/kagra/camera/images/[0-9]*/[0-9]*/[0-9]*/OMC_TRANS*')
2026-02-06_13:26:40.966347Z   File "/opt/rtcds/userapps/release/omc/k1/guardian/OMC_LSC.py", line 42, in get_latest_file
2026-02-06_13:26:40.966347Z     return file_name
2026-02-06_13:26:40.966347Z UnboundLocalError: local variable 'file_name' referenced before assignment

This time launched the "rm" command by the background subprocess as follows.

 886         elif self.timer['snapshot clean up timer']:
 887             if not (self.analyzing_mode or self.taking_snapshot):
 888                 self.p2 = sp.Popen(['/bin/rm /kagra/camera/images/[0-9]*/[0-9]*/[0-9]*/OMC_TRANS*'], shell=True)                                                                                                                                                                                                                                                       
 889                 self.timer['snapshot clean up timer'] = 120

If the subprocess is delayed by merely ~1 second in some reason, taken picture was removed by delayed subprocess before finding picture file by the get_latest_file function. (It is not surprising that there is a slight delay due to the process crossing multiple networks such as NFS.) It still remains imagination. But now, there is no enough information and reproducibility of this issue.

takafumi.ushiba - 13:45 Monday 09 February 2026 (36320)

It is difficult to confirm the exact cause of user code errors, but it is very ikely that the problem was happened due to the very unfortune timing of the subprocess to remove the snapshot files.
To avoid this error, I deleted the function to remve the snapshot files during the OMC scan from the FIND_RESONANCE, FIND_RESONANCE_FOR_10W, and FIND_RESONANCE_FOR_IAL_WITH_RF states while adding this function at DOWN state.

Figure 1 shows the script I modified at DOWN state.
Figure 2 shows the script I commented out at FIND_RESONANCE state, which is same for the other FIND_RESONANCe states.

Images attached to this comment

36320_1770611760_Screenshot from 2026-02-09 13-35-53.png

36320_1770612329_Screenshot from 2026-02-09 13-45-20.png

takahiro.yamamoto - 18:32 Tuesday 10 February 2026 (36328)

The new pylon-camera-server seems to allow to take pictures only with 1min.+ cadence as its specification (please see also the original code on the LIGO Git)

Following logs are around the error in this time. Guardian tried to take images several times with a few seconds cadence. But a same file was chosen as a latest image. When 2min.-long timer was executed, all image files were removed with leaving directories. After then guardian tries to take an image again, but a new image file wasn't created because it hadn't passed 1min. cadence yet which was a specification of pylon-camera-server. Probably, this is the full picture of what happened this time.

controls@k1grd0:~$ journalctl --user --since '2026-02-06 22:00:00' --until '2026-02-06 23:00:00' | grep OMC_LSC | grep 'INTERVAL\|tif$\|ERROR'
Feb 06 22:24:42 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 1
Feb 06 22:24:44 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 0
Feb 06 22:24:44 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] /kagra/camera/images/2026/02/06/OMC_TRANS_2026-02-06-13-24-42.tif
Feb 06 22:24:55 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 1
Feb 06 22:24:57 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 0
Feb 06 22:24:57 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] /kagra/camera/images/2026/02/06/OMC_TRANS_2026-02-06-13-24-42.tif
Feb 06 22:25:01 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 1
Feb 06 22:25:03 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 0
Feb 06 22:25:03 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] /kagra/camera/images/2026/02/06/OMC_TRANS_2026-02-06-13-24-42.tif
Feb 06 22:25:05 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 1
Feb 06 22:25:07 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 0
Feb 06 22:25:07 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] /kagra/camera/images/2026/02/06/OMC_TRANS_2026-02-06-13-24-42.tif
Feb 06 22:25:09 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 1
Feb 06 22:25:11 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 0
Feb 06 22:25:11 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] /kagra/camera/images/2026/02/06/OMC_TRANS_2026-02-06-13-24-42.tif
Feb 06 22:25:21 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 1
Feb 06 22:25:23 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 0
Feb 06 22:25:23 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] /kagra/camera/images/2026/02/06/OMC_TRANS_2026-02-06-13-24-42.tif
Feb 06 22:25:36 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 1
Feb 06 22:25:38 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 0
Feb 06 22:25:38 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] /kagra/camera/images/2026/02/06/OMC_TRANS_2026-02-06-13-24-42.tif
Feb 06 22:25:58 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 1
Feb 06 22:26:00 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 0
Feb 06 22:26:00 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] /kagra/camera/images/2026/02/06/OMC_TRANS_2026-02-06-13-25-58.tif
Feb 06 22:26:02 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 1
Feb 06 22:26:05 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 0
Feb 06 22:26:05 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] /kagra/camera/images/2026/02/06/OMC_TRANS_2026-02-06-13-25-58.tif
Feb 06 22:26:26 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 1
Feb 06 22:26:29 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 0
Feb 06 22:26:29 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] /kagra/camera/images/2026/02/06/OMC_TRANS_2026-02-06-13-25-58.tif
Feb 06 22:26:38 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 1
Feb 06 22:26:40 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] ezca: K1:CAM-OMC_TRANS_ARCHIVE_INTERVAL => 0
Feb 06 22:26:40 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC [FIND_RESONANCE_FOR_IAL_WITH_RF.run] USERMSG 0: USER CODE ERROR (see log)
Feb 06 22:26:41 k1grd0.kagra.icrr.u-tokyo.ac.jp guardian[2552]: OMC_LSC ERROR in state FIND_RESONANCE_FOR_IAL_WITH_RF: see log for more info (LOAD to reset)

I believe Ushiba-kun's update will allow us to avoid this error itself. But a taking image can still be done only with 1min. cadence. From the view point of a taken time for the lock acquisition, the combination of the pylon-camera-server and KAGRA's OMC lock scheme is unrealistic. At least OMC_TRANS camera may be better to be backed to the old camlan camera-server during IR1.

camlan camera-server was already expired 8 years ago and it doesn't work modern OS. So we need to decide toward O5 whether we will continue to maintain old OS and softwares, will develop a new camera-server application as a fork of pylon-camera-server (may finally be merged) to satisfy our purpose, or will change the lock aquisition scheme.