KAGRA Logbook

DetChar (General)

hirotaka.yuzurihara - 10:05 Wednesday 19 August 2026 (37374)

DetChar cluster: computing environment, automatic segment production

I resumed preparations for making effective use of the DetChar cluster, including k1detms0 and the computing nodes k1detcl[012]. (past log: klog31462, related slide)
I configured the computing environment to avoid potential issues with Conda and the system clocks.
I found that the wrong disk had been mounted via NFS, so I corrected the configuration to use the proper server.
I started testing the automatic production of segment files using HTCondor. I will continue monitoring the generated files and logs to confirm stable operation.

Details

Conda environment
- I installed the IGWN Conda environment required for the DetChar tools. I used the receipe updated on 2026/08/14. I used the text-based recipe instead of the yaml one to avoid the newer glibc requirement.
- The conda environment is shared through NFS and is located at /users/DET/conda/miniforge3/envs/igwn-py311/. I confirmed that it works on all four nodes.
System clocks on the computing nodes
- The system clocks on the computing nodes were not synchronized correctly.
- I enabled and started chronyd so that the system clocks are automatically synchronized.
NFS mount configuration
- The storage storing frame files is mounted at /frame1 via NFS.
- I updated the NFS configuration to use the current server and confirmed that the frame data can be accessed correctly.
Segment generation
- I manually tested the Segment production script using the new Conda environment.
- After fixing issues of system clock and NFS maount, the script successfully generated Segment files.
Segment generation using HTCondor
- I set up an HTCondor job to run the Segment production automatically every 15 minutes. I also prepared a script and an HTCondor job to automatically copy the generated Segment files to the product directory at 9:23 every day.
- Working directory: /users/yuzu/work/20260818_cluster
- The jobs are currently being tested, and I am monitoring the output, log files, and memory usage to confirm stable operation.

Comments to this report:

hirotaka.yuzurihara - 17:27 Tuesday 06 October 2026 (37617)

To test the latest version of the DetChar tools, I checked the Segment files automatically generated using HTCondor and compared them with those generated on k1det1.

The files from August 19 and 20 were not consistent. The likely cause was that the HTCondor job ran at 0, 15, 30, and 45 minutes past each hour, before all data for each 15-minute period were available.
I changed the schedule to 3, 18, 33, and 48 minutes past each hour, following the crontab configuration on k1det1.
I will check the results again after a few days.

hirotaka.yuzurihara - 8:42 Wednesday 14 October 2026 (37661)

I tested the automatic Segment production and GauCh analysis as part of the validation of the KAGRA DetChar tools with the latest conda environment.
Overall, both jobs worked well under HTCondor.

The Segment job took 15.5 s on average and up to 25 s to treat 15 minutes data. The memory usage was about 200 MB.

The GauCh job took 116 s on average and up to 136 s to treat one hour data. Its memory usage was about 0.8 GB on average and up to about 1.0 GB.
However, the current test data were taken while the interferometer was not locked, so these resource requirements may be underestimated.
I plan to check the memory usage again after the interferometer is locked and update the requested resources in the HTCondor submit files if necessary.

hirotaka.yuzurihara - 11:19 Tuesday 20 October 2026 (37698)

Here is a progress on segment production on the DetChar cluster.

Since fixing the bug on October 6 (klog37617), segment files have been generated automatically on the DetChar cluster using an HTCondor job.
I compared the segment files generated on the DetChar cluster with those generated on k1det1 for the period between 10/7 0:0:0 UTC and 10/20 0:0:0 UTC. No differences were found. The comparison was performed using: /users/DET/tools/Segments-dev/Script/compareSegmentDict.py

The scripts and submit files used for the test are located at: /users/DET/tools/Segments/Script/detchar-cluster
The generated segment files and log files are stored at: /users/yuzu/work/20260818_cluster

hirotaka.yuzurihara - 17:06 Tuesday 20 October 2026 (37700)

Here is a progress on Gauch on the DetChar cluster.
I have been monitoring the Gauch jobs since submitting them on 10/14 (klog37374).
On 10/16, the job was held with the following error:

cpu usage (10) exceeded request_cpu (8)  Job was held.

To avoid this issue, I increased the requested number of CPUs from 8 to 12. I also added options to the shell script to limit the number of CPUs used by libraries such as NumPy and SciPy.
I will continue monitoring the jobs to check whether the same error occurs again.

The scripts related to the Gauch job are located in: /users/yamamura/Gau_Work/
The submit files are: condor_plot_gauch.sub,  condor_html_gauch.sub

hirotaka.yuzurihara - 17:11 Thursday 22 October 2026 (37721)

Here is an update on Gauch on the DetChar cluster.
To clearly separate the code currently running on the DetChar cluster from the old code, I moved new codes to a dedicated working directory (/users/yamamura/gauch-detchar-cluster/). These codes are running on the cluster.
The code was uploaded to the Git repository: https://git.ligo.org/yuzurihara/gauch-detchar-cluster

I will continue to monitor the status of the HTCondor jobs.

hirotaka.yuzurihara - 9:50 Wednesday 28 October 2026 (37752)

I checked the recent HTCondor logs for Gauch on the DetChar cluster and found that the plotting job had been on hold since October 25.
The job was held because its CPU usage (11 CPUs) exceeded the requested number of CPUs (10 CPUs).
I modified the code to add diagnostic logs to identify which part of the multiprocessing operations causes the CPU usage to exceed the requested limit.

In addition, I removed the currently submitted HTCondor jobs to free up the DetChar cluster for Oshino-san's work. Once Oshino-san has finished the work on the cluster, I will resubmit the jobs.