KAGRA Logbook

DGS (General)

takahiro.yamamoto - 20:54 Wednesday 13 August 2025 (34822)

Frame Writers become unstable after recovering k1dc0

After Data Concentrator came back online, daqd on Frame Writer seemed to become unstable.

3 hang-ups on daqd@k1fw0 and 1 hang-up on daqd@k1fw1 were detected during 08/12 16:00 JST - 08/13 20:00 JST.
Lost frames on each Frame Writer were as follows.

k1fw0
[full] 2025-08-13 06:09:50 UTC /frame0/full/14391/K-K1_C-1439100608-32.gwf 2025-08-13 06:10:22 UTC /frame0/full/14391/K-K1_C-1439100640-32.gwf 2025-08-13 09:52:46 UTC /frame0/full/14391/K-K1_C-1439113984-32.gwf 2025-08-13 09:53:18 UTC /frame0/full/14391/K-K1_C-1439114016-32.gwf 2025-08-13 10:23:42 UTC /frame0/full/14391/K-K1_C-1439115840-32.gwf 2025-08-13 10:24:14 UTC /frame0/full/14391/K-K1_C-1439115872-32.gwf [science] 2025-08-13 06:09:50 UTC /frame0/science/14391/K-K1_R-1439100608-32.gwf 2025-08-13 06:10:22 UTC /frame0/science/14391/K-K1_R-1439100640-32.gwf 2025-08-13 09:52:46 UTC /frame0/science/14391/K-K1_R-1439113984-32.gwf 2025-08-13 09:53:18 UTC /frame0/science/14391/K-K1_R-1439114016-32.gwf 2025-08-13 10:23:42 UTC /frame0/science/14391/K-K1_R-1439115840-32.gwf 2025-08-13 10:24:14 UTC /frame0/science/14391/K-K1_R-1439115872-32.gwf

k1fw1
[full] 2025-08-12 13:41:34 UTC /frame9/full/14390/K-K1_C-1439041312-32.gwf 2025-08-12 13:42:06 UTC /frame9/full/14390/K-K1_C-1439041344-32.gwf [science] 2025-08-12 13:41:34 UTC /frame9/science/14390/K-K1_R-1439041312-32.gwf 2025-08-12 13:42:06 UTC /frame9/science/14390/K-K1_R-1439041344-32.gwf

And also, trend frames were 0-filled during above time though GWF files themselves were exists.

k1fw0
[trend/second] /frame0/trend/second/14391/K-K1_T-1439100600-600.gwf /frame0/trend/second/14391/K-K1_T-1439115600-600.gwf /frame0/trend/second/14391/K-K1_T-1439113800-600.gwf [trend/minute] /frame0/trend/minute/14391/K-K1_M-1439100000-3600.gwf /frame0/trend/minute/14391/K-K1_M-1439110800-3600.gwf /frame0/trend/minute/14391/K-K1_M-1439114400-3600.gwf

k1fw1
[trend/second] /frame9/trend/second/14390/K-K1_T-1439041200-600.gwf [trend/minute] /frame9/trend/minute/14390/K-K1_M-1439038800-3600.gwf

Comments to this report:

takahiro.yamamoto - 23:01 Wednesday 13 August 2025 (34825)

Attached figure was comparison plot of Science-mode flag taken via NDS0 (upper window) and NDS1 (lower window). Due to ndscope specifications, missing data is regarded as 0.

Three orange arrows and 1 red arrow represent unexpected daqd restart on k1fw0 and k1fw1 (see also klog#34822), respectively. An yerrow arrow represents one occurred on k1fw0 after posting klog#34822. As we can see that 4 of 5 events chopped Science-mode. So it must be filled by using data on both streams at least for the data during Science-mode.

Because daqd on Frame Writer can be often runing longer than 1hr continuously, it seems not to be related to the writing process. Now I have no enough information to locallize a problematic point and need to continue the investigation. Until this issue was solved, careful treatment on raw data transfer, and segment upload on DQSEGDB (and what else?) will be required.

By the way, Low-latency calibration stream isn't affected by this issue.

Images attached to this comment

nobuyuki.kanda - 21:49 Thursday 14 August 2025 (34835)

Currently (after switching hyades-2), bulk data transfer paths separately : (1) fw0 -> hyades-2 -> kagra-dsr-b1, (2) fw1 -> hyades-1 -> aldebaran.
In my understanding, this does not follow the previous system of prioritizing the first data that arrived at aldebaran, or complementing the data that arrived from either hyades-0 or 1. So, we have to send by hand for complemental data-set from aldebaran to Kashiwa system.

And, thank you very much for
> I made copy of GWF files on k1fw0 and k1fw1 during problematic time as ~takahiro.yamamoto/share/MissingFiles_klog34822.