Reports 1-1 of 1 Clear search Modify search
DetChar (General)
takahiro.yamamoto - 21:08 Sunday 03 August 2025 (34729) Print this report
Lack of some GWF files in cache files due to a delay on the data transfer

When I tried to automate a work in klog#34703, lack of GWF files in cache files were found.

Missing files I found are /data/KAGRA/raw/full/14382/K-K1_C-1438233056-32.gwf and /data/KAGRA/raw/full/14382/K-K1_C-1438245664-32.gwf. Although these two files surely exist on the Kashiwa cluster, they are not contained in the cache files. A relevant part of the cache file contents and the results of the ls command are shown below. Missing files have a larger delay between GPS time of the file head and file timestamp than other files. There may be a problem with cache file generation when the data transfer order is reversed for some reason.

/data/KAGRA/raw/full/14382/K-K1_C-1438233056-32.gwf
$ grep /data/KAGRA/raw/full/14382/K-K1_C-14382330 ~detchar/cache/Cache_GPS/14382.cache
/home/detchar/cache/Cache_GPS/14382.cache:K K1_C 1438233024 32 file://localhost/data/KAGRA/raw/full/14382/K-K1_C-1438233024-32.gwf
/home/detchar/cache/Cache_GPS/14382.cache:K K1_C 1438233088 32 file://localhost/data/KAGRA/raw/full/14382/K-K1_C-1438233088-32.gwf

$ grep /data/KAGRA/raw/full/14382/K-K1_C-14382330 ~detchar/cache/Cache_GPS/14382.ffl
/home/detchar/cache/Cache_GPS/14382.ffl:/data/KAGRA/raw/full/14382/K-K1_C-1438233024-32.gwf 1438233024 32 0 0
/home/detchar/cache/Cache_GPS/14382.ffl:/data/KAGRA/raw/full/14382/K-K1_C-1438233088-32.gwf 1438233088 32 0 0

$ ls -l --full-time /data/KAGRA/raw/full/14382/K-K1_C-14382330*
-rw-r--r-- 1 daq 471247162 2025-08-03 14:10:54.000000000 +0900 /data/KAGRA/raw/full/14382/K-K1_C-1438233024-32.gwf
-rw-r--r-- 1 daq 470978416 2025-08-03 14:12:17.000000000 +0900 /data/KAGRA/raw/full/14382/K-K1_C-1438233056-32.gwf
-rw-r--r-- 1 daq 471010349 2025-08-03 14:11:55.000000000 +0900 /data/KAGRA/raw/full/14382/K-K1_C-1438233088-32.gwf

/data/KAGRA/raw/full/14382/K-K1_C-1438245664-32.gwf
$ grep /data/KAGRA/raw/full/14382/K-K1_C-14382456 ~detchar/cache/Cache_GPS/14382.cache
K K1_C 1438245600 32 file://localhost/data/KAGRA/raw/full/14382/K-K1_C-1438245600-32.gwf
K K1_C 1438245632 32 file://localhost/data/KAGRA/raw/full/14382/K-K1_C-1438245632-32.gwf
K K1_C 1438245696 32 file://localhost/data/KAGRA/raw/full/14382/K-K1_C-1438245696-32.gwf

$ grep /data/KAGRA/raw/full/14382/K-K1_C-14382456 ~detchar/cache/Cache_GPS/14382.ffl
/data/KAGRA/raw/full/14382/K-K1_C-1438245600-32.gwf 1438245600 32 0 0
/data/KAGRA/raw/full/14382/K-K1_C-1438245632-32.gwf 1438245632 32 0 0
/data/KAGRA/raw/full/14382/K-K1_C-1438245696-32.gwf 1438245696 32 0 0

$ ls -l --full-time /data/KAGRA/raw/full/14382/K-K1_C-14382456*
-rw-r--r-- 1 daq 456124886 2025-08-03 17:40:27.000000000 +0900 /data/KAGRA/raw/full/14382/K-K1_C-1438245600-32.gwf
-rw-r--r-- 1 daq 456837281 2025-08-03 17:41:00.000000000 +0900 /data/KAGRA/raw/full/14382/K-K1_C-1438245632-32.gwf
-rw-r--r-- 1 daq 456100787 2025-08-03 17:42:18.000000000 +0900 /data/KAGRA/raw/full/14382/K-K1_C-1438245664-32.gwf
-rw-r--r-- 1 daq 456047603 2025-08-03 17:42:11.000000000 +0900 /data/KAGRA/raw/full/14382/K-K1_C-1438245696-32.gwf

Comments to this report:
takahiro.yamamoto - 10:11 Monday 04 August 2025 (34734) Print this report

Similar problem was (is still being?) repeated.
In the first two events, data transfer seemed to become timeout and failed files were sent the re-transfer process at 3am.
The third event occurred around 7am today and a failed file is still absence on Kashiwa cluster.
The failed file exists on Kamioka. So it will be probably re-send at next 3am.

/data/KAGRA/raw/full/14382/K-K1_C-1438263904-32.gwf
$ grep /data/KAGRA/raw/full/14382/K-K1_C-1438263[89] ~detchar/cache/Cache_GPS/14382.cache
K K1_C 1438263808 32 file://localhost/data/KAGRA/raw/full/14382/K-K1_C-1438263808-32.gwf
K K1_C 1438263840 32 file://localhost/data/KAGRA/raw/full/14382/K-K1_C-1438263840-32.gwf
K K1_C 1438263872 32 file://localhost/data/KAGRA/raw/full/14382/K-K1_C-1438263872-32.gwf
K K1_C 1438263936 32 file://localhost/data/KAGRA/raw/full/14382/K-K1_C-1438263936-32.gwf
K K1_C 1438263968 32 file://localhost/data/KAGRA/raw/full/14382/K-K1_C-1438263968-32.gwf

$ grep /data/KAGRA/raw/full/14382/K-K1_C-1438263[89] ~detchar/cache/Cache_GPS/14382.ffl
/data/KAGRA/raw/full/14382/K-K1_C-1438263808-32.gwf 1438263808 32 0 0
/data/KAGRA/raw/full/14382/K-K1_C-1438263840-32.gwf 1438263840 32 0 0
/data/KAGRA/raw/full/14382/K-K1_C-1438263872-32.gwf 1438263872 32 0 0
/data/KAGRA/raw/full/14382/K-K1_C-1438263936-32.gwf 1438263936 32 0 0
/data/KAGRA/raw/full/14382/K-K1_C-1438263968-32.gwf 1438263968 32 0 0

$ ls -l --full-time /data/KAGRA/raw/full/14382/K-K1_C-1438263[89]*
-rw-r--r-- 1 daq 482317613 2025-08-03 22:43:57.000000000 +0900 /data/KAGRA/raw/full/14382/K-K1_C-1438263808-32.gwf
-rw-r--r-- 1 daq 482233503 2025-08-03 22:44:27.000000000 +0900 /data/KAGRA/raw/full/14382/K-K1_C-1438263840-32.gwf
-rw-r--r-- 1 daq 481581101 2025-08-03 22:45:05.000000000 +0900 /data/KAGRA/raw/full/14382/K-K1_C-1438263872-32.gwf
-rw-r--r-- 1 daq 481812561 2025-08-04 03:00:08.000000000 +0900 /data/KAGRA/raw/full/14382/K-K1_C-1438263904-32.gwf
-rw-r--r-- 1 daq 481314142 2025-08-03 22:46:09.000000000 +0900 /data/KAGRA/raw/full/14382/K-K1_C-1438263936-32.gwf
-rw-r--r-- 1 daq 483906327 2025-08-03 22:46:36.000000000 +0900 /data/KAGRA/raw/full/14382/K-K1_C-1438263968-32.gwf


/data/KAGRA/raw/full/14382/K-K1_C-1438265376-32.gwf, /data/KAGRA/raw/full/14382/K-K1_C-1438265408-32.gwf
$ grep /data/KAGRA/raw/full/14382/K-K1_C-1438265[34] ~detchar/cache/Cache_GPS/14382.cache
K K1_C 1438265312 32 file://localhost/data/KAGRA/raw/full/14382/K-K1_C-1438265312-32.gwf
K K1_C 1438265344 32 file://localhost/data/KAGRA/raw/full/14382/K-K1_C-1438265344-32.gwf
K K1_C 1438265440 32 file://localhost/data/KAGRA/raw/full/14382/K-K1_C-1438265440-32.gwf
K K1_C 1438265472 32 file://localhost/data/KAGRA/raw/full/14382/K-K1_C-1438265472-32.gwf

$ grep /data/KAGRA/raw/full/14382/K-K1_C-1438265[34] ~detchar/cache/Cache_GPS/14382.ffl
/data/KAGRA/raw/full/14382/K-K1_C-1438265312-32.gwf 1438265312 32 0 0
/data/KAGRA/raw/full/14382/K-K1_C-1438265344-32.gwf 1438265344 32 0 0
/data/KAGRA/raw/full/14382/K-K1_C-1438265440-32.gwf 1438265440 32 0 0
/data/KAGRA/raw/full/14382/K-K1_C-1438265472-32.gwf 1438265472 32 0 0

$ ls -l --full-time /data/KAGRA/raw/full/14382/K-K1_C-1438265[34]*
-rw-r--r-- 1 daq 484744461 2025-08-03 23:08:59.000000000 +0900 /data/KAGRA/raw/full/14382/K-K1_C-1438265312-32.gwf
-rw-r--r-- 1 daq 484754345 2025-08-03 23:11:18.000000000 +0900 /data/KAGRA/raw/full/14382/K-K1_C-1438265344-32.gwf
-rw-r--r-- 1 daq 485229068 2025-08-04 03:00:32.000000000 +0900 /data/KAGRA/raw/full/14382/K-K1_C-1438265376-32.gwf
-rw-r--r-- 1 daq 484477094 2025-08-04 03:00:56.000000000 +0900 /data/KAGRA/raw/full/14382/K-K1_C-1438265408-32.gwf
-rw-r--r-- 1 daq 484117973 2025-08-03 23:11:41.000000000 +0900 /data/KAGRA/raw/full/14382/K-K1_C-1438265440-32.gwf
-rw-r--r-- 1 daq 483749699 2025-08-03 23:11:45.000000000 +0900 /data/KAGRA/raw/full/14382/K-K1_C-1438265472-32.gwf

/data/KAGRA/raw/full/14382/K-K1_C-1438294880-32.gwf
$ grep /data/KAGRA/raw/full/14382/K-K1_C-1438294[789] ~detchar/cache/Cache_GPS/14382.cache
K K1_C 1438294720 32 file://localhost/data/KAGRA/raw/full/14382/K-K1_C-1438294720-32.gwf
K K1_C 1438294752 32 file://localhost/data/KAGRA/raw/full/14382/K-K1_C-1438294752-32.gwf
K K1_C 1438294784 32 file://localhost/data/KAGRA/raw/full/14382/K-K1_C-1438294784-32.gwf
K K1_C 1438294816 32 file://localhost/data/KAGRA/raw/full/14382/K-K1_C-1438294816-32.gwf
K K1_C 1438294848 32 file://localhost/data/KAGRA/raw/full/14382/K-K1_C-1438294848-32.gwf
K K1_C 1438294912 32 file://localhost/data/KAGRA/raw/full/14382/K-K1_C-1438294912-32.gwf
K K1_C 1438294944 32 file://localhost/data/KAGRA/raw/full/14382/K-K1_C-1438294944-32.gwf
K K1_C 1438294976 32 file://localhost/data/KAGRA/raw/full/14382/K-K1_C-1438294976-32.gwf

$ grep /data/KAGRA/raw/full/14382/K-K1_C-1438294[789] ~detchar/cache/Cache_GPS/14382.ffl
/data/KAGRA/raw/full/14382/K-K1_C-1438294720-32.gwf 1438294720 32 0 0
/data/KAGRA/raw/full/14382/K-K1_C-1438294752-32.gwf 1438294752 32 0 0
/data/KAGRA/raw/full/14382/K-K1_C-1438294784-32.gwf 1438294784 32 0 0
/data/KAGRA/raw/full/14382/K-K1_C-1438294816-32.gwf 1438294816 32 0 0
/data/KAGRA/raw/full/14382/K-K1_C-1438294848-32.gwf 1438294848 32 0 0
/data/KAGRA/raw/full/14382/K-K1_C-1438294912-32.gwf 1438294912 32 0 0
/data/KAGRA/raw/full/14382/K-K1_C-1438294944-32.gwf 1438294944 32 0 0
/data/KAGRA/raw/full/14382/K-K1_C-1438294976-32.gwf 1438294976 32 0 0

$ ls -l --full-time /data/KAGRA/raw/full/14382/K-K1_C-1438294[789]*
-rw-r--r-- 1 daq 478908272 2025-08-04 07:19:10.000000000 +0900 /data/KAGRA/raw/full/14382/K-K1_C-1438294720-32.gwf
-rw-r--r-- 1 daq 480288904 2025-08-04 07:19:40.000000000 +0900 /data/KAGRA/raw/full/14382/K-K1_C-1438294752-32.gwf
-rw-r--r-- 1 daq 480812523 2025-08-04 07:20:13.000000000 +0900 /data/KAGRA/raw/full/14382/K-K1_C-1438294784-32.gwf
-rw-r--r-- 1 daq 480483696 2025-08-04 07:22:14.000000000 +0900 /data/KAGRA/raw/full/14382/K-K1_C-1438294816-32.gwf
-rw-r--r-- 1 daq 480454286 2025-08-04 07:21:17.000000000 +0900 /data/KAGRA/raw/full/14382/K-K1_C-1438294848-32.gwf
(/data/KAGRA/raw/full/14382/K-K1_C-1438294880-32.gwf)
-rw-r--r-- 1 daq 480316241 2025-08-04 07:23:27.000000000 +0900 /data/KAGRA/raw/full/14382/K-K1_C-1438294912-32.gwf
-rw-r--r-- 1 daq 480368106 2025-08-04 07:23:17.000000000 +0900 /data/KAGRA/raw/full/14382/K-K1_C-1438294944-32.gwf
-rw-r--r-- 1 daq 480390431 2025-08-04 07:23:38.000000000 +0900 /data/KAGRA/raw/full/14382/K-K1_C-1438294976-32.gwf

Search Help
×

Warning

×