KAGRA Logbook

DGS (General)

shoichi.oshino - 15:39 Monday 19 August 2019 (10029)

Weekly Maintenance on Aug. 19, 2019

[Miyakawa, Yamamoto, Oshino]

1. We replaced k1dc0 to new PC.
During this maintenance, frame files were not recorded.
Down time was 09:56-10:08.

2. We replaced k1ioo1 to faster CPU.
Because of limited number of CPUs, we changed CPU id of some models(k1iopioo1, k1alsfib, k1psliss).

3. We added one BIO cards to k1als0.
After these works, we restarted k1als0 and k1ioo1 and then connected to Dolphin network.

After this maintenance, we found that only 16GB memory was installed on k1dc0.
Because the daqd was using 96% of memory, we added memory on k1dc0.
Down time was 13:52-14:03.

Comments to this report:

takahiro.yamamoto - 16:39 Monday 19 August 2019 (10030)

After k1dc0 replacement, daqd's unknown hung-up occurred 3 times.

[Mon Aug 19 14:24:31 JST 2019] dc0: Producer: mean=0.0625453 max=0.938204 min=0.0593059 stddev=0.00598433 n=22093
[Mon Aug 19 14:48:04 JST 2019] dc0: Producer: mean=0.0625467 max=0.967852 min=0.0524828 stddev=0.00631834 n=21453
[Mon Aug 19 16:34:37 JST 2019] dc0: Producer: mean=0.0625099 max=0.972969 min=0.04353 stddev=0.00306776 n=101135

takahiro.yamamoto - 19:59 Monday 19 August 2019 (10034)

daqd hung up again. Hung-up rate seems to become worse...

[Mon Aug 19 19:27:10 JST 2019] dc0: Producer: mean=0.0625061 max=1.01388 min=0.0459921 stddev=0.0026647 n=164493

takahiro.yamamoto - 21:35 Monday 19 August 2019 (10035)

I cannot stop periodic hung-ups...
I manually stopped daqd and booted up as a debug mode.
[Mon Aug 19 21:28:01 JST 2019] dc0: Session terminated, killing shell... ...killed.

[Mon Aug 19 20:23:43 JST 2019] dc0: Producer: mean=0.062519 max=0.961998 min=0.059294 stddev=0.00403983 n=53133
[Mon Aug 19 20:24:36 JST 2019] dc0: Producer: mean=0.0639772 max=0.968132 min=0.0423481 stddev=0.0360086 n=635
[Mon Aug 19 20:35:49 JST 2019] dc0: Producer: mean=0.0626042 max=0.960673 min=0.059305 stddev=0.00922213 n=9613
[Mon Aug 19 21:10:42 JST 2019] dc0: Producer: mean=0.0625311 max=0.953344 min=0.059305 stddev=0.00513705 n=32335
[Mon Aug 19 21:23:35 JST 2019] dc0: Producer: mean=0.0625896 max=0.982438 min=0.059325 stddev=0.00875799 n=11214

takahiro.yamamoto - 22:29 Monday 19 August 2019 (10036)

Even in the debug mode, I cannot obtain any additional information.

WARNING: GPS time jumped from 1250255713 (812500000) to 1250255714 (875192000)
Producer: mean=0.0625264 max=0.0830319 min=0.046186 stddev=0.00184026 n=5775
net_writer_c::consumer(): sending block 19 of 1635220 bytes
WARNING: GPS time jumped from 1250255713 (875000000) to 1250255714 (954691000)
Producer: mean=0.0626734 max=0.911307 min=0.046186 stddev=0.0113187 n=5776
framexmit write gps=1250255713 period=625000000
tp framexmit write gps=1250255713 period=625000000
net_writer_c::consumer(): sending block 19 of 1635220 bytes
GPS card time jumped from 1250255713 (937500000) to 1250255715 (39149000)
Producer: mean=0.0626773 max=0.911307 min=0.046186 stddev=0.0113216 n=5777

[Mon Aug 19 22:07:39 JST 2019] dc0: Producer: mean=0.0625273 max=0.965739 min=0.0448232 stddev=0.00490379 n=36818
[Mon Aug 19 22:14:57 JST 2019] dc0: Producer: mean=0.0626773 max=0.911307 min=0.046186 stddev=0.0113216 n=5777
[Mon Aug 19 22:24:55 JST 2019] dc0: Producer: mean=0.0626204 max=0.967821 min=0.0458889 stddev=0.0100207 n=8331

takahiro.yamamoto - 8:26 Tuesday 20 August 2019 (10041)

[Mon Aug 19 22:34:13 JST 2019] dc0: 1250256869
[Mon Aug 19 23:02:12 JST 2019] dc0: Producer: mean=0.0625391 max=0.995628 min=0.0449781 stddev=0.00605443 n=25614
[Mon Aug 19 23:27:50 JST 2019] dc0: Producer: mean=0.0625429 max=0.973121 min=0.0430031 stddev=0.00660529 n=23374
[Tue Aug 20 01:19:29 JST 2019] dc0: Producer: mean=0.0625095 max=1.00021 min=0.0459452 stddev=0.00328784 n=105939
[Tue Aug 20 01:30:42 JST 2019] dc0: Producer: mean=0.0626044 max=0.966839 min=0.0593672 stddev=0.0092727 n=9614
[Tue Aug 20 02:20:35 JST 2019] dc0: Producer: mean=0.0625214 max=0.965596 min=0.0438421 stddev=0.00437817 n=46733
[Tue Aug 20 02:24:48 JST 2019] dc0: Producer: mean=0.0628465 max=0.977072 min=0.0442629 stddev=0.0170418 n=2895
[Tue Aug 20 03:49:21 JST 2019] dc0: Producer: mean=0.0625127 max=0.952153 min=0.0593081 stddev=0.0033663 n=80013
[Tue Aug 20 04:23:54 JST 2019] dc0: Producer: mean=0.0625313 max=0.969231 min=0.0427239 stddev=0.00520415 n=32014
[Tue Aug 20 04:24:47 JST 2019] dc0: Producer: mean=0.063981 max=0.953663 min=0.0423641 stddev=0.0354399 n=635
[Tue Aug 20 04:33:00 JST 2019] dc0: Producer: mean=0.0626487 max=1.00025 min=0.0430729 stddev=0.0115183 n=6733
[Tue Aug 20 05:52:53 JST 2019] dc0: Producer: mean=0.0625135 max=0.942263 min=0.0433071 stddev=0.00345328 n=75530
[Tue Aug 20 06:06:06 JST 2019] dc0: Producer: mean=0.062587 max=1.00718 min=0.0470231 stddev=0.00885905 n=11534
[Tue Aug 20 06:46:19 JST 2019] dc0: Producer: mean=0.0625267 max=0.961998 min=0.04531 stddev=0.00489838 n=37454

takahiro.yamamoto - 1:08 Wednesday 21 August 2019 (10061)

[Oshino, Yamamoto]

We swapped a new computer (E5-2623v3) for an old computer (X5690).
After swapping, repeated hangup was stopped.
'top' command shows that CPU usage is 10-15% lower with X5690 (~80%) than with E5-2623v3 (~95%).

It may be better to
- use more faster CPU
- rebuild daqd on new (E5-2623v3) CPU.