Reports of 23401
DGS (General)
takahiro.sawada - 5:35 Friday 09 June 2023 (25560) Print this report
Relationship between IPC glitch and the (new) gwistat monitor page, + etc.

Issue #1
The k1bcst0 transmits 1 second-long data every second. However, when an IPC glitch occurs, the transmission occurs at intervals slightly less than 1 second from the previous transmission (Approximately 0.000 to 0.004 seconds faster. However, this number is not an accurate representation of the actual time of change, as it is not known how quickly the Frame_Log used for testing can respond to the arrival of a new frame.) This data is input to the low latency calibration pipeline, and its output is transferred to the Kashiwa by Framelink and further transferred to the CIT by Kafka. The IPC glitched data arriving at CIT was delayed about 1 second more than normal frames. As a result, the gwistat displayed on KAGRA status switches from Observing to Down, and Duration is reset to 0, even though KAGRA keeps locked and observing state. It is not yet known at why the data is transmitted early in the first place and what stage the data transmitted early is slowed down (Negative delay@k1bcst0 --> positive delay@CIT).

Issue #2
In low latency data stream, there is a loss of one frame approximately every few 10k seconds, which may also be resetting Duration on gwistat to 0. It is not yet clear at what stage this missing is occurring.

Issue #3
KAGRA status is sometimes shown as Down on the gwistat page even though the KAGRA interferometer is in Observing Mode and the summary page in KAGRA shows it as Observing Mode. This phenomenon occurs with a frequency of about a few hours per day. It is not yet clear what is causing it yet.

DMG (Data transfer & archiving)
takahiro.yamamoto - 2:04 Friday 09 June 2023 (25559) Print this report
Comment to Missing data on Kashiwa due to network error (25505)
These frames seemed to be re-sent.
They are now available on Kashiwa cluster (nds2).
DGS (General)
takahiro.yamamoto - 1:13 Friday 09 June 2023 (25557) Print this report
Comment to Unexpected reset of GigE settings (25543)
Today some GigE movies were stopped again and rebooted.

-----
Memory usage by the free command was very small (~500MiB).
And also free region was also very small (~200MiB).
Most of memory region was categorized as "buff/cache" (~31GiB).

I checked the Slab cache as the next.
According to the slabtop, most of memory (~99%) was used by kmalloc-1024.
The details of Slab cache is as follows and most of them are uncollectable.
$ cat /proc/meminfo
Slab: 32101440 kB
SReclaimable: 27112 kB
SUnreclaim: 32074328 kB

So it cannot be released by using /proc/sys/vm/drop_caches.

And also, by seeing /proc/buddyinfo, we can know that fragments of memory region occurs on k1cam1.
It cannot be also improved by using /proc/sys/vm/compact_memory.
Memory leakage might occurs on some processes...
Anyway, k1cam1 should be rebooted once in order to clean up uncollectable RAM.
OBS (Summary)
hiromi.yasui - 0:08 Friday 09 June 2023 (25558) Print this report
Operation Shift Summary

Operators name: Nakagaki, Yasui

Shift time: 16-24(JST)

Summary:

VAC, CRY, and TEMP had no problem.

16:00 OBSERVATION from the previous shift time

21:50 LOCK LOSS (21:49 M3.2 earthquake on the Noto Peninsula)

21:58 OBSERVATION (recovered automatically)

24:00 still in OBSERVATION

PEM (Center)
tomotada.akutsu - 18:22 Thursday 08 June 2023 (25556) Print this report
Comment to Current daytime ground motion (25158)
MIF (General)
hirotaka.yuzurihara - 17:20 Thursday 08 June 2023 (25555) Print this report
Investigation of the relationship between the beam spot, the structure of 300~400Hz, and the inspiral range

I started to investigate the relationship between beam spot on ETMX and sensitivity curve of 300~400Hz. Summary slide is uploaded on JGW doc,  [slide].

Motivation is to find the characteristic relationship between the beam spot, the structure of 300~400Hz, and the inspiral range. For example, when the beam spot is off-center to specfic direction, the characteristic structure on the sensitivity curve of 300~400Hz will be mitigated or the inspiral range will be higher/lower than usual.

About plots : 

  • left bottom : TCam image which was taken during PR3 alignment. At that time, we record the good oplev value of PR3.
  • left top : time series of inspiral range, several minutes after the interferometer in observing mode.
  • right top : whitened spectrogram of h(t), several minutes after the interferometer in observing mode. We focus on the specific frequency band 300 ~ 400Hz.
  • right bottom : ASD of h(t) channel, several minutes after the interferometer in observing mode.  We fixed the frequency band and y-range.

Comments for the result

  • The case of 5/31 (shown in page 8) might be good example. In the PR3 alignment, the beam spot is located at left lower from center [TCam image]. The inspiral range is lower than usual. ASD also shows broad bump.
    • I'm not sure this is a regular relationship. We need more samples.
  • The case of 6/4 (shown in page 12) doens't show the characteristic large bump on 300 ~ 400Hz. I'm not sure the reason.

Comments for the further invetigation are very helpful!

Images attached to this report
OBS (General)
takashi.uchiyama - 16:22 Thursday 08 June 2023 (25554) Print this report
Updated the reference sheet for check items of cryogenics

Kimura, Uchiyama

2023/06/08

Kimura-sama updated the reference sheet for check items of cryogenics and URL of the sheet was pasted on the page of "Check Items during Shift".

Images attached to this report
OBS (Summary)
satoru.ikeda - 15:44 Thursday 08 June 2023 (25553) Print this report
Operation Shift Summary

Operators name: Miyakawa, Ushiba, Ikeda

Shift Time: unscheduled operations

Summary:

13:52 PR3 Alignment due to IPC glitch 
 => Miyakawa's PR3 Alignment Training
15:11 OBSERVATION

OBS (Summary)
Nobuyuki Tanaka - 8:00 Thursday 08 June 2023 (25552) Print this report
Operation Shift Summary
Operators name: N.Sato, N.Tanaka

Shift time: 00-08 (JST)
Summary:
Check Items:
VAC,CRY,TEMP There was no problem.

00:00 OBSERVING
No Lock loss.
08:00 OBSERVING
OBS (Summary)
koji.nakagaki - 7:22 Thursday 08 June 2023 (25551) Print this report
Operation Shift Summary

Operators name: Yasui, Nakagaki
Shift time: 16-24 (JST)

Summary:

Check Items:
VAC,CRY,TEMP There was no problem.

16:00 OBSERVING from previous shift time
19:22 LOCK LOSS (corresponding earthquake is unknown)
19:32 OBSERVING (automatically recovered)
24:00 still in OBSERVING

DetChar (General)
hirotaka.yuzurihara - 7:21 Thursday 08 June 2023 (25550) Print this report
Segment production for missing/broke frame files

I started to test the segment production for missing/broken frame files. The script checks the list of missing/brok frame files, which was prepared by DMG. (see klog#25341)

The result segment file can be seen at /home/detchar/git/kagra-detchar/tools/Segments/Script/tmp/K1-DET_FRAME_AVAILABLE/2023/ , at Kashiwa cluster.

PEM (Center)
shinji.miyoki - 7:16 Thursday 08 June 2023 (25548) Print this report
Comment to Current daytime ground motion (25158)
  • The distances from the road construction area near the SK entrance to the KAGRA corner station: ~ 400m 
    • Excavators work for shaping the ground. ---> This is generating the excess seismic noise at MCF/IX(Y)V.
    • Dump truck transportation for filling ground.
  • The distances from the future tunnel construction area around "Do" area to the KAGRA corner station: ~440m ~ 680m
    • Blasting
    • Excavators work for removing rocks outside.
    • Dump and concrete mixer truck transportation.
  • The distances from the Sakonishi construction area to the KAGRA EX station: ~ 1300m
  • The distances from the Sakonishi construction area to the KAGRA Corner station: ~ 1800m
  • The distances from the Sakonishi construction area to the KAGRA EY station: ~ 4000m
    • Excavators work for moving/resetting rocks.
    • Rock crushing with a jackhammer.
    • Dump and concrete mixer truck transportation
DGS (General)
takahiro.yamamoto - 18:23 Wednesday 07 June 2023 (25549) Print this report
Comment to Weekly maintenance on Jun. 6 (25522)
As discussed in the morning briefing, I fixed the bug in IFO guardian during the time for PR3 alignment.
Unnecessary speech during the lock acquisition can be suppressed by this modification.
OBS (Summary)
satoru.ikeda - 15:54 Wednesday 07 June 2023 (25547) Print this report
Operation Shift Summary

Operators name: Ushiba, Ikeda

Shift Time: unscheduled operations

Summary:

13:57 PR3 Alignment due to IPC glitch 
13:20 request PRFPMI_RF_LOCKED
 => During the process, Ushiba-san LOCKED up manually.
15:53 OBSERVATION
 

DGS (General)
satoru.ikeda - 15:47 Wednesday 07 June 2023 (25546) Print this report
Comment to Unexpected reset of GigE settings (25543)

Ushiba-san, Ikeda.

GigE camera settings that had a value of 0 after restart have been restored to the value of K-Log#25214.

MCE_TRANS, MC_REFL, POP, TMSX_G, TMSX_IR, TMSY_G, TMSY_IR
 ⇒Restored
AS, IFO_REFL, OMC_TRANS, PMC_REFL, PMC_TRANS
 ⇒Unchanged

Non-image files attached to this comment
PEM (Center)
shinji.miyoki - 14:35 Wednesday 07 June 2023 (25545) Print this report
Comment to Current daytime ground motion (25158)

Here is the ground motion comparison between MCF and "E"XYV. By eyeballs, the motion at MCF cannot be seen at EXYV. (Be careful of the vertical scale difference.)

Images attached to this comment
DGS (General)
takahiro.yamamoto - 13:28 Wednesday 07 June 2023 (25543) Print this report
Unexpected reset of GigE settings

Abstract


This is an inspection of GigE trouble reported in klog#25542.
Camera services were (maybe accidentally) restarted.
Around that time, some kernel module process went to out of control.

Details


According to the screen captures, camera settings seemed to be reset between 7:50 JST and 8:00 JST.

When I login camera server, power management module (acpi_pad) went out of control on k1cam1 as shown in Fig.1 and it's difficult to input something from a keyboard.
So I removed acpi_pad process at first and a keyboard response came back.
Because power management function is unnecessary for k1cam, it can be removed from the kernel module list which are launched at booting up computer after O4a.

I'm not sure this out of control is related to the camera trouble, camera service have surely been restarted as shown in Fig.2.

I don't know why system locale is set EDT.
It's not a problem but in addition to this, NTP doesn't work on camera server and system time is ~5min different from correct time.
So it's very tough work to confirm the time of error log.

Anyway, camera service was restarted around 8:00 JST and parameters were reset at that time.
Images attached to this report
Comments to this report:
satoru.ikeda - 15:47 Wednesday 07 June 2023 (25546) Print this report

Ushiba-san, Ikeda.

GigE camera settings that had a value of 0 after restart have been restored to the value of K-Log#25214.

MCE_TRANS, MC_REFL, POP, TMSX_G, TMSX_IR, TMSY_G, TMSY_IR
 ⇒Restored
AS, IFO_REFL, OMC_TRANS, PMC_REFL, PMC_TRANS
 ⇒Unchanged

Non-image files attached to this comment
takahiro.yamamoto - 1:13 Friday 09 June 2023 (25557) Print this report
Today some GigE movies were stopped again and rebooted.

-----
Memory usage by the free command was very small (~500MiB).
And also free region was also very small (~200MiB).
Most of memory region was categorized as "buff/cache" (~31GiB).

I checked the Slab cache as the next.
According to the slabtop, most of memory (~99%) was used by kmalloc-1024.
The details of Slab cache is as follows and most of them are uncollectable.
$ cat /proc/meminfo
Slab: 32101440 kB
SReclaimable: 27112 kB
SUnreclaim: 32074328 kB

So it cannot be released by using /proc/sys/vm/drop_caches.

And also, by seeing /proc/buddyinfo, we can know that fragments of memory region occurs on k1cam1.
It cannot be also improved by using /proc/sys/vm/compact_memory.
Memory leakage might occurs on some processes...
Anyway, k1cam1 should be rebooted once in order to clean up uncollectable RAM.
PEM (Center)
shinji.miyoki - 12:25 Wednesday 07 June 2023 (25544) Print this report
Comment to Current daytime ground motion (25158)

I confirmed that the activities of excavators near the Kamioka mine company office generated the seismic noise enhancement at MCF and IXV as the attached file.

The numbers in the graph completely matched the excavators' activities status by my eyeballs.

  1. I passed through the construction area and two excavators stopped (no engine sound at this time). I parked my car near the SK entrance and started monitoring the MCF/IXV ground motion via the EDUROM network.
  2. I noticed the ground motion enhancement and moved my car to the near area of construction area where the mobile phone EM can be scarcely reachable. So I could keep monitoring the MCF/IXV ground motion. Only one excavator was working.
  3. The second excavator started working. Then the excitation level was enhanced more!
  4. One excavator stopped. So only one excavator was working.
  5. One excavator started working. So two excavators were working.
  6. One excavator stopped. So only one excavator was working.
  7. One excavator started working. So two excavators were working.
  8. Both excavators stopped.
  9. Two excavators started working.
  10. Both excavators stopped.

The sharp enhancement seemed to correspond to the horizontally large angle swing of arm of the excavator. I could not check the engine status when they stopped motions.

Images attached to this comment
OBS (Summary)
masahide.tamaki - 8:42 Wednesday 07 June 2023 (25542) Print this report
Operation Shift Summary

Operators name: M.Aoumi, M.Tamaki

Shift time: 00-08 (JST)
Summary:
Check Items:Vac, CRY,TEMP → There was no problem.
07:41 Lock loss(Automatic recovery).
08:14 OBSERVING

Although there were several earthquakes, we could keep the lock.

btw, it looks like the camera exposure for IMC REFL has been set to 0 before one knows (or maybe i just dont know).

VAC (General)
takaaki.yokozawa - 3:37 Wednesday 07 June 2023 (25541) Print this report
Comment to Current vacuum monitor (25517)
[Ikeda, Hayakawa, Yokozawa]

Ikeda-san noticed that the used memory was almost full (Reason is unknown).
(Even sometimes we could not perform ls)
So we decided to reboot the raspberry pi by unplug and plug the power supply at Xend.
After come back to analysis building, the memory problem was solved, but one USB was not recognized.
I enter the mine again and found unplugged USB, after fix it, all raspberry pi at Xend start working.
Anyway, please let me know if you noticed some trouble in vacuum monitor.
VAC (General)
takahiro.yamamoto - 2:09 Wednesday 07 June 2023 (25540) Print this report
Comment to Current vacuum monitor (25517)
I'm not sure that following things are related to the instability of the vacuum monitor or not.
They are just my impression by seeing raspberry pi itself, script, etc.

On some of raspberry pi, system time is several days different from correct date.
=> There is some possibility to appear conflicts related to a file timestamp.

Some kind of errors related to Wi-Fi/bluetooth are periodically shown in dmesg, journalctl, etc.
=> Disabling them helps to reduce CPU load. Because wired LAN is used on vacuum monitors(?), wireless communication can be stopped.

Bundling SSH sessions may reduce the authentication load.
=> e.g. "-oControlMaster auto -oControlPath ~/.ssh/vacuum-%r@%h:%p -oControlPersist 600"

EOL treatment should be done correctly in the cc-10 script.
=> Though cc-10 returns CR, but serial.readline() expects "LF" as EOL. So serial.readline() is always finished by timeout.

Error log by vacuum.sh becomes always blank. So we cannot do bug checks at all.
=> Redirect ">" is used instead of ">>". (Maybe it's just a coding bug.)
OBS (Summary)
hiromi.yasui - 0:08 Wednesday 07 June 2023 (25539) Print this report
Operation Shift Summary

Operators name: Nakagaki, Yasui
Shift time: 16-24(JST)
Summary:

VAC, CRY, TEMP; There was no problem.
The vacuum monitor problem that had been going on since yesterday(klog#25516) has not occurred for this shift time.

16:00 OBSERVATION from previous shift time
18:18 LOCK LOSS (18:17 M3.0 earthquake at Hida region)
          While LOCKING, the IFO guardian spoke an alert message. →Yamamoto-san is handling this matter(klog#25538).
18:33 OBSERVATION (recovered automatically)
24:00 still in OBSERVATION

DGS (General)
takahiro.yamamoto - 19:41 Tuesday 06 June 2023 (25538) Print this report
Comment to Weekly maintenance on Jun. 6 (25522)
I found that a new implementation of IFO guardian was bad.

It works well when ISS goes to DOWN before LSC_LOCK goes to DOWN as I have expected.
But it unnecessarily speaks during the LOCK acquisition.

When IFO guardian goes up from LOCKING to LOCKED, it speaks once the message which
should be heard in the case of going down from OBSERVING to LOCKED.

So it should be modified again in order to avoid confusing operators...

Because loading IFO guardians makes gap of science mode if it's loaded in OBSERVING,
we should load it when IFO is down.
PEM (Center)
shinji.miyoki - 17:28 Tuesday 06 June 2023 (25537) Print this report
Comment to Current daytime ground motion (25158)

I went to the Sakonishi-area and the road construction area near the Kamioka mine company office from 15:20~16:30.

No activities using heavy machinery around the Sakonishi area.

While there was excavator activity at the road construction area.

When I passed this area around 15:20, there was no activities of the excavator. After that, I went to the Sakonishi area, and there were no activities. After that, I went to the KAGRA entrance to get an internet connection and I monitored the ground motion, then I confirmed that the level denoted by green in the figure was low. After several decades of minutes, I noticed that the ground motion level was enhanced again. So, firstly I went to Sakonishi-area, but no activities. Then I went to the road construction area and found the excavator was working. I stayed near the SK entrance to obtain the EDUROM net to monitor the ground motion and the excavator activities. After the activity of the excavator was postponed, the ground level seemed to become low. Although I kept staying, the excavator did not restart its working until around 16:30.  I gave up and went back to the KAGRA DAB. However, the figure showed one more enhancement of the ground just before 17:00. I should be patient :(.

I would like to go there and check again if there is the ground excitation again tomorrow.

On the other hand, some activities around the Sakonishi-area was informed from the mid of June. So it is also important to check the relation.

Images attached to this comment
OBS (SDF)
takahiro.yamamoto - 16:27 Tuesday 06 June 2023 (25536) Print this report
[O4a] SDF accept

This is a duplication post of CAL measurement (klog#25532).

I accepted the EPICS records related to the weekly calibration measurement as shown in Fig.1.
Channel list is also available in Secsion 1.2 of JGW-L2314962.

Today, I loaded down.snap once after accepting changes on observation.snap.
After then, I surely re-loaded observation.snap again and revert changes coming from
numerical rounding errors.
So I hope that troubles such as klog#25462 will not be occurred.

Images attached to this report
Search Help
×

Warning

×