CAL (General)takahiro.yamamoto - 21:01 Saturday 18 October 2025 (35374)
Print this reportWrong routing for LL data transferWhile investigating the issue of missing data transfers related to the k1gate restart in klog#35318, I finally found that the network settings on the low-latency server were incorrect.
LL server is connected to the DMT network for receiving data from DAQ servers and the DMG network for sending LL frames. I couldn't find exact log about the routing configuration for LL data transfer, it had to be done as a part of work in klog#32886 on March 3rd. According to the discussion in e-mail on around that day, routing configuration seemed to be done by route add command. I guess configuration has been done on that day properly. But route add command can make just runtime settings not permanent settings. And also, LL server was rebooted before joining O4c. So routing configuration was vanished and LL data is being transferred via general ICRR network instead of DMG network.
I'm not sure using general ICRR network has made additional latency and packet loss inducing retransmission or not. Probably it's not so serious for now because nobady complained about this issue. But it's not good situation that maintenance in DGS and/or ICRR private network affects LL data transfer. So it should be fixed. I plan to do it next Monday because it doesn't affect to DGS network and commissioning activities.
Comments to this report:
takahiro.yamamoto - 19:13 Monday 20 October 2025 (35382)
Print this report[Kanda, YamaT]
We set a correct routing for the LL data transfer. Now LL data is being transferred via DMG network. We expected to switch over without any downtime at first, but the receiver application at Kashiwa also needed to be restarted. This was likely because the IP address of sender at Kamioka changed from the receiver's perspective, existing sockets cannot be reused.
We also set same settings on the secondary server.
----- Connection to the receiver at Kashiwa was configured as through the DMG VPN device by using the "route add" command. This is working a runtime configuration. In addition, I made a permanent routing by using the "nmcli" command. NIC hasn't restarted yet, so a runtime setting by "route add" is only working now. When NIC or the server itself will be restarted, a runtime settings will be vanished and a permanent settings by "nmcli" will be enabled.
Though we made these settings and it worked fine according to the traceroute, ping, etc., a new LL frames didn't appears on Kashiwa at first. LL data transfer was resumed after restarted framelink and receiver at Kashiwa.
There was some missing data in LL h-of-t but it's not serous because of out-of-observing.