I installed a new 10G switch at U35 of B1 rack (computer room in the mine) for the test of new data transfer servers.
Writing speed of k1fw1 to hyades-2 didn't improve in this situation.
Thanks to this new network switch, rebooting k1fw1 is not required for switching target disks of hyades-1 and -2.
-----
Originally, k1fw1 and hyades-1 were directly connected.
Because of this, we need to do a cold-boot in order to change a disk server from hyades-1 to hyades-2.
So we was able to do only 1 or 2 tests during one maintenance day.
In order to improve this situation, I installed a new 10G switch at U35 of B1 rack.
(After solving current problem, this switch should be uninstalled due to the rack space, thermal issue, etc.)
k1fw1 and hyades-1 are now connected via the new 10G switch and hyades-2 is also connected to the same switch.
A disk mount is set as follows.
- hyades-1:/data1 => k1fw1:/data1
- hyades-1:/data2 => k1fw1:/data2
- hyades-2:/data => k1fw1:/data9
Original situation is /frames/full, /frames/science, and /frames/trend which are symbolic links to /data2/full, /data2/science /data1/trend, respectively.
In this case, daqd writes down frame files to hyeades-1.
At first, I changed destinations of all three symbolic link as /data9/full, /data9/science, /data9/trend to switch the disk sytem as a new one on hyades-2.
DAQ hang-up was reproced at this time because it takes ~130s for writing 32 second-long data.
So direct connection of k1fw1 and hyades is not a cause of this problem.
As the next step, I set symbolic links as follows.
/frames/full -> /data9/full
/frames/science -> /data2/science
/frames/trend -> /data2/trend
In this case, full frames was written down on hyades-2 and science frames and trend frames were writting down on hyades-1.
DAQ hang-up was reproduced also in this case. But science frames can be written on hyades-1 with ~3s.
A cause of hang-up was the writing time of full frames (~120s).
I also tried to write full frames and science frames to hyades-1 and hyades-2, respectively (swapped destination from above situation).
In this case, writing time on hyades-1 is ~7s (full).
But writing time on hyades-2 is ~110s (science) and daqd hung up.
Because file size of full frames is roughly twice larger than one of science frames.
So writing time on hyades-1 seems to be reasonable (7s vs. 3s).
On the other hand, there seems to be ~100s overhead on hyades-2.
It's better to check the NIC and NFS settings on hyades-2.
By changing these options, writing speed is drastically improved (10~500 times faster).
I did just a simple disk-writing test. Writing frames by daqd will be tested on next Friday.
-----
In /etc/fstab, following options are set on k1fw1 for the connection to hyades-1, and -2.
sync,hard,intr,rw,nolock,rsize=8192,wsize=8192
Connection between k1fw0 and hyades-0 is same as k1fw1 and hyades-1.
On the other hand, enabled options are as follows.
rw,relatime,rsize=1048576,wsize=1048576,hard,local_lock=none for hyades-0 and -1
rw,sync,relatime,rsize=8192,wsize=8192,hard,nolock,local_lock=all for hyades-2
sync, rsize, wsize and nolock options are ignored on hyades-0 and -1 but they are enabled on hyades-2.
And also, local_lock which is not set on /etc/fstab shows different values.
So I tried to modify /etc/fstab in order to set same options which were enabled between hyades-0, -1 and hyades-2.
Finally, writing speed by the cp command and simple programs on hyades-1 became same one on hyades-2 by removing sync and setting 1MiB for rsize and wsize.
nolock and local_lock options doesn't seem to affect writing speed.
Final mount options are as follows.
hard,intr,rw,rsize=1048576,wsize=1048576,local_lock=none
I'll try frame dump by daqd with these options on next Friday.
k1fw1 is now connected to hyades-1. It will be replaced after the data transfer to Kashiwa will be established.
-----
Writing frames by daqd was tested on hyades-2 with new NFS configurations.
With previous configurations, daqd took ~130s for writing a 32-second long frame.
Because of too slow writing speed, daqd repeatedly hung up (See also klog#22520).
NFS performance was drastically improved by changing the NFS configuration.
In current configuration, daqd takes ~7s for writing a 32-second long frame, which is same performance of a combination of k1fw0 and hyades-0 as shown in Fig.1.
After this test, k1fw1 writes frames to the disk on hyades-1 because data transfer from hyades-2 to Kashiwa haven't been established yet.
So after preparing a new data transfer path, I will officially switch from hyades-1 to hyades-2 as a disk server of k1fw1.
