[yamat, yuzu]
Today, 10:30~12:40, I performed the stress tests for each real-time model by reading multiple channels via nds. We concluded the ASC model is problematic for reading multiple channels. Although I developed the shell script to avoid the DAQ issue, it failed this time. We will test another idea to solve the situation, when I can try.

Details

After several tests, we concluded the issue is related to the ASC model. For the other models (ASCADS and VIS), the daq issue didn't occur.

• 20 channels from ASCADS model (2 kHz sampling)
  • channel list: /users/yuzu/work/nds_stress_test/channels_ADS_2kHz_20.txt . There was no DAQ down
• 20 channels from VIS model (2 kHz sampling)
  • channel list: /users/yuzu/work/nds_stress_test/channels_VIS_2kHz_20.txt . There was no DAQ down
• 20 channels from LSC model (16 kHz sampling)
  • channel list: /users/yuzu/work/nds_stress_test/channels_LSC_16kHz_20.txt
  • Although there was no DAQ down, we could not read the data due to the low-level daq error. See fig. During reading the data, the medm screen shows the red alert on DAQ and DC (fig).
• 30 channels from ASCADS model (2 kHz sampling)
  • channel list: /users/yuzu/work/nds_stress_test/channels_ADS_2kHz_30.txt . There was no DAQ down.
• 30 channels from VIS model (2 kHz sampling)
  • channel list: /users/yuzu/work/nds_stress_test/channels_VIS_2kHz_30.txt . There was no DAQ down.
• 20 channels from ASC model (2 kHz sampling)
  • The ASCADC model was not problematic for reading the data. Next, we doubted whether the ASC model was problematic or not. channel list: /users/yuzu/work/nds_stress_test/channels_ASC_2kHz_20.txt
  • The DAQ down happened before the low-level daq error happened. See fig.YamaT-san quickly recovered the DAQ.
• 15 channels from ASC model (2 kHz sampling)
  • To investigate the acceptable number of channels, I reduced the number of channels (20 -> 15). channel list: /users/yuzu/work/nds_stress_test/channels_ASC_2kHz_15.txt
  • There was no DAQ down. We could read the data. See fig. But, the medm shows the red alert (fig). The situation continues for a while. YamaT-san rebooted the ASC computer to remove the red lamp.
• 12 channels from ASC model (2 kHz sampling)
  • channel list: /users/yuzu/work/nds_stress_test/channels_ASC_2kHz_15.txt and  /users/yuzu/work/nds_stress_test/channels_ASC_2kHz_15_1.txt
  • There was no DAQ down. We could read both the data.

Trial to avoid the DAQ issue

I developed the shell script to avoid the DAQ issue. The idea is the same as the 5th item in klog#30154. This script read the original PD list (pd_20240628.txt), split them into small chunks including 12 channels, and run the python script to read 12 channels.
After reading 12 channels, the python script will be finished and the shell script launch the python script with the next channel list. I added a time interval of 5 seconds (by `sleep` command) between launching the Python script. The shell script is `/users/yuzu/work/nds_stress_test/run_stress_test.sh`.
However, the DAQ issue happened again (fig), and the trial failed. We doubted that even though the script was finished by the interrupt of crtl+C, the process to the nds remains. That might be the reason for the remained red lamp after the process finished (which we saw above) and also the failure to read the third data set as reported in klog#30153.
To clean up the DAQ access manually, I plan to add the step to call `/opt/rtcds/userapps/release/cds/common/scripts/awgtpclear.sh`. I'm not sure if this works well or not. I will test it on the next maintenance day if possible.