Ikeda, アドバイス:Yamamoto-san, Oshino-san DAQ TimingError(0x4000),0xbadが発生する不具合についての対処です。 K-Log#20435,19393,18668,17079が関連します。 概要 ・DAQのログを確認していたところ、いくつかのエラーが発生していた。 $ cat /opt/rtcds/kamioka/k1/target/fb/logs/daqd.dc0.log 一部抜粋 [Thu Apr 21 19:39:04 2022] mx receiver 001 thread priority error Operation not permitted [Thu Apr 21 19:39:04 2022] mx receiver 004 thread priority error Operation not permitted [Thu Apr 21 19:39:04 2022] mx receiver 007 thread priority error Operation not permitted mx receiver 00a thread priority error Operation not permitted 調査 ・エラーの原因を調査したところ、LinuxのスケジュールポリシーがRCGの意図した値を受け付けない為、エラーが発生していることが分かった。 $ ulimit -aのログを確認したところreal-timeが0になっていた。 またdc0は起動時のdaqdの優先度を変更している行が抜けていた。 対応 ・FIFO設定が処理されるように設定を変更した。 dc0,nds0,nds1,nds2,fw0,fw1,tw0,tw1,bcst0 /etc/security/limits.confを編集して再起動 controls - rtprio 99 controls - nice -20 各サーバーにログインして下記を実施 $ sudo emacs inittab daqをコメントアウトする。 $ sudo kill -HUP 1 daq停止 $ sudo emacs inittab daq戻す $ sudo kill -HUP 1 DAQ再起動させる nds2はkillではなく sudo systemctl restart k1nds2daqd.service sudo systemctl restart k1nds2nds.service ・k1dc0についてはdaqdのnice値の変更も必要なので次の変更も行なった。 start_daqd.dc0.inittabの書き換えてこちらも上記と同じように再起動 $ sudo emacs /opt/rtcds/kamioka/k1/target/fb/start_daqd.dc0.inittab # Execute frame builder exec su controls -c '/opt/rtcds/kamioka/k1/target/k1daqdc0/daqd -c /opt/rtcds/kamioka/k1/target/k1daqdc0/daqdrc' > /opt/rtcds/kamioka/k1/target/fb/logs/daqd.dc0.log 2>& 1 & PID=$! sleep 1 LISTPID=$(pidof daqd); PIDS=(${LISTPID// / }); for val in "${PIDS[@]}"; do sudo chrt -f -p 99 ${val}; done wait $PID 確認結果 ・dc0,nds0,nds1,fw0,fw1,tw0,tw1,bcst0は正常に変更されてFIFOになっていることを確認。 (参考)k1dc0の場合、DAQ関連のスレッドがFFであること、並びにdaqdのRTPRIOは99であることを確認した。 $ ps x -L -o comm,pid,cls,rtprio,priority,nice,user COMMAND PID CLS RTPRIO PRI NI USER sshd 629 TS - 20 0 controls bash 630 TS - 20 0 controls su 2947 TS - 20 0 controls daqd 2949 FF 99 -100 - controls dqpmain 2949 TS - 0 -20 controls dqxmit 2949 TS - 0 -20 controls dqxmit 2949 TS - 0 -20 controls daqd 2949 TS - 0 -20 controls dqprod 2949 FF 5 -6 - controls dqmx000 2949 FF 10 -11 - controls dqmx001 2949 FF 10 -11 - controls dqmx002 2949 FF 10 -11 - controls dqmx003 2949 FF 10 -11 - controls dqmx004 2949 FF 10 -11 - controls dqmx005 2949 FF 10 -11 - controls dqmx006 2949 FF 10 -11 - controls dqmx007 2949 FF 10 -11 - controls dqmx008 2949 FF 10 -11 - controls dqmx009 2949 FF 10 -11 - controls dqmx00a 2949 FF 10 -11 - controls dqmx00b 2949 FF 10 -11 - controls dqmx00c 2949 FF 10 -11 - controls dqmx00d 2949 FF 10 -11 - controls dqmx00e 2949 FF 10 -11 - controls dqmx00f 2949 FF 10 -11 - controls dqmx001 2949 FF 10 -11 - controls dqmx001 2949 FF 10 -11 - controls dqmx004 2949 FF 10 -11 - controls dqmx004 2949 FF 10 -11 - controls dqmx007 2949 FF 10 -11 - controls dqmx007 2949 FF 10 -11 - controls dqmx00a 2949 FF 10 -11 - controls dqmx00a 2949 FF 10 -11 - controls dqmx00d 2949 FF 10 -11 - controls dqmx00d 2949 FF 10 -11 - controls dqmx000 2949 FF 10 -11 - controls dqmx000 2949 FF 10 -11 - controls dqmx005 2949 FF 10 -11 - controls dqmx005 2949 FF 10 -11 - controls dqmx009 2949 FF 10 -11 - controls dqmx009 2949 FF 10 -11 - controls dqmx00e 2949 FF 10 -11 - controls dqmx00e 2949 FF 10 -11 - controls dqmx003 2949 FF 10 -11 - controls dqmx003 2949 FF 10 -11 - controls dqmx00b 2949 FF 10 -11 - controls dqmx00b 2949 FF 10 -11 - controls dqmx002 2949 FF 10 -11 - controls dqmx002 2949 FF 10 -11 - controls dqmx00c 2949 FF 10 -11 - controls dqmx00c 2949 FF 10 -11 - controls dqmx008 2949 FF 10 -11 - controls dqmx008 2949 FF 10 -11 - controls dqmx00f 2949 FF 10 -11 - controls dqmx00f 2949 FF 10 -11 - controls dqmx006 2949 FF 10 -11 - controls dqmx006 2949 FF 10 -11 - controls dqedcu 2949 TS - 0 -20 controls dqedcu 2949 TS - 0 -20 controls dqedcu 2949 TS - 0 -20 controls dqepics 2949 TS - 0 -20 controls dql8087 2949 TS - 0 -20 controls dqedcu 2949 TS - 0 -20 controls dqedcu 2949 TS - 0 -20 controls dqedcu 2949 TS - 0 -20 controls dqedcu 2949 TS - 0 -20 controls dqedcu 2949 TS - 0 -20 controls dqedcu 2949 TS - 0 -20 controls dql8088 2949 TS - 0 -20 controls dqedcu 2949 TS - 0 -20 controls 以降省略 ・nds2は設定が反映されなかった。 　原因は不明。 --- Ikeda, advice:Yamamoto-san, Oshino-san This is to address the issue of DAQ TimingError(0x4000),0xbad. K-Log#20435,19393,18668,17079 are relevant. Summary I was checking the DAQ logs and found some errors. $ cat /opt/rtcds/kamioka/k1/target/fb/logs/daqd.dc0.log Some excerpts [Thu Apr 21 19:39:04 2022] mx receiver 001 thread priority error Operation not permitted [Thu Apr 21 19:39:04 2022] mx receiver 004 thread priority error Operation not permitted [Thu Apr 21 19:39:04 2022] mx receiver 007 thread priority error Operation not permitted mx receiver 00a thread priority error Operation not permitted Investigation After investigating the cause of the error, it was found that the error occurred because the Linux scheduling policy did not accept the intended value of RCG. Checking the $ ulimit -a log, we found that real-time was set to 0. Also, dc0 was missing a line that changes the priority of daqd at startup. Actions taken The configuration was changed so that the FIFO settings are processed. dc0,nds0,nds1,nds2,fw0,fw1,tw0,tw1,bcst0 Edit /etc/security/limits.conf and restart controls - rtprio 99 controls - nice -20 Log in to each server and do the following $ sudo emacs inittab Comment out daq. $ sudo kill -HUP 1 Stop daq $ sudo emacs inittab Revert daq $ sudo kill -HUP 1 Restart DAQ nds2 is not a kill. sudo systemctl restart k1nds2daqd.service sudo systemctl restart k1nds2nds.service For k1dc0, the following changes were also made because it is necessary to change the nice value of daqd. Rewrite start_daqd.dc0.inittab and restart as above $ sudo emacs /opt/rtcds/kamioka/k1/target/fb/start_daqd.dc0.inittab # Execute frame builder exec su controls -c '/opt/rtcds/kamioka/k1/target/k1daqdc0/daqd -c /opt/rtcds/kamioka/k1/target/k1daqdc0/daqdrc' > /opt/rtcds/kamioka/k1/target/fb/logs/daqd.dc0.log 2>& 1 & PID=$! sleep 1 LISTPID=$(pidof daqd); PIDS=(${LISTPID// / }); for val in "${PIDS[@]}"; do sudo chrt -f -p 99 ${val}; done wait $PID Confirmation Result Confirm that dc0,nds0,nds1,fw0,fw1,tw0,tw1,bcst0 have been changed normally and FIFOed. (Reference) In the case of k1dc0, it is confirmed that DAQ related threads are FF and RTPRIO of daqd is 99. $ ps x -L -o comm,pid,cls,rtprio,priority,nice,user COMMAND PID CLS RTPRIO PRI NI USER sshd 629 TS - 20 0 controls bash 630 TS - 20 0 controls su 2947 TS - 20 0 controls daqd 2949 FF 99 -100 - controls dqpmain 2949 TS - 0 -20 controls dqxmit 2949 TS - 0 -20 controls dqxmit 2949 TS - 0 -20 controls daqd 2949 TS - 0 -20 controls dqprod 2949 FF 5 -6 - controls dqmx000 2949 FF 10 -11 - controls dqmx001 2949 FF 10 -11 - controls dqmx002 2949 FF 10 -11 - controls dqmx003 2949 FF 10 -11 - controls dqmx004 2949 FF 10 -11 - controls dqmx005 2949 FF 10 -11 - controls dqmx006 2949 FF 10 -11 - controls dqmx007 2949 FF 10 -11 - controls dqmx008 2949 FF 10 -11 - controls dqmx009 2949 FF 10 -11 - controls dqmx00a 2949 FF 10 -11 - controls dqmx00b 2949 FF 10 -11 - controls dqmx00c 2949 FF 10 -11 - controls dqmx00d 2949 FF 10 -11 - controls dqmx00e 2949 FF 10 -11 - controls dqmx00f 2949 FF 10 -11 - controls dqmx001 2949 FF 10 -11 - controls dqmx001 2949 FF 10 -11 - controls dqmx004 2949 FF 10 -11 - controls dqmx004 2949 FF 10 -11 - controls dqmx007 2949 FF 10 -11 - controls dqmx007 2949 FF 10 -11 - controls dqmx00a 2949 FF 10 -11 - controls dqmx00a 2949 FF 10 -11 - controls dqmx00d 2949 FF 10 -11 - controls dqmx00d 2949 FF 10 -11 - controls dqmx000 2949 FF 10 -11 - controls dqmx000 2949 FF 10 -11 - controls dqmx005 2949 FF 10 -11 - controls dqmx005 2949 FF 10 -11 - controls dqmx009 2949 FF 10 -11 - controls dqmx009 2949 FF 10 -11 - controls dqmx00e 2949 FF 10 -11 - controls dqmx00e 2949 FF 10 -11 - controls dqmx003 2949 FF 10 -11 - controls dqmx003 2949 FF 10 -11 - controls dqmx00b 2949 FF 10 -11 - controls dqmx00b 2949 FF 10 -11 - controls dqmx002 2949 FF 10 -11 - controls dqmx002 2949 FF 10 -11 - controls dqmx00c 2949 FF 10 -11 - controls dqmx00c 2949 FF 10 -11 - controls dqmx008 2949 FF 10 -11 - controls dqmx008 2949 FF 10 -11 - controls dqmx00f 2949 FF 10 -11 - controls dqmx00f 2949 FF 10 -11 - controls dqmx006 2949 FF 10 -11 - controls dqmx006 2949 FF 10 -11 - controls dqedcu 2949 TS - 0 -20 controls dqedcu 2949 TS - 0 -20 controls dqedcu 2949 TS - 0 -20 controls dqepics 2949 TS - 0 -20 controls dql8087 2949 TS - 0 -20 controls dqedcu 2949 TS - 0 -20 controls dqedcu 2949 TS - 0 -20 controls dqedcu 2949 TS - 0 -20 controls dqedcu 2949 TS - 0 -20 controls dqedcu 2949 TS - 0 -20 controls dqedcu 2949 TS - 0 -20 controls dql8088 2949 TS - 0 -20 controls dqedcu 2949 TS - 0 -20 controls The following is omitted. The setting of $ ps x -L -o comm,pid,cls,rtprio,priority,nice,user was not reflected. 　The cause is unknown.