Reports 1-1 of 1 Clear search Modify search
VIS (General)
satoru.ikeda - 14:09 Friday 11 June 2021 (17079) Print this report
Organizing DAQ channels for VIS

TOWER_MASTER and PAYLOAD_MASTER have been changed.
The DAQ channels that have been changed are

[Tower].
K1:VIS-{optic}_{F0,F1,F2,F3,SF,BF}_LVDTINF_GAS_IN1 2048 -> 512
K1:VIS-{optic}_{F0,F1,F2,F3,SF,BF}_LVDTINF_GAS_OUT 256 (New)
K1:VIS-{optic}_{F0,F1,F2,F3,SF,BF}_DAMP_GAS_IN1 2048 -> 1024
K1:VIS-{optic}_{F0,F1,F2,F3,SF,BF}_DAMP_GAS_OUT 256 -> 256
K1:VIS-{optic}_{F0,F1,F2,F3,SF,BF}_MASTER_OUT_GAS 512 (New)
K1:VIS-{optic}_BF_DAMP_{L,T,Y,V,R,P,Y}_IN1* 2048 -> 1024 (Non Science Frame)
K1:VIS-{optic}_BF_LVDTINF_{H1,H2,H3,V1,V2,V3}_IN1 512 (New)
K1:VIS-{optic}_BF_LVDTINF_{H1,H2,H3,V1,V2,V3}_OUT 256 (New)
K1:VIS-{optic}_BF_MASTER_OUT_{V1,V2,V3,H1,H2,H3} 512 (New)

[Payload]
K1:VIS-{optic}_IM_DRIVEALIGN_{L,P,Y}_OUT* 2048 -> 2048 (Non Science Frame)
K1:VIS-{optic}_OSEMINF_{V1,V2,V3,H1,H2,H3}_IN1* 256 -> 1024 (Non Science Frame)
K1:VIS-{optic}_OSEMINF_{V1,V2,V3,H1,H2,H3}_OUT 256 (New)
K1:VIS-{optic}_OLDAMP_{L,P,Y}_IN1* 256 -> 1024 (Non Science Frame)
K1:VIS-{optic}_OLDAMP_{L,P,Y}_OUT 256 (New)
K1:VIS-{optic}_MASTERSWITCH_OUT_{V1,V2,V3,H1,H2,H3}* 2048 -> 2048 (None Science Frame)

optic:
 Update: PRM
 Next Update: ETMX, ETMY, ITMX, ITMY, BS, SRM, SR2, SR3, PR2, PR3, MCI, MCE, MCO, IMMT1, IMMT2, OMMT1, OMMT2, OSTM


During the installation of k1visprmpn, K1PRM, K1LSC0, K1ASC0, and K1SRM need to be restored.

[Problems]
When k1visprmp was updated, it hung at the restart of the model.

[Recovery procedure].
1. K1PRM recovery was performed.
 After disabling Dolphin in K1PRM, after a while, the FE PC hung with K1LSC0 in the way.
 I rebooted K1PRM from WebInterface.
2. Next, I tried to restore K1LSC0 as well.
 When I tried to disable Dolphin on k1lsc0, K1PRM was in the middle of rebooting and the Dolphin Disable process hung.
 When Yamamoto-san killed the process and ran it again after K1PRM started, it was able to disable Dolphin correctly.
 At this point, the Dolphin connection between K1ALS0 and K1SRM was broken.
 I restarted K1LSC0 from WebInterface.
 I had to restart K1LSC0 from the WebInterface twice because some of the reboots did not recognize Dolphin.
3. Next, I logged in to K1ALS0 via ssh and rebooted it via command (sudo shutdown -r now).
 However, Dolphin was not recognized in the same way, so I rebooted via WebInterface.
4. Next, I logged into K1SRM via ssh and rebooted via command.
 This time Dolphin was recognized successfully and the entire system was restored.
 The period when the system was down was approximately 10:45-12:15.
 

Non-image files attached to this report
Comments to this report:
satoru.ikeda - 18:19 Friday 11 June 2021 (17089) Print this report

Following PRM, I updated ETMX, ETMY, ITMX, ITMY (PAY only), BS, PR 2, PR3, SR2, SR3, and SRM.
DAQ Timing error (0x4000,0x5000) occurred in all models and 0xbad occurred after a while.
Therefore, all models including the PRM were returned to the state before the work of K-Log#17079.

mci,mce,mco,immt1,immt2,ommt1,ommt2,osm no diff but make with build confirmation and make install but
It is necessary to undo for the recovery, but I took over to miyo-san.
 

Non-image files attached to this comment
takahiro.yamamoto - 19:16 Friday 11 June 2021 (17090) Print this report
Both PMC and IMC were recovered.
Images attached to this comment
satoru.ikeda - 10:48 Monday 14 June 2021 (17102) Print this report

The following table summarizes the amount of change in DAQ Channels before and after the problem occurred.
The values are shown as the sum of the data rates.
In my experience, when the commissioning frame decreases, DAQ timing error occurs.

Increase/decrease of DAQ channel modified by TOWER_MASTER
commissioning frame : -11,008
science frame : -12,288 

Increase/decrease of DAQ channel modified by PAYLOAD_MASTER
commissioning frame : +15,360
science frame: -22,272
 

Non-image files attached to this comment
satoru.ikeda - 13:25 Tuesday 15 June 2021 (17120) Print this report

Survey of DAQ Channels at the time of o3gk
I can't do everything, so I investigated prm and srm as examples.
Values are shown as the sum of data rate.

o3gk:
 PRM(Tower+Payload): 34,816
 SRMT: 24,576
 SRP: 43.008

Calculated again with the total data rate of all DAQ channels in K-Log#17079 for comparison:
Tower:
 Before change (OK): 70,912
 After change (NG): 26,980
Payload:
 Before change(OK): 48,640
 After change(NG): 64,000

The results of the verification on 2021/06/05 were also added.
 TOWER_MASTER_origin(OK): 70.912
 TOWER_MASTER_ng(NG): 26,980
 TOWER_MASTER_neworigin(NG): 50,944
 TOWER_MASTER_Only2K (OK): 69,376

From the above data, it seems that the DAQ Timing Error occurs when the total amount of data crosses the 64K boundary.
 

Non-image files attached to this comment
satoru.ikeda - 16:59 Friday 18 June 2021 (17163) Print this report

I investigated the DAQ error in K-Log#17079.

1. Isolate whether the problem is with PAYLOAD or TOWER.
 Installed again using the file that caused the DAQ error in K-Log#17079.
 Update PAYLOAD_MASTER
 optic: ETMX, ETMY, ITMX, ITMY, BS, SRM, SR2, SR3, PRM, PR2, PR3
 => Updated without problems.

 GitID: release/0.1.9
 GpsTime: 1308021324

ETMXP's CPU_METER_MAX is now a bit smaller, 38->37.
=> Not sure how this affects it. However, this seems to be a specification since the time fluctuates every time I reboot DAQ. s

I used this as a save point to do the following checks.

2.Next, without updating the TOWER model, I overwrote the ini file of DAQ at the time of the error occurrence with the TOWER model and checked whether the problem occurred or not.
=> No problem occurred when updating PRMT and PR2T, but DAQ error occurred when updating PR3T was finished.
The frequency of occurrence of the timing error is every 1-2 minutes.

3.I commented out PRMT, PR2T and PR3T from master and restarted DAQ with DAQ error.
=> The error situation did not change.
 I removed the problematic ini file from the master file, but the error status did not change, so it is not a DAQ problem.

4. Returned the master and ini files to the state after 1 above. 

5.From the above 3, I found out that the problem was occurring on the FrontEnd side.
 From the above 3, I guessed that the problem occurred on the FrontEnd side, because the TOWER model is 2K and the DAQ output rate is different from 256, 512, 1024.
 I set all the rates of the channels I changed in the ini file of the DAQ where the problem occurred to the following values for comparison.
  All 2K: waited 10+ minutes, no error
  All 1K: 1 DAQ timingerror after 12 minutes
  All 512: timing error occurs every 1-2 minutes. This was the same interval as 2 above.
  All 256: 2 timing errors occur every 7-9 minutes

Finally, I reverted to the TOWER ini file as of 1 above.

What I found out from the above.
DAQ Timing error occurs on the FrontEnd side, DAQ is not affected.
Payload (16K) model has no problem.
  No problem with Payload(16K) model, problem occurs with Tower(2k) model.
DAQ Timing error occurs when using DAQ with PRM, PR2, PR3 other than 2K.
  DAQ timing error occurs when using DAQ with PRM, PR2 and PR3 other than 2K, but the frequency is high when the data rate is 512.
The combination other than PRM, PR2, and PR3 has not been confirmed today. (Work time is up)

Images attached to this comment
Non-image files attached to this comment
satoru.ikeda - 17:26 Monday 28 June 2021 (17308) Print this report

Added DAQ channel to the following models.

1. Addition and change of DAQ channel to MN of PAYLOAD_MASTER
K1:VIS-{optic}_MN_DRIVEALIGN_{L,P,Y}_OUT 2048
K1:VIS-{optic}_MN_OSEMINF_{V1,V2,V3,H1,H2,H3}_IN1 1024
K1:VIS-{optic}_MN_OSEMINF_{V1,V2,V3,H1,H2,H3}_OUT 256

K1:VIS-{optic}_MN_OLDAMP_{L,P,Y}_IN1 1024
K1:VIS-{optic}_MN_OLDAMP_{L,P,Y}_OUT 256
K1:VIS-{optic}_MN_DAMP_{L,T,V,R,P,Y}_IN1* 1024 -> 1024 {no science frame}
K1:VIS-{optic}_MN_DAMP_{L,T,V,R,P,Y}_OUT 256
K1:VIS-{optic}_MN_MASTER_OUT_{V1,V2,V3,H1,H2,H3} 2048

optic: ITMX, ITMY, ETMX, ETMY

Git:Release/0.1.11

2. Added and changed DAQ channel to BF of TOWER_MASTER

DAMP_GAS_IN1 2048 -> 1024
DAMP_{L,T,V,R,P,Y}_IN1* 2048 -> 1024 {no science frame}
DAMP_GAS_OUT 256 => no change
DAMP_{L,T,V,R,P,Y}_OUT 256 => no change
LVDTINF_GAS_IN1 2048 -> 512
LVDTINF_{H1,H2,H3,V1,V2,V3}_IN1 512
LVDTINF_GAS_OUT 256
LVDTINF_{H1,H2,H3,V1,V2,V3}_OUT 256
MASTER_OUT_{V1,V2,V3,H1,H2,H3,GAS} 512

optic: PRM, PR2, PR3, ETMX, ETMY, ITMX, ITMY, BS, SR2, SR3
SRM is not reflected because it is an independent environment, not a common file.

Git:Release/0.1.12

3.DAQ Timing Error was investigated.

3-1. First, I tried the ini file that caused the error on 6/18.
This is the difference between the above 2 and only the change of DAQ channel of GAS.
[Change]
K1:VIS-{optic}_{F0,F1,F2,F3,SF}_DAMP_GAS_IN1_DQ 2048 => 1024
K1:VIS-{optic}_{F0,F1,F2,F3,SF}_LVDTINF_GAS_IN1_DQ 2048 => 512
[Add].
K1:VIS-{optic}_{F0,F1,F2,F3,SF}_LVDTINF_GAS_OUT_DQ 256
K1: VIS-{optic}_{F0,F1,F2,F3,SF}_MASTER_OUT_GAS_DQ 512
optic: PRM, PR2, PR3
=> I changed PR2T, PR3T, PR3T and got DAQ timing error as before.

3-2. Next, check whether the problem occurred by adding or changing
 File with [Add] only
  => TimingError is not generated.
 The file which only [Change] was done
  => TimingError occurred.
  The file which changed only LVDTINF_GAS_IN1_DQ of [Change] 
   => TimingError occurred.
  The file which changed only DAMP_GAS_IN1_DQ of [Change] 
   => TimingError occurred.
   File with all DAQ channels removed in [Change].
   => TimingError was raised, although the frequency is very low.

  [under /users/ikeda/20210628/ folder].
  Science frames were irrelevant.
  The problem occurs in the direction of reducing the data rate of DAQ channels in GAS

4. I checked the dmesg of each FE.
 It seems that the following logs keep appearing in k1pr2, k1pr3, and k1sr2.
 The same was also true for k1asc0, k1ioo1, and k1imc0.
 I don't know if this is related to the glitch.
 [xxx.xx] (get_path_table_index:580):Cannot handle more than 3 hops
 [xxx.xx] (dx_seg_pe_setup:922):unable to get path table index

 Does not occur in k1prm,k1bs,k1sr3,k1omc0,k1ix1,k1iy1,k1ex1,k1ey1.
 k1lsc0,k1als0,k1ioo

Images attached to this comment
Non-image files attached to this comment
satoru.ikeda - 17:03 Tuesday 29 June 2021 (17329) Print this report

Lowering the data rate of the channel currently registered in the DAQ channel of GAS below 2K or deleting the channel raises a TimingError.
However, as the VIS work requires the addition of DAQ channels, the channel addition work was done first.

[Add]
K1:VIS-{optic}_{F0,F1,F2,F3,SF}_LVDTINF_GAS_OUT 256
K1:VIS-{optic}_{F0,F1,F2,F3,SF}_MASTER_OUT_GAS 512

optic: PRM,PR2,PR3,ETMX,ETMY,ITMX,ITMY,BS,SR2,SR3
(SRM is excluded because it doesn't use common files)

GitID: release/0.1.14

With the above action, the concern about the progress of VIS is gone.
However, it does not solve the root problem, and we will be working on the two channels that do not need 2K DAQ DataRate to see if they can be changed.
 

Non-image files attached to this comment
satoru.ikeda - 18:05 Friday 09 July 2021 (17460) Print this report

Since the DAQ channel was added in K-Log#17368, I tried to see if I could reduce the data rate of the DAQ using the ini file that caused the error that I checked in K-Log#17308.
Here is the next diff file.
[Change]
K1:VIS-{optic}_{F0,F1,F2,F3,SF}_DAMP_GAS_IN1_DQ 2048 => 1024
K1:VIS-{optic}_{F0,F1,F2,F3,SF}_LVDTINF_GAS_IN1_DQ 2048 => 512

The above changes to the ini file caused a DAQ TimingError.
Therefore it cannot be reduced even after adding DAQ.
 

Non-image files attached to this comment
Search Help
×

Warning

×