Reports 1-1 of 1 Clear search Modify search
VIS (IX)
takafumi.ushiba - 20:22 Tuesday 14 December 2021 (19212) Print this report
ITMX F3 GAS signals became glitchy from yesterday

I unfortunately (fortunately?) found that F3 GAS filter signals became glitchy from about 14:18 JST yesterday.
Figure 1 shows the time series data of GAS filter signals.
As you can see, F3 GAS signals became glitchy after t=0 (about 14:18 JST yesterday).
Moreover, current F3 signals are much glitchy compared to the yesterday signals (fig2).

Around this time, IXV vacuum leak test of new pump unit was performed.
So, I suspected the electrical coupling due to the new pump units but Kimura-san said the unit hasn't been connected electrically yet.

What happens?

Images attached to this report
Comments to this report:
hyungwon.lee - 4:18 Wednesday 15 December 2021 (19219) Print this report
Are any indirect coupling via power lines or some power lackness?
That is, operating new pump will consumes more power that expected casuing some VCC value instability?
nobuhiro.kimura - 5:46 Wednesday 15 December 2021 (19222) Print this report
As you can see attached photo.
New vacuum pomp system is not connected electrical power source yet.
Only connected with IXV vacuum chamber mechanically.
Images attached to this comment
shinji.miyoki - 8:12 Wednesday 15 December 2021 (19225) Print this report

One of what we can check from ouside is maybe the cable reconnection for this signals.

Images attached to this comment
shinji.miyoki - 20:43 Wednesday 15 December 2021 (19239) Print this report

Around 20:30 on 15th Dec, there seems to be no glitches in F3 although the vertical axis values are different with those of Ushiba-kun's data.

Images attached to this comment
ryutaro.takahashi - 22:14 Wednesday 15 December 2021 (19242) Print this report

[Takahashi, Sato]

We went to investigate the glitch in the F3 signals. We could not identify the cause of the glitch, because there were no frequent glitches today.

  • Shaked the cables and the connectors between the feedthrough and the LVDT distributor. Any linked signals were not found.
  • Touched (push/pull) the connector on the rear panel of the LVDT distributor. A large jump of the signal was found. After the tight screws, it did not appear.
  • Touched (beat) the LVDT driver chassis. Large glitches were found not only in the F3 signal but also in other all GAS filter signals.
  • Reconnected all cables between the feedthrough and the LVDT distributor. There were still a few glitches per 10min., but not so many.
  • Checked the resistance of the coils. There were not any changes and groundings.
Images attached to this comment
shinji.miyoki - 8:19 Thursday 16 December 2021 (19244) Print this report

I found glitches around 8 am. And it continues for 30 minutes at least.

Not only short (~ 100 useconds or less) spikes, but also longer (~ seconds) DC level shift rarely happened.

Images attached to this comment
shinji.miyoki - 14:23 Thursday 16 December 2021 (19247) Print this report

I checked the influence of LED lights on the tunnel ceiling, LED lights in the clean booth, all (10) FFUs, k1ctr14, and the air cooler. However, there is no change in the glitchy signal in F3.

takafumi.ushiba - 11:19 Monday 20 December 2021 (19275) Print this report

Abstract:
I performed a cable swapping test for investigating the problem.
As a result, the problem doesn't seem to be inside the vacuum chamber, and we can probably proceed the evacuation without solving this problem.

Detail:
I did the cable swapping test for investigating if the source of the glitches is inside the vacuum chamber or not.
I disconnected the F2 and F3 LVDT cables at the feedthrough port and swap them.
Swapping work was started from 10:43 JST and finished at 10:51 JST.

Figure 1 shows the LVDT signals when F2 and F3 cables were swapped.
As you can see, glitches are still on the F3 signals even though the cables were swapped.
This implies that the origin of the glitches are outside of the vacuum chamber.

I also checked the spectra before and after swapping the cables (fig2, red: after swapping, blue: before swapping).
As you can see, F3 signals became better when cables were swapped while F2 signals became worse.
Note that F2 and F3 signals after swapping (red) can be seen at F3 and F2 channels, respectively, because of the swapping.
So, the electrical circuits are doubtful.

Note:
Measured spectra were stored at /users/VISsvn/TypeAtower/ITMX/Measurements/20211220/SPE_ITMX_GAS_swappingtest_20211220.xml.

Images attached to this comment
ryutaro.takahashi - 18:14 Friday 24 December 2021 (19321) Print this report

[Takahashi, Sato]

We confirmed the location of the noise source swapping the F2 and F3 cables not at the feedthrough but at the front of the LVDT distributor. In both cases, the glitch appeared in the F3 channel on the DGS. This means that the problem is not in the vacuum side but in the driver side.

Images attached to this comment
ryutaro.takahashi - 12:44 Friday 07 January 2022 (19384) Print this report

[Takahashi, Sato]

The rate of the glitch in the F3 was small today, a few times per 30 min. We replaced the LVDT driver (main card, in/out card, SG) completely. We could see still glitches at the same rate. We disconnected the cable from the output of the LVDT driver. We could see still glitchy signals in the DGS output. This means the problem is on the DGS side.

Images attached to this comment
takahiro.yamamoto - 23:30 Wednesday 12 January 2022 (19424) Print this report
[Oshino, YamaT]

Abstract
According to the last report by Takahashi-san (klog#19384), there seems to be causes of glitches at downstream of LVDT output. So we searched around digital system.
We was not able to identify the cause. But it doesn't seem to be just a single circuit. It seems to be related to the GND around IXV racks.

Details
As the Takahashi-san's report, glitches were still presence even when he unplugged the D-sub cable between LVDT and AA at the LVDT side.
At first we reproduced this situation and glitches really remained.

After then, in order to narrow down the cause of glitches, we changed the unplugged point and checked glitches were presence or absence as follows.
1. Dsub cable between LVDT and AA at the AA side.
2. Same cabling situation, but turned off AA power supply.
3. SCSI cable between AA and ADC at the AA side.

The glitches were presence in case 1 and 2, and they were absence in case 3.
So we doubted the AA chassis at first and replaced it as a spare one.
However glitches didn't disappear by replacing the AA chassis.

Next, because the GND nightmare crossed my mind, I checked glitch behavior in following situations.
1. Only a SCSI cable is connected between AA and ADC with putting AA on the floor.
2. Only a SCSI cable is connected between AA and ADC with mounting AA on the IXV1 rack.
3. A SCSI cable between AA and ADC and a power cable are connected with putting AA on the floor.
4. A SCSI cable between AA and ADC and a DB9 cable between LVDT and AA are connected with putting AA on the floor.

Only in case 1, glitches were absence.
Some kind of path to connect GND of DC18V to AA seems to affect the glitch presence.
(In case 2, GND of DC18V is connected to AA chassis via the screws for routing a 19-inch rack.)

I didn't have enough time to do a detailed investigation, but during these investigation I noticed that glitches appear just by touching the AA chassis via tester pins.
I will continue this work on Friday.

Finally, everything was restored to the original state and I confirmed ALIGNED state is available.
takahiro.yamamoto - 2:09 Friday 14 January 2022 (19452) Print this report
[Ushiba, YamaT]

For investigating GND issue around IXV racks, we prepared a new AA chassis (JGW-S2213729)
Some jumper pins were removed in order to avoid local GND loops.
We will try to replace AA chassis and check the state of F3 GAS glitches tomorrow.

-----
JP1, JP2 (both 1-2 and 3-4), JP3 and JP6 (both 1-2 and 3-4) were removed for all 32 channels.
See also an attachment and JGW-D1909586
Images attached to this comment
takahiro.yamamoto - 9:44 Monday 17 January 2022 (19482) Print this report
This is Friday's work.

We replaced AA chassis as a modified one but glitches still existed.
In this modification, GND of DC18V is still connected to chassis body.
We may need to consider to the GND of DC power supply.

Finally, we backed to the original state and modified AA chassis was stored in computer room in the mine.
takahiro.yamamoto - 20:09 Monday 02 May 2022 (20639) Print this report

[Ushiba, YamaT]
 

Abstract

- Similar issue on ITMY ACC_H2 was reported as klog#20636.
- Glitches on F3 GAS were removed by replacing the ADC board, the ADC adapter board, and the ADC internal cable.
- Glitch problem was solved.

- On the other hand, ADC timing error is occurred with new ADC.
- It shouldn't affect ITMY local measurement.
- Timing error probably affects measurements with channels on other real-time computer.
- This will be discussed in klog#20642.
 

Details

Differences from klog#20636
Replaced info.
- ADC: S1402718 -> S2113340.

Time series of ADC noise on GAS channels are shown in Fig.1.
Behavior of F3_GAS was drastically improved and noise level of other channels were also slightly imporved.

According to spectra show in Fig.2, noise level became roughly two times smaller than ones before replacement.
(6e-3 ct/rHz => 3e-3 ct/rHz above ~50Hz)

Images attached to this comment
Search Help
×

Warning

×