Reports 1-1 of 1 Clear search Modify search
kiwamu.izumi - 17:36 Thursday 22 February 2018 (4254) Print this report
The glitches are from the DACs

Yutaro, Kiwamu,

We investigated the glitch issue (4072) today and traced it down to the DACs.

The glitches are actually generated from the DACs --- the output voltages go to zero for a few 100 msec and 1 sec occasionally.

Miyakawa-san and company will be swapping the IO chassis at some point soon to see if it eliminates the glitches.

[The investigations]

  • We brought an oscilloscope to the Y end.
  • We looked at the coil driver output for the IPs and saw that the requested DC voltage occasionally went to zero for a short period of time (100 msec - 1 sec).
    • While looking at the IP drive voltage, we also intentionally applied a DC bias to the BF coil to see if the glitch happens for both IP coil signal and BF coil signals.
    • The glitch seems to coincide between the IP and BF coils, according to this test.
    • So we determined that this was a global phenomenon rather than an local one which can be in a particular coil channel only.
  • We looked at the 18 V power directly with the oscilloscope, but didn't find the glitches at all.
    • So we think that the issue is not from the 18 V power supply.
  • Then we looked at the DAC raw outputs by hacking an AI chassis which had a SCSI-DB9 converter inside.
  • We found that the DAC raw outputs also had the glitches in DAC0 (which is the one controlling the ETMY tower).
  • We checked DAC1 while keeping DAC0 connected to the ETMY coils.
    • We learned that the glitches happen in both DACs 0 and 1 simultaneously.
    • So, again, this is an indication that this is some kind of a global phenomenon.


The attached picture shows an example of the glitches that we saw with the oscilloscope.

This is a glitch from DAC0 with the oscilloscope directly probing the raw DAC output. The rise time is typically very fast and is on the order of 100 usec which smells like a glitch from the digital system.

Images attached to this report
Comments to this report:
yutaro.enomoto - 10:42 Friday 23 February 2018 (4261) Print this report

== (not perfect) Coincidence between sudden increase of CPU load and ETMY glitch ==

[Miyakawa, Yamamoto, Enomoto]

As Miyakawa-san suggested, we looked at CPU meter and ETMY oplev output.
Please find the attached figure.
At first glance it seemed they show perfect coherence, but, looking carefully, you can find a few examples where only the glitch happened but CPU increase did not, and vice versa.
Need to continue investigating.

Images attached to this comment
kiwamu.izumi - 15:53 Sunday 25 February 2018 (4275) Print this report

I wonder why this occurs in the ETMY suspension controller only.

The number of digital filters we use in this particular system is not exactly a lot - we probably have handful of the filter modules activeuly running on a daily basis. Very curious to hear the results from the test of reducing the samping rate.

yutaro.enomoto - 8:13 Tuesday 27 February 2018 (4289) Print this report

[Ushiba, Enomoto]

No glitches found last night, which used to come from the DACs of ETMYyes

The sampling frequency of ETMY model was changed (4269), and we left ~ 1000 cnts DC offsets applied to the MN stage before we leave the office last night.
There were no glitches last night (see the attached), except for the one probably due to an earthquake at Papua New Guinea.
Unfortunately some fraction of data is lacking on Data Viewer, but we can say that the rate of the glitch dropped by far.

We still don't know why, but decreasing samplig frequency to 2 kHz works for this issue.

Images attached to this comment
Search Help