We investigated the glitch issue (4072) today and traced it down to the DACs.
The glitches are actually generated from the DACs --- the output voltages go to zero for a few 100 msec and 1 sec occasionally.
Miyakawa-san and company will be swapping the IO chassis at some point soon to see if it eliminates the glitches.
- We brought an oscilloscope to the Y end.
We looked at the coil driver output for the IPs and saw that the requested DC voltage occasionally went to zero for a short period of time (100 msec - 1 sec).
- While looking at the IP drive voltage, we also intentionally applied a DC bias to the BF coil to see if the glitch happens for both IP coil signal and BF coil signals.
- The glitch seems to coincide between the IP and BF coils, according to this test.
- So we determined that this was a global phenomenon rather than an local one which can be in a particular coil channel only.
We looked at the 18 V power directly with the oscilloscope, but didn't find the glitches at all.
- So we think that the issue is not from the 18 V power supply.
- Then we looked at the DAC raw outputs by hacking an AI chassis which had a SCSI-DB9 converter inside.
- We found that the DAC raw outputs also had the glitches in DAC0 (which is the one controlling the ETMY tower).
We checked DAC1 while keeping DAC0 connected to the ETMY coils.
- We learned that the glitches happen in both DACs 0 and 1 simultaneously.
- So, again, this is an indication that this is some kind of a global phenomenon.
The attached picture shows an example of the glitches that we saw with the oscilloscope.
This is a glitch from DAC0 with the oscilloscope directly probing the raw DAC output. The rise time is typically very fast and is on the order of 100 usec which smells like a glitch from the digital system.