Issue #1
The k1bcst0 transmits 1 second-long data every second. However, when an IPC glitch occurs, the transmission occurs at intervals slightly less than 1 second from the previous transmission (Approximately 0.000 to 0.004 seconds faster. However, this number is not an accurate representation of the actual time of change, as it is not known how quickly the Frame_Log used for testing can respond to the arrival of a new frame.) This data is input to the low latency calibration pipeline, and its output is transferred to the Kashiwa by Framelink and further transferred to the CIT by Kafka. The IPC glitched data arriving at CIT was delayed about 1 second more than normal frames. As a result, the gwistat displayed on KAGRA status switches from Observing to Down, and Duration is reset to 0, even though KAGRA keeps locked and observing state. It is not yet known at why the data is transmitted early in the first place and what stage the data transmitted early is slowed down (Negative delay@k1bcst0 --> positive delay@CIT).

Issue #2
In low latency data stream, there is a loss of one frame approximately every few 10k seconds, which may also be resetting Duration on gwistat to 0. It is not yet clear at what stage this missing is occurring.

Issue #3
KAGRA status is sometimes shown as Down on the gwistat page even though the KAGRA interferometer is in Observing Mode and the summary page in KAGRA shows it as Observing Mode. This phenomenon occurs with a frequency of about a few hours per day. It is not yet clear what is causing it yet.

did it, around 11:30 am, 2023-05-26.

An additional bandpass FIR filter will be applied to the output of the LL-CAL pipeline. This filter has the same response as described in document https://gwdoc.icrr.u-tokyo.ac.jp/DocDB/0122/G2012204/002/bandpass.pdf and suppresses below 10 Hz and above 6,000 Hz. The main motivation for doing this is to ensure that the BNS range are calculated and displayed correctly on various monitors and summary pages even without pre-filtering for h(t).

This update will be applied at some time when the interferometer is being unlocked.

This change will not affect interferometer control in any way. It also does not significantly affect the data analysis in the frequency band guaranteed by the CAL team. Making this change was approved by both RC and PM.

The figures showed the time series when the glitch occurred on the "K1:CAL-CS_PROC_DARM_STRAIN_DBL_DQ" channel.
The glitch data always reproduces the waveform one second earlier.

Fig. 1 shows a bit-by-bit view of K1:CAL-CS_PROC_DARM_ERR_DBL_DQ from the /data/full/13670/K-K1_C-1367000000-32.gwf, as an example.

A channel with DBL precision consists of 64 bits: 1 bit for the sign, 11 for the exponent part, and 52 for the fraction part. The lower bits of the fraction part are usually random 0 and 1, with a probability of 0.5 for a 1 to appear, but here we can see that some lower bits are always filled with 0.

Fig. 2 shows a plot of the relationship between the position of each bit and the ratio of bits that were 1 for some channels. The lower bits of some channels are apparently filled with zeros.

What could be the cause of this? Is it just a problem in output or is something happening upstream? All we know now is that if these outputs are used for something, the effect of reduced dynamic range, i.e., increased digital noise, is inevitable.

In most cases, IPC glitches occurred one sample before GPS seconds were divisible by 5.

We set these parameters to the low latency and restarted it on 17:22 JST. The filter file name is 2020-04-21_v2_FIRfilters.npz.

After updating the parameters of online, the low latency could output the correctly calibrated h(t).

We restarted low latency at 14:10 JST only with minor modifications for the parameters in the A residual FIR filter: anti- decimation filter(16k->4k) coefficients, from numerical form to the SOS representation, and the parameter of LPF, dropping from 1.8kHz (from 1.5kHz, previous).

After CAL measurements, we will restart the pipeline again with the updated parameters.

A summary of the modifications for low latency applied last night.

- We changed the calculation of C10 h(t) from err + (ctrlY - ctrlX) to err - (ctrlY + ctrlX). This is the same definition of signal signs as online C00 h(t).

- We found that the high latency aggregation did not match the main low latency. It seems that the divergence had begun at the time of the last parameter update, last week. We operated two low latency codes respectively for hoft_cal and hoft_dmg output, and the two config settings were slightly different. We integrated them into one. After this update, the consistency of both outputs, hoft_cal and hoft_dmg, is ensured.

- The update process of the low latency parameters had requires a lot of and very complicated manual works. It took a time consuming and easy to make mistakes. We refined the code and the way of update procedures.

- The function of rewinding the time stamp of the pipelines for half the time of the FIR filter length doesn't work well at present. Currently we are trying to debug it, but not completed yet. As a provisional solution, the filter lengths of 1/C and A were set to the same (2 seconds) to eliminate the time lag between both paths, except for tau_C and tau_A. This update eliminated the effect of phase interference of each path, however, the GPS offset of the h(t) signal by 1 second is still unresolved.


 

When entering the observation run several hours ago, various CAL parameters for h(t) reconstruction were updated and the low-latency pipeline was restarted. However, after checking the h(t) data in the observation run, we found a mistake in the update of the parameters. It made the h(t) to be biased.

Around 2:20 am today, the parameters were updated again, renew the FIR filters, and the low-latency pipeline was restarted. Please note that the low-latency data before and after the relevant time has been lost. I've also restarted the DMG script on cal-gst1 for the data transfers.

 

I restarted the low latency pipeline with the updated FIR filters for A and C. These parameters are the ones obtained from the CAL measurement this Tue and fitted by Yuki (#13305)

We restarted the low-latency just before starting the observation run. The config settings of the pipeline are CalibrationMode: partial2, (TM is assigned to EX_TM, on the other hand, IM to EY_TM), 

ComputeCalibStateVector: Yes, FiltersFileName: 2020-02-25_FIRfilters.npz, ComputeFactors: Yes, MedianSmoothingTime: 10, FactorsAveragingTime: 5, DefaultToMedian: Yes, ComputeCCResponse: No, 

ComputeSRCDetuning: No, ComputeKappaTM/IM: Yes, ComputeKappaMN: No, ApplyKappaTM/IM/MN: No. It's running with calculating partial calculations of kappa_EX/EY_TMs and without applying TDCs for h(t). The calibration-measurement results performed just before starting this run which were reported on #13163 and #13164 are not applied in this pipeline and parameters because of lack of time to switch them. It could be applied with TDCs in offline mode. 

We also restated the replica-pipeline for high latency h(t) aggregation after synchronized with the main pipeline.

I started to run a new low-latency pipeline in cal-gst1 around 10 pm on Feb. 22 (JST). It's just a replica of the main pipeline with running by my personal account (user: "sawada", not "cal"). It's running with parallel to the main one, and output to the new shared memory partition "hoft_dmg". This works successfully as the temporal solution to solve the problem seen in the high latency h(t) aggregation.

I stopped the low-latency pipeline's and dpushM's deamons around 16:22JST, deleted the shared memory partitions Online_cal and hoft_cal, and restarted the deamons around 16:25 JST, with the config options: ConfigVersion: k0.5, CalibrationMode: partial-1, ComputeCalibStateVector: Yes, FiltersFileName: 2020-02-13_FIRfilters.npz, ComputeCCResponse: No, ComputeSRCDetuning: No,  ComputeKappaTM/IM/MN: Yes/No/No, ApplyKappaTM/IM/MN: No/No/No,  SensFIRSR: 16384, ActFIRSR: 4096. The purpose of this restarting the pipeline, with refreshing the shared memory partitions, is to test the high latency h(t) aggregation in cal-gst1 server, and DMG data transfer.

I stopped the low-latency pipeline around 10:30 JST and restarted it around 10:35 JST for the test purpose.

I stopped the low-latency pipeline around 17:30JST and started the new version of the pipeline (git #74757971beb317b34464abf1cd1211a74776bdfc) around 17:50JST, with the config options: ConfigVersion: k0.5, CalibrationMode: partial-3, ComputeCalibStateVector: Yes, FiltersFileName: 2020-02-13_FIRfilters.npz, ComputeCCResponse: No, ComputeSRCDetuning: No,  ComputeKappaTM/IM/MN: Yes, ApplyKappaTM/IM/MN: No,  SensFIRSR: 16384, ActFIRSR: 4096.

The primary purpose of this switching the pipeline is to test the high latency h(t) aggregation in cal-gst1 server, DMG data transfer, and also the stability check of the renewed version of the pipeline.

for the PCAL measurement, we should also measure the "PCAL_EX_X_PCAL"  ;-(

We restarted low latency pipeline with updated FIR filters. It increased the latency slightly.

We measured the transfer functions.

Attachment#1 shows

Exc: VIS-ETMY_TM_LOCK_L_EXC
   - LSC-DARM1_IN1 / VIS-ETMY_TM_LOCK_L_OUT
   - LSC-DARM1_IN1 / VIS-ETMY_TM_LOCK_L_EXC
   - VIS-ETMY_TM_LOCK_IN1 / VIS-ETMY_TM_LOCK_IN2

   /users/CAL/sweptsine/sweptsine_20191216_tm_lock_l_etmy_meas5th.xml

Attachment#2 shows

Exc: PCAL_EX_0_INJ_V_EXC
   - LSC-DARM1_IN1 / CAL-PCAL_EX_1_PD_RX_V

   /users/CAL/sweptsine/sweptsine_20191216_xpcal_meas1st.xml

I found the unknown noises ~450Hz around GPS 1259962320. The attachment shows a time-frequency representation of the online strain channel (K1:CAL-CS_PROC_C00_STRAIN_DBL_DQ).

current settting for low latency is

dcs-calib_strain_c10 (16kHz)
=  cal-cs_proc_mich_residual_dbl_dq (16kHz) * aAA^-1 * dAA^-1
   + cal-cs_proc_mich_delta_ctrl_im_dbl_dq (4kHz) * aAI * dAI * (tau_A+tau_C)
   + cal-cs_proc_mich_delta_ctrl_tm_dbl_dq (4kHz) * aAI * dAI * (tau_A+tau_C)