I checked the behavior of the OSEM gains when the Type-B/Bp chambers were evacuated in June, 2024. The gains of some OSEMs in any IMs (PRM, PR2, PR3, BS, SRM, SR2, SR3) were reduced in vacuum.
Takano, Fujimoto, Tanaka, Dan
Summary
We performed the initial alignment of the IFO by following the procedure in klog36759. After that, we tried to find the SRC flash by shaking SRM and moving SRM IP in yaw direction. Finally, we observed clear SRC flashes.
What we did
Initial alignment of IFO
First, we performed the initial alignment of the IFO. Since the transmissivity of SRM was changed, the usual initial alignment procedure cannot be used for the AS side alignment. Therefore, we adjusted the alignment by referring to klog36759.
For X-arm and Y-arm, we used the usual initial alignment procedure. After that, we continued the alignment according to the procedure described in klog36759. As written in the klog, we moved OMMT1 while monitoring OMMT1_TRAS_QPD1. When the beam position on OMMT1_TRAS_QPD1 became almost centered, we regarded the AS side alignment as good.
Search for SRC flash
Next, we tried to find the SRC flash by adjusting the alignment of SRM. We set only ITMY to LOCK_ACQUISITION and misaligned the other Type-A suspensions. We also restored the SRM IM OPTICALIGN pitch value to yesterday’s value of 3825. In this condition, we largely moved only the yaw alignment of SRM. We used the stepping motor and moved SRM IP F0_Y from about -12073 to -27073. Then the flash could be seen. After that, we continued the alignment while monitoring the flash signal. Finally, we found good SRC flashes with the following SRM IM OPTICALIGN values:
- P: 3900
- Y: 480
The flash signal was monitored with K1:LSC-POS_PPOL_DC_OUT_DQ. This is the signal from the temporary PD installed yesterday on the POS table, and the flash appeared clearly in this channel.
However, since the suspension was still oscillating, it was difficult to further optimize the alignment at this moment.
Next, we entered the mine, center the QPDs, and try to turn on the local controls.
This foton error occurred because a filter bank block with one or more filter modules defined was removed from the model.
It was fixed by removing filter module definition of the removed filter bank block manually.
-----
In such case, "File errors" is displayed on a red background, and the error details pop up in a separate window when foton is opened as shown in Fig.1. In many cases, this type of error occurs when a filter bank is removed from the model while its filter design remains. This is because, when the model is rebuilt after deleting a filter bank block, the module name definitions at the top of the foton file are generated based on the new filter bank list, but the filter coefficient definitions are not cleaned up. Normally, this specification should not become a problem, because filter bank blocks are not repeatedly added or removed without sufficient consideration and reviews. But it often becomes an issue within KAGRA’s management and operation framework.
Anyway, VIS-SRM_TM_SENSCORR_L was removed from the k1vissrmp model probably in klog#36457 though some filter module definition was remained on that filter bank. An only safe method is to clean up the filter modules on the filter bank that will be deleted before installing the updated model. But in this case, nobody probably noticed fact that some filter banks would be removed because it's not mentioned in the model update plan (klog#36374). The other method is to edit the foton file manually after installing the updated model. Though foton files are in plain text format, their formatting is so strict and they can very easily become unreadable from foton or models by a manual edit. So it's never recommended. But this is only way if we couldn't notice before installing updated models and I did it in this time. Removed filter modules are shown in Fig.2.
After manual fixing, all errors were cleared on foton. But a warning still remains related to the too low (but non-zero) zero/pole as shown in Fig.3. Though it's not a problem on reading filters by foton and models, numerical errors often occurs on that filter bank. Please remove it if it's unnecessary, or please change corner frequency as a higher value with a careful design of the control if it's used.
Dan, Takano, Fujimoto, Saito, Tanaka
## Abstract
We tried to find the SRC flash by moving alignment SRM. Yaw alignment seems to be larger than the range of IM actuator. We observed SRC flash after moving SRM and SR3.
## What we did
This morning, we tried find the resonancre in SRC by moving the alignment of SRM largely in both P and Y directions. Unfortunately, SRM DC control seems not work well. So we stopped the any DC controls of payload parts, and we decided to use OPTICALIGN to move alignments. This time, we aligned ITMY and misaligned other Type-As, that is, we made up SRY cavity. Moreover, we increased IMC power up to 10 W. We used awggui and set the excitation amplitude ~1000 cnts and the frequnecy 0.05 Hz in Yaw and 0.04 Hz in Pit, respectively. In this parameters, SRM alignments moved largely enough TM oplev gots out of ranges. However, any flash could not be observed in AS RF34 IQ signals and DC power in AS PDs. Therefore, we put the additional PD at the POS table to observe the flash.
This afternoon, we entered the mine and installed the additional PD just after the 2 inch lens in main IR beam path on POS table. Also, we used the POS PPOL PD channel (K1:LSC-POS_PPOL_DC) for this PD tempolarly. After that, we tried again. However, any flash could not be observed. We used TCam to find the beam spot reflected SRM on ITMY.
Then, we increased Yaw scan amlitude to ~5000 cnts, which is the almost maximum. Then, Saito-kun found that the scattering at the edge of the TCam image.So we adjusted the actuator output when we observed the scattered light. Then, we oscllated the SRM only in Yaw direction. During the oscillation, we adjusted PItch by applying the DC offset with monitoring TCam image. After adjusting, the scattered light could be observed in the edge of both ITMY and ITMX TCam images once per 20 seconds. This duration seems to be consistent with the Yaw excitaition frequency. Therefore, the beam reflected from SRM seems to be shifted largely in Yaw direction. The shift amounts seems to be larger than the IM actuator range. So, it is necessary to use IP to move SRM more largely.
This time, we tried to moved SR3 in Yaw direction with oscllating SRM in Yaw direction. When SR3 moved to -100 urad from initial good setpoint, we can see the beam spot refleted from SRM in POP-SPOL camera. Moreover, the flash signals were observed in the additional PD and REFL PD when the beam spot reflected from SRM was overlaped on the beam spot reflected from ITM. Then, we stopped the excitation and set the actuator output when flash signal is observed in PDs (fig.1).
However, the beam went away from AS PDs and camera when SRC flashes.
Takano, Fujimoto, Tanaka, Dan
Summary
We performed the initial alignment of the IFO by following the procedure in klog36759. After that, we tried to find the SRC flash by shaking SRM and moving SRM IP in yaw direction. Finally, we observed clear SRC flashes.
What we did
Initial alignment of IFO
First, we performed the initial alignment of the IFO. Since the transmissivity of SRM was changed, the usual initial alignment procedure cannot be used for the AS side alignment. Therefore, we adjusted the alignment by referring to klog36759.
For X-arm and Y-arm, we used the usual initial alignment procedure. After that, we continued the alignment according to the procedure described in klog36759. As written in the klog, we moved OMMT1 while monitoring OMMT1_TRAS_QPD1. When the beam position on OMMT1_TRAS_QPD1 became almost centered, we regarded the AS side alignment as good.
Search for SRC flash
Next, we tried to find the SRC flash by adjusting the alignment of SRM. We set only ITMY to LOCK_ACQUISITION and misaligned the other Type-A suspensions. We also restored the SRM IM OPTICALIGN pitch value to yesterday’s value of 3825. In this condition, we largely moved only the yaw alignment of SRM. We used the stepping motor and moved SRM IP F0_Y from about -12073 to -27073. Then the flash could be seen. After that, we continued the alignment while monitoring the flash signal. Finally, we found good SRC flashes with the following SRM IM OPTICALIGN values:
- P: 3900
- Y: 480
The flash signal was monitored with K1:LSC-POS_PPOL_DC_OUT_DQ. This is the signal from the temporary PD installed yesterday on the POS table, and the flash appeared clearly in this channel.
However, since the suspension was still oscillating, it was difficult to further optimize the alignment at this moment.
Next, we entered the mine, center the QPDs, and try to turn on the local controls.
As the initial preparation of the IO chassis replacement, we laid the MTP-LC breakout cable from EX1 rack at the front-room of the 2nd floor to EXV1 rack.
The new cable is housed in the existing corrugated tube and will be used when IO chassis of K1EX1 will be replaced as V2 one.
V2 IO chassis for K1EX1 was also transported to the end station.
We will replace the IO chassis for K1EX1 on the maintenance day in near future.
Abstract
All the necessary items had been gathered just before holidays, so K1EX0 front-end computer was moved to EX1 rack at the 2nd floor.Though some equipment which is no longer necessary such as Timing Fanout are still in the EX0 rack, replacement of EX0 IO chassis itself was completed.
Clean-up of unnecessary equipment will be able to be done not on the maintenance day because we don't need to stop anything for this work.
Because an IRIG-B issue (klog#36705, klog#36819) was reproduced on K1EX0, IRIG-B card was also replaced as one which has no problem on the test stand.
After then, the issue was removed, so we kept to use the new IRIG-B card for K1EX0.
Problematic one will be checked again on the test stand.
Details
Although the IO chassis had been replaced in klog#36654, the move of the front-end computer to the second floor had been postponed due to a shortage of necessary cables. Because the necessary cables were obtained just before holidays, we moved the front-end computer from U13-14 of EX0 rack around the EXC chamber to the U22-23 of EX1 rack at the 2nd floor. There were no major issues with this work, and we didn't encounter any problems around PCIe cards such as ADC and DAC. With this work completed, the replacement of the EX0 I/O chassis is now completed, and the equipment in the EX0 rack such as the timing fanout, IRIG-B chassis, KVM, and 12V DC power supply unit is no longer needed. However, this unnecessary equipment is still left at EX0. Since it can be removed without disrupting the digital system, we plan to remove it at an convenient time.After moving the computer to the second floor, the IRIG-B issue reproduced during the first startup, and neither a reboot nor a cold boot resolved the problem. Though we have not identified a clear cause or solution for this issue yet, we replaced the IRIG-B card with one that has no problem on the test bench, and the issue was removed. So we kept to use the new IRIG-B card for K1EX0. We cannot conclude this issue is caused by an individual difference of IRIG-B card from today's result yet. If so, a problematic cards on another front-end computers also must be replaced (at least 3 cards). We faced this issue in 3 of 4 front-ends that we worked on recently. Though I'm not sure it comes from a problem rate or just bad luck, it’s possible that a certain percentage of the cards currently in use at the coner station have the same problem, which may require replacements for more units.
Error: Permission denied, please try again.
Permission denied, please try again.
Permission denied, please try again.
A permission error occurred during the callback process from the HWP control PC to k1script.
k1script0
The callback from HWP, which had been specified in .ssh/authorized_keys, was commented out.
Since this was an emergency response, we first restored the original settings. Because the callback for the final move command was malfunctioning, we first performed a command-line callback from the HWP control PC to k1script0 to restore the callback that had been causing the permission error.
After that, we confirmed that HWP was responding correctly to commands from Guardian and performing the necessary processing.
I implemented the hierarchical control for TM with the Oplev. To feed back the DC signals to the IM pitch and yaw, the transfer functions from the IM actuator to the TM Oplev were measured (Ptot 1 and 2). The "DC" (FM9) in the OLDAMP FILTERS in the TM loop was switched off, and the "int" (FM10) in the OLDAMP FILTERS in the IM loop was switched on. The cross-frequency between the TM and IM loops is now below 0.1Hz (I couldn't edit the filter modules in Foton due to the file errors).
This foton error occurred because a filter bank block with one or more filter modules defined was removed from the model.
It was fixed by removing filter module definition of the removed filter bank block manually.
-----
In such case, "File errors" is displayed on a red background, and the error details pop up in a separate window when foton is opened as shown in Fig.1. In many cases, this type of error occurs when a filter bank is removed from the model while its filter design remains. This is because, when the model is rebuilt after deleting a filter bank block, the module name definitions at the top of the foton file are generated based on the new filter bank list, but the filter coefficient definitions are not cleaned up. Normally, this specification should not become a problem, because filter bank blocks are not repeatedly added or removed without sufficient consideration and reviews. But it often becomes an issue within KAGRA’s management and operation framework.
Anyway, VIS-SRM_TM_SENSCORR_L was removed from the k1vissrmp model probably in klog#36457 though some filter module definition was remained on that filter bank. An only safe method is to clean up the filter modules on the filter bank that will be deleted before installing the updated model. But in this case, nobody probably noticed fact that some filter banks would be removed because it's not mentioned in the model update plan (klog#36374). The other method is to edit the foton file manually after installing the updated model. Though foton files are in plain text format, their formatting is so strict and they can very easily become unreadable from foton or models by a manual edit. So it's never recommended. But this is only way if we couldn't notice before installing updated models and I did it in this time. Removed filter modules are shown in Fig.2.
After manual fixing, all errors were cleared on foton. But a warning still remains related to the too low (but non-zero) zero/pole as shown in Fig.3. Though it's not a problem on reading filters by foton and models, numerical errors often occurs on that filter bank. Please remove it if it's unnecessary, or please change corner frequency as a higher value with a careful design of the control if it's used.
Previous locked loss 9th May. 02:12 (JST)
From my eye, the HWP PSL didn't move, could someone check it?
Error: Permission denied, please try again.
Permission denied, please try again.
Permission denied, please try again.
A permission error occurred during the callback process from the HWP control PC to k1script.
k1script0
The callback from HWP, which had been specified in .ssh/authorized_keys, was commented out.
Since this was an emergency response, we first restored the original settings. Because the callback for the final move command was malfunctioning, we first performed a command-line callback from the HWP control PC to k1script0 to restore the callback that had been causing the permission error.
After that, we confirmed that HWP was responding correctly to commands from Guardian and performing the necessary processing.
However, in March, an issue was observed on k1mon3 where the displayed positions of the GigE cameras became unstable. This phenomenon has not reoccurred since then, suggesting that its correlation with network speed may be limited.
Even so, the control room displays a large number of ndscope channels along with camera images. To further improve the network environment, a new network switch has been installed. This switch is connected to the upstream network at 10 Gbps, and the available bandwidth is currently considered sufficient.
The switch is installed on the PC wagon of k1mo10. It is connected via an optical fiber cable laid from the computer room next to the control room.
For installation, k1ctr2 and k1ctr3 were temporarily shut down to ensure accessibility. For k1mon[0–4,10] and k1ctr[4–5], the LAN cables were switched without shutting down the workstations. After the work, connectivity with all these WS was confirmed.
To be safe, the previous LAN cables and network switch have been retained in case any issues arise. If no problems are observed after a period, removal of the old equipment will be carried out.
Though a mitigation measures for this issue were applied all affected DGS servers including intranet ones, a temporal solution is still used for k1gate and gwdet because vendor's patches for Red Hat like OS hadn't been released yet in last time. Today vendor's patches were released but there was no enough time to test and apply them because of HDD trouble on k1script0. So I will apply them to k1gate and gwdet in next time and will remove a temporal solution applied in klog#36851.
I could check that there's an intermittent loud noise today. (I couldn't tell what sounds. fan?)
This sounds came from the power supply unit of DC+18V for PRM/PR3 racks.
In the past, we had a trouble on the power supply unit of DC-18V for PRM/PR3 (see also klog#20372).
Situation is quite different from the past case, both two cases occurred on PRM/PR3.
(These two units located at the most warm area in the server room).
It seems better to investigate a cause of this issue and to consider the replacement of the power supply unit.
For replacing it, we need to check spare equipment at first and then, to stop all circuits in PR3 and PRM racks.
These are the results of verification conducted using the test bench at the SK Computer room.
K-Log#36698
> Remaining concern
> Only remaining concern is that it takes a few hours for synchronizing IRIG-B when a cold boot of the front-end computer is performed. In the past, similar things often occurred when down time became so long and a room temperature changed largely maybe because it took long time to be stable in the temperature of a crystal oscillator of timing slave. On the other hand, in this time, it takes long time even when a down time is only a few seconds. Such a thing wasn't reproduced in the test bench. So we have no idea to solve this issue now.
The abnormal behavior of IRIGB_TIME could also be reproduced in the SK computer room.
We believe the cause lies in the IRIG-B card itself.
Using a test bench that operates normally, when the IRIG-B card (S/N: 3799) was installed, both decreases and increases in the value were observed.
Afterward, the issue still occurred even when replacing the system with the old IO chassis (V1 IO Chassis).
No issues have been observed with the following IRIG-B cards at SK computer room:
02181
3796
02174
3793
02158
02172
=> Tested in slot 2 with full-height configuration; all others are Low-profile
02171
=> Both the yellow and green LEDs light up, and the yellow LED does not turn off (possible connection failure on this card?)
Based on the above observations, we believe this issue is specific to individual IRIG-B cards.
Additional note: We need to consider that this issue is not caused by the IRIG-B card alone, but is occurring in combination with other factors.
After investigation, I found files required to fix broken files and sectors were broken, and gave up to fix broken files and sectors on the original system disk.
Finally, I recovered the script server by using the backup disk taken in last month.
-----
Backup disk was slightly older than current configuration, so I applied all modification based on JGW-T2617213. After unifying the backup disk to the current nominal configuration, I confirmed all services on k1script0 were recovered. Then, security update I had planned to apply was applied to k1script0. Finally, a new backup disk of latest system disk was made.
Complete user code check hasn't been done because it requires to enter each GuardState one-by-one.
The GStreamer environment must be set up for a mode analysis by OMC_LSC guardian.
All guardian nodes should work fine with minor (but might be many) user code fixing after setting up the GStreamer environment.
Special Notes
- python3-foton package was installed for loading kagralib.py. Because the automatic filter generation function by Guardian that Nakano-kun used is likely no longer in use, python3-foton may no longer be necessary.
- Log-in configuration from the new guardian server to k1dc0 and TCam servers was set for SYS_DAQ and TCam guardians. Now DAQ kill and Taking TCam photos work fine on the new guardian server.
- Thanks to the net-masquerade for PICO network by k1script, HWP in LSC_LOCK guardian can be used without any modification.
- Mode analysis binary can be used also on the new server without any linker mismatch.
- Taking GigE picture for mode analysis of OMC_TRANS haven't work yet due to missed some plugins of GStreamer. Set up of them on the new guardian server will be required.
I opened GVsrm and closed the GV between TMP at SRM and the T-duct at 9:40. The vacuum pressure around SRM changed as follows,
(1) Open GVsrm: 4e-5Pa -> 3e-5Pa,
(2) Closed GV between TMP and T-dauct at SRM: 3e-5Pa -> 3hour -> 5.5e-5Pa.
Since the vacuum pressure became stable at 5.5e-5Pa, I stopped TMP.
Before stopping TMP, I set SRM and SR3 safe. After stopping TMP, I set SRM isolated and SR3 Lock_acquistion.
Because the guardctrl, guardlog, and guardutils commands determine which Guardian server to connect to based on the GUARD_HOST, GUARDCTRL_HOST and GUARDLOG_HOST environment variables, we cannot currently view logs of SYS_SDF from the MEDM screen. Additionally, we cannot use guardctrl to perform start/stop/restart operations (not EXEC/PAUSE/STOP via MEDM screens) without changing these environment variables to "k1grd1". Since I don’t think we would perform the above operations including viewing logs on SYS_SDF when it is not under observing runs, it shouldn't be a problem. But, please be careful if such a situation arises.
Any other guardian nodes are still running on k1grd0 which is a current production server and haven't been tested yet. So the others can be operated as usual.
Complete user code check hasn't been done because it requires to enter each GuardState one-by-one.
The GStreamer environment must be set up for a mode analysis by OMC_LSC guardian.
All guardian nodes should work fine with minor (but might be many) user code fixing after setting up the GStreamer environment.
Special Notes
- python3-foton package was installed for loading kagralib.py. Because the automatic filter generation function by Guardian that Nakano-kun used is likely no longer in use, python3-foton may no longer be necessary.
- Log-in configuration from the new guardian server to k1dc0 and TCam servers was set for SYS_DAQ and TCam guardians. Now DAQ kill and Taking TCam photos work fine on the new guardian server.
- Thanks to the net-masquerade for PICO network by k1script, HWP in LSC_LOCK guardian can be used without any modification.
- Mode analysis binary can be used also on the new server without any linker mismatch.
- Taking GigE picture for mode analysis of OMC_TRANS haven't work yet due to missed some plugins of GStreamer. Set up of them on the new guardian server will be required.
Then we can access k1ctr15 remotely, so I applied the same security patch and rebooted it.
Updates for all workstations (k1ctr0-21, 27, k1mon0-11, and k1naoj02-07) were completed.
Tanikawa, Uchiyama
Around 12:40, we noticed a sudden pressure increase and then a decrease again.
Since we suspected that the ion pump had stopped unexpectedly, we entered KAGRA and checked the ion pump. At the KAGRA site, we found, however, that the ion pump and TMP worked normally.
We continue to see changes in the vacuum pressure in SRM.
Tanikawa, Uchiyama
We turned of the ion pump at SRM (#14) at 9:34. After that the vacuum pressure in SRM increased and reached 1E-3Pa at about 10:50.
I checked the TM transfer functions in a vacuum. The IP setpoints were reset to the original (-164 to -550 for L and -24 to -50 for T), so the Length Oplev was also aligned. The transfer functions were consistent with the references.
I checked the IM transfer functions in a vacuum. They were the same as the recent situation in the vacuum (the DC gain is smaller than the reference, except for L and H1).
I checked the GAS transfer functions in a vacuum. They were consistent with the references.
I checked the IP transfer functions in a vacuum. IDAMP signals are not treated well, so please see the LVDT responses in BLEND signals. The resonant frequencies of the IP were 63mHz for L, 74mHz for T, and 0.32Hz for Y, respectively.
EYC1F temperature seemed to have decreased by 2 C degrees for 2 months.
I increased the heater power as follows,
- Delonghi 0 -> 1.2kW
- Sunrise 1 : 900W -> 1.2kW
- Sunrise 2: 600W -> 1.2kW