I tried to reproduced a change in the pressure value (KGRVAC:CCG_BSY_01:VAL:PRESS) after it became larger than the threshold of guardian notification from the guardian log (See Fig.1).
Log is recorded only when pressure value was larger than the threshold of guardian notification and pressure value is changed, so sampling interval is not a constant value.

Guardian threshold was set as 8.0e-6Pa in klog#32116. The first log appears at 10:55:35.885674 JST. That is, the time when the pressure value first reached the Guardian threshold. Interlock threshold which was shown as red line in Fig.1 was set as 9.4e-6Pa in klog#32173. Readout value first reached interlock threshold at 10:57:52.902407 JST. It took 147s to increase from 8.0e-6Pa (Guardina threshold) to 9.4e-6Pa (interlock threshold). Fig.2 is a same plot with longer duration. Attached text file is an original guardian logs used to make these plots.
 

We got the following message in JST, 

• @miyoki@uchiyama@Nobuhiro KIMURA [vacuum-alert] GV is closed 2025-01-12 08:03:28.332043 GV_ITMY GV is closed
• @miyoki@uchiyama@Nobuhiro KIMURA [vacuum-notification] GV is opened 2025-01-12 08:03:32.829372

Only 4 seconds difference between these messages.

But, all GVs are open according to webcam images(Fig.1 and 2), except for the IYC-BS side(Fig.3. The high elevation stand is an obstacle!!), and the web-monitor page.

The GV closing setting values are listed in klog#32173.

Uchiyama-kun found a NULL or zero signal for GV_ITMY as in the attached file (Fig.4). Only the message about closing seemed to be sent, and no closing GV happened.

No remarkable enhancement of vacuum level in any places as Fig.5. (-1 hour and 10 minutes is the timing when the warning was sent via slack in this Fig.)

We got the following message in JST,

• 2025-01-12 13:12:16.893314 GV_BSY GV is closed GV_BSX GV is closed.
• 13:12@miyoki@uchiyama@Nobuhiro KIMURA [vacuum-alert] out of the nominal range 2025-01-12 13:12:32.844101
• KGRVAC_CCG_BSY_01_VAL_PRESS:  current=8e-06,  threthold=8e-06

Then, actually, we confirmed that GVs were closed. As before, the vacuum level at the MICH area started increasing because of the loss of the powerful cryo pumping effect of IXYC cryostat themselves.

Fig.4 and 5 show temps in shields of IXYC cryostat. In IY, 4K bar ref and 8k shields reached around 35K which is the near the temp of the nitrogen adsorption temp (~ 30K). Is there any other temps at which some gas is adsorbed? On the other hand, in IX, these temps reached ~23K which is below the nitrogen adsorption temps.

I checked the temp status of IXYC coolers. I found 

• temps at 4K heads/bars of REF1/3 showed several K enhancements.
• while, the 50K heads/bars of REF1/3 showed several K reduction.
• REF1 4K Bar shows temp fluctuation from about 5 days ago, and became once stable about 2.5 days ago. 
• REF3 4K head temp suddenly jumped up about a half day ago.

These phenomena cannot be found in coolers in IXC as Fig.1. So some cooling unbalance in cryocoolers in REF1/3 in IY happened.

As a result, the temp at 8K shield in IY also increased by several K from 30K which is just the same temp to the Nitrogen adsorption temp. So the Nitrogen release could be the vacuum level enhancement source between IYC and BS.

It is better to check the Nitrogen partial pressure in IY immediately.

About the false alarm at 8:03:28 reported in klog#32316
> @miyoki@uchiyama@Nobuhiro KIMURA [vacuum-alert] GV is closed 2025-01-12 08:03:28.332043 GV_ITMY GV is closed
> @miyoki@uchiyama@Nobuhiro KIMURA [vacuum-notification] GV is opened 2025-01-12 08:03:32.829372
A NULL or zero signal for GV_ITMY found by Uchiyama-san have rarely been seen on StripTool before. But we had thought it's a reading error on the StripTool itself. In this time, we was able to confirm that this issue was occurred on two independent clients (k1mon10 and k1grd0) by newly implementing the Guardian alert about GVs. So I wonder this issue is not a reading error on the clients but some kind of a sending error on MOXA/Saba-taro which is the interface of CC-10 for making EPICS records.

A important thing is that an interval of two alerts ("GV is closed" and "GV is opened") is ~4 seconds. It means a NULL or zero signal was provided in continuous 4 seconds (We need DAQ for these channels for more careful and detailed checks). If an interval of two alerts is only 1 or 2 samples (1/16s or 2/16s), Guardian can take care about this issue easily by changing the trigger condition as closed flag is enabled in continuous two samples. But a wrong signal in 4 seconds is too long for Guardian and also for the safety checks. So I think that this issue should be removed or at least mitigated on MOXA/Saba-taro side.

About the actual alert at 13:12:16 reported in klog#32319
> 2025-01-12 13:12:16.893314 GV_BSY GV is closed GV_BSX GV is closed.
> 13:12@miyoki@uchiyama@Nobuhiro KIMURA [vacuum-alert] out of the nominal range 2025-01-12 13:12:32.844101
> KGRVAC_CCG_BSY_01_VAL_PRESS: current=8e-06, threthold=8e-06
An alert about closing GVs were sent before an alert about increasing a pressure value.

The threshold for the alert about increasing a pressure value was set as 8.0e-6Pa in klog#32116. This value was decided based on the past threshold for closing GVs as 9.4e-6Pa in klog#32173. Unfortunately, the threshold for closing GVs was changed to 7.5e-6Pa which was lower than one for increasing a pressure value (klog#32306) and it's not shared before entering weekend. So the guardian alert for increasing a pressure value no longer worked. For this reason, there is no guardian log before closing GVs and I cannot do a same thing as klog#32290.

Kimura-san reported about nitrogen partial pressure in IYC as,

• Jan 11 07:43 --- 6.04 x 10^-7 Pa (before GV closing)
• Jan 13 19:06 --- 1.93 x 10^-6 Pa (  after GV closing)

Nitrogen partial pressure reduction should be expected if GV closing happens because more outgas from the BS area will be shut. However, an increase was observed. So this nitrogen partial pressure increase may be because of the nitrogen gas release from the 8K shield or so as I supposed in the klog input.

Although it depends on the amount of outgassing, it is better to set a higher trigger vacuum level. Of course, the unstable cooling performance should be removed if possible.

We got the following messages in JST,

• @miyoki@uchiyama@Nobuhiro KIMURA [vacuum-alert] GV is closed
• 2025-01-13 08:43:48.838831
• GV_BSY GV is closed
• GV_BSX GV is closed
• GV_ITMX GV is closed

However, "GV_ITMX GV is closed" is wrong judging from the web control panel and webcam image.

The KGRVAC:GV_ITMX:STAT:OPEN showed NULL or zero signal as Fig.1 in this time. The similar phenomenon was reported in klog#32316, and Yamamoto-kun explained as klog#32321.