Again on 31 Dec.

• 6:41, stop cooling
• 6:47, start cooling

Accidentally (or not), this event happened around the same time of ~6:40 with on 29th Dec.

Again on 1 Jan.

• 5:16, stop cooling
• 5:21, start cooling

Again on 1 Jan.

• 20:23, stop cooling,
• 20:29, start cooling,

To check the relation between the temp around the BS-Daikin cooler and these temporal stops of cooling, I turned on the Sunrise heater near the BS-Daikin cooler at level 1 (~ 600W) around 22:00.

Again on 2 Jan.

• 11:39, stop cooling,
• 11:45, start cooling,

Again on 3 Jan.

• 22:41, stop cooling,
• 22:47, start cooling,

Again on 5 Jan.

• 3:12, stop cooling,
• 3:18, start cooling,

Again on 19 Jan (Sunday).

• 21:11, stop cooling (COOLER_BS:  current=21.9,  nominal=14.5,  diff=7.399999999999999)
• 21:18, start cooling (back to the nominal range)

Yamamoto-kun suspected that it should be a bug on Ondotori device or the data acquisition script because the temperature was changed ~12degC immediately. On the other hand, 21.9C is a reasonable temp that is the same as the surrounding temp and the temp could increase immediately to the room temperature when the Daikin cooler stopped cooling.

[Oshino, YamaT]

We investigated what was the actual reason of this issue and finally concluded that it came from the timing conflict of reading and writing of measured temperature of Ondotori XML files.
We are now testing an improved code by using DAQ of the room temperature at Mozumi server room.
After we can confirm it works well, we plan to deploy the new code for the DAQ of the temperature in the mine.

-----
At first we found the immediate change as ~12degC in EPICS sampling (16Hz) on that channel as shown in Fig.1. It shouldn't be a real temperature change. So we were able to focus on the software or Ondotori device issue soon. According to a check of raw data of Ondotori (XML file shared via ftp@k1nfs0), BS cooler channel stably showed  9.6~9.7degC and we couldn't see a value of 21.9degC which was shown in slack alerts. This fact means that Ondotori device took a correct data and DAQ script couldn't read a correct value from XML files. Because there was some possibility that this problem came from some network or IOC troubles, I also checked a log of caput in the DAQ script just in case. Then, we could confirm DAQ script surely put 21.9degC to that channel.

After checks above, we considered how to put wrong value by the DAQ script and found soon that Ondotori DAQ script didn't take care about the timing relation between reading XML files and writing XML files by Ondotori device. And also the data array for read values are not initialized. In common case, when some function is called multiple time heap and stack region for local variables are assigned in same memory addresses. An another complicated issue is that XML file contains the data from multiple child devices and order of data is not constant (depending on the network speed at that time). As the result of these several intertwined issues, uncleared old value of another child device is read from memory region when DAQ script read XML files while Ondotori device are writing XML files.

After finding an actual problem, Oshino-san prepared modified DAQ code to avoid reading XML files while Ondotori device are writing files. It's running as DAQ for the room temperature monitor of Mozumi server room. After we can confirm that it works well, we plan to deploy it for all Ondotori devices in the mine.