STH Recent Preflight: 61209

Executing preflight for 61209 at Wed Jun 17 20:19:19 SAST 2026.
Log can be found in /atlas/obs/03/61209/preflight.dat.
Setting donefile '/atlas/obs/03/61209/preflight_complete.20260617181919.21852' on successful script completion.
Sitech host is tcs.
Disabling voice.exe countdown.
Disabling chiller checks.
Disabling dehumidifier power check.
Logging errors to /tmp/preflight.20260617181919.21852.
Wed Jun 17 20:19:19 SAST 2026 [elapsed=0 sec] DFM (initialization)
  get service string...OK
  service is running...OK
  is initialized...OK

Wed Jun 17 20:19:19 SAST 2026 [elapsed=0 sec] Getting TCS lock.

OK TCS lock granted to preflight-STH [6b227c9b-b62d-4276-aa20-dff162298199;1800]
Wed Jun 17 20:19:23 SAST 2026 [elapsed=4 sec] Enabling red LEDs for preflight.
doorpsw  3  ON redLED
Wed Jun 17 20:19:24 SAST 2026 [elapsed=5 sec] About to start STH preflight.  Point your web browser at http://localhost:7354.
Wed Jun 17 20:19:24 SAST 2026 [elapsed=5 sec]   auxcam is disabled.
Wed Jun 17 20:19:24 SAST 2026 [elapsed=5 sec]   Skipping auxcam.
error: unknown module: vent
Wed Jun 17 20:19:24 SAST 2026 [elapsed=5 sec] DEADMAN
  Get heartbeat host ... got 'tcs'
  Fetching timestamp (redis-cli -h tcs) ... done. Got response: 1781720357
OK
Wed Jun 17 20:19:24 SAST 2026 [elapsed=5 sec] OPSEC
  AUTO mode:  ...OK
Wed Jun 17 20:19:24 SAST 2026 [elapsed=5 sec] CLUSTER NFS MOUNTS
  admin proc1:/export/atlas-cal...OK
  admin proc1:/export/atlas-red...OK
  proc1 admin:/export/atlas-obs...OK
  proc2 admin:/export/atlas-obs...OK
  proc2 proc1:/export/atlas-cal...OK
  proc2 proc1:/export/atlas-red...OK
  proc3 admin:/export/atlas-obs...OK
  proc3 proc1:/export/atlas-cal...OK
  proc3 proc1:/export/atlas-red...OK
  proc4 admin:/export/atlas-obs...OK
  proc4 proc1:/export/atlas-cal...OK
  proc4 proc1:/export/atlas-red...OK

Wed Jun 17 20:19:25 SAST 2026 [elapsed=6 sec] SCREENS
  admin dash-tcsdata...OK
  admin dash-recent...OK
  admin dash-skycoverage...OK
  admin dash-logstamp...OK
  proc1 redq...OK
  proc1 prochain-redpost-03a...OK
  proc1 prochain-redpost-03x...OK

Wed Jun 17 20:19:26 SAST 2026 [elapsed=7 sec] QUEUE
  0 observations queued.

Wed Jun 17 20:19:26 SAST 2026 [elapsed=7 sec] CLOCKS
  admin:      Wed Jun 17 18:19:26 UTC 2026
  met:        OK
  proc1:      OK
  proc2:      OK
  proc3:      OK
  proc4:      OK
  archon:     OK

  NTP:
       remote           refid      st t when poll reach   delay   offset  jitter
  ==============================================================================
   192.168.4.5     .GPS.            1 u  94m  512    0    1.018    0.345   0.000
  *metbox2.sth.atl .PPS.            1 u  199  256  377    0.279   -0.077   0.018
   ntp.ubuntu.com  .POOL.          16 p    -   64    0    0.000    0.000   0.000
  -zibbi.ngei.csir 155.232.19.6     2 u   47  256  377   23.168    0.088   0.054
  -ntp.as3741.net  210.182.58.253   2 u  240  256  377   22.725   -0.119   4.725
  +ntp2.as3741.net 155.232.19.7     2 u   93  256  377    7.155   -0.122   0.299
  -ntp2.inx.net.za 210.182.58.253   2 u  109  256  377    7.216    0.095   0.127
  -ntp5.inx.net.za 102.176.229.251  2 u  126  256  377    7.446   -0.029   0.037
  -102.222.156.151 196.21.187.2     2 u  246  256  377   23.276   -0.197   0.776
  -102.222.103.99  80.72.67.48      3 u   95  256  377    7.370    0.091   1.035
  +139.84.231.68 ( 196.21.187.2     2 u  205  256  377   22.765   -0.090   0.073
  -ntp8.inx.net.za 210.182.58.253   2 u  257  256  377   16.785   -0.132   0.076

Wed Jun 17 20:19:27 SAST 2026 [elapsed=8 sec] DISKS
  200GB available in /atlas/obs...OK (6208GB)
  200GB available in /atlas/red...OK (9033GB)

  Checking disks on admin ... archon ... tcs ... proc1 ... proc2 ... proc3 ... proc4 ... done

  Filesystem                             Size  Used  Avail  Use%  Mounted_on
  admin:/dev/md0p2                       11T   4.3T  6.0T   43%   /
  admin:/dev/loop0                       106M  106M  0      100%  /snap/core/17284
  admin:/dev/loop1                       106M  106M  0      100%  /snap/core/17292
  archon:/dev/sda2                       117G  22G   89G    20%   /
  archon:/dev/loop1                      106M  106M  0      100%  /snap/core/17284
  archon:/dev/loop0                      106M  106M  0      100%  /snap/core/17292
  archon:/dev/sda1                       511M  6.6M  505M   2%    /boot/efi
  tcs:/dev/mapper/ubuntu--vg-ubuntu--lv  98G   37G   56G    40%   /
  tcs:/dev/nvme0n1p2                     2.0G  131M  1.7G   8%    /boot
  tcs:/dev/nvme0n1p1                     1.1G  6.1M  1.1G   1%    /boot/efi
  proc1:/dev/md0p1                       15T   4.8T  8.7T   36%   /
  proc1:/dev/loop0                       106M  106M  0      100%  /snap/core/17284
  proc1:/dev/loop1                       106M  106M  0      100%  /snap/core/17292
  proc2:/dev/md0p1                       15T   1.7T  12T    13%   /
  proc2:/dev/loop0                       106M  106M  0      100%  /snap/core/17284
  proc2:/dev/loop1                       106M  106M  0      100%  /snap/core/17292
  proc3:/dev/md0p1                       15T   736G  13T    6%    /
  proc3:/dev/loop0                       106M  106M  0      100%  /snap/core/17292
  proc3:/dev/loop1                       106M  106M  0      100%  /snap/core/17284
  proc4:/dev/md0p1                       15T   1.1T  13T    8%    /
  proc4:/dev/loop1                       106M  106M  0      100%  /snap/core/17284
  proc4:/dev/loop0                       106M  106M  0      100%  /snap/core/17292

Wed Jun 17 20:19:27 SAST 2026 [elapsed=8 sec] SERVICES1
qcam
  chiller...OK
  dome...OK
  mount...OK
  modo...OK
  cam...OK
  acam...OK
  weather...OK
  met...OK
  sentry...OK
  dfm...OK
  duplicates...OK

Wed Jun 17 20:19:27 SAST 2026 [elapsed=8 sec] Disabling mount tracking.
Wed Jun 17 20:19:27 SAST 2026 [elapsed=8 sec] SERVICES2

Wed Jun 17 20:19:27 SAST 2026 [elapsed=8 sec] CHILLER
  status...OK
  chiller: 3.0/3.0C ? chiller=ON dryer=? IDLE


Wed Jun 17 20:19:31 SAST 2026 [elapsed=12 sec] DOME
  status...OK
  audio setup: using amcam-mezz-s to record
  Pointing webcam to azcheck position ... done
  Homing the dome ... waiting ... done
  Attempting audio conversion ... size=     910kB time=00:00:29.12 bitrate= 256.0kbits/s speed=1.43e+03x    
audio converted to domehome.wav.
  Copy dome audio to dashboards ... done
  slewing to azimuth 0 (N)...OK
  Copy azcheck JPEG to dashboards ... done
  parking...OK
  dome: AZ=90.00 CLOSED/OPEN (0%) IDLE


Wed Jun 17 20:22:02 SAST 2026 [elapsed=163 sec] MOUNT
  blinky  status...OK
  motor logs...OK
  slewing to az=70,alt=70...OK
  slewing to az=290,alt=70...OK
  parking...OK
  mount: 200.23 -29.49, az/alt=324.77 +86.49 idle(e) IDLE


Wed Jun 17 20:22:53 SAST 2026 [elapsed=214 sec] MODO
  status...OK
  slewing to az=60,alt=60...OK
  modo: dome=stopped mount=tracking IDLE


Wed Jun 17 20:23:12 SAST 2026 [elapsed=233 sec] CAM SERVICE
  status...OK
  cam: main=[idle] aux=[off] N=0/0 spi=38.8 IDLE

Wed Jun 17 20:23:12 SAST 2026 [elapsed=233 sec] ACAM
  status  Camera tripped exists... OK
  Camera tripped is current... OK (10 seconds old)
  Software failsafe ready... OK (NA)
  CCD on...OK
  ION pump on...OK
  CCD temp...OK (-54.3C)
  Checking CCD plane: Z0=+1706.00 DZDX=-885.00  DZDY=+2314.00
  acam: -54C CCD TEC ION e=30.0s d=0.6s N=0 IDLE

Wed Jun 17 20:23:12 SAST 2026 [elapsed=233 sec] WEATHER
  status  weather: clear 12.2C 66% 9/10kph@55 DP5.7C/-18C/d IDLE
...OK

Wed Jun 17 20:23:13 SAST 2026 [elapsed=234 sec] MET
  status...OK
  met: c=N o=Y Ro5d Co1d Wo5d Ho19h To5d Do3d  IDLE

Wed Jun 17 20:23:13 SAST 2026 [elapsed=234 sec] SENTRY
  status...OK
  sentry: close=NO open=YES drop=YES IDLE

Wed Jun 17 20:23:13 SAST 2026 [elapsed=234 sec] DFM (SHUTTERS)
  initializing and homing shutters...
mount          229.14 -14.64, az/alt=59.90 +60.06 tracking(w)      IDLE      1s
A     0  0.0mm   15 N F R | B     0  0.0mm   15 N F R
initialized motor A constants
initialized motor B constants
A     0  0.0mm   15 N F R | B     0  0.0mm   15 N F R
A  0.020s     3   0.0%
B  0.020s    -3   0.0%
A     2  0.1mm   15 N F R | B    -2 -0.1mm   15 N F R
finding forward (close) limit for shutter A
...done
backing off limit
............................backing off 4 sec...
at limit
done
finding forward (close) limit for shutter B
...done
backing off limit
................backing off 4 sec...
at limit
done
A     0  0.0mm   15 N F R | B     0  0.0mm   15 N F R
  exercising shutters...Using amcam-mezz-s to record
A  0.020s     9   0.1%
A  0.051s   113   1.2%
A  0.081s   402   4.4%
A  0.111s   746   8.1%
A  0.142s  1206  13.2%
A  0.172s  1803  19.7%
A  0.202s  2489  27.2%
A  0.232s  3286  35.9%
A  0.263s  4194  45.8%
A  0.293s  5145  56.2%
A  0.323s  6018  65.7%
A  0.354s  6827  74.6%
A  0.384s  7476  81.7%
A  0.414s  8035  87.8%
A  0.445s  8491  92.8%
A  0.475s  8811  96.3%
A  0.505s  9033  98.7%
A  0.536s  9156 100.0%
B  0.020s     0   0.0%
B  0.050s    90   1.0%
B  0.081s   374   4.1%
B  0.111s   713   7.8%
B  0.141s  1179  12.9%
B  0.172s  1756  19.2%
B  0.202s  2440  26.7%
B  0.232s  3231  35.3%
B  0.262s  4136  45.2%
B  0.293s  5074  55.4%
B  0.323s  5962  65.1%
B  0.353s  6752  73.8%
B  0.383s  7416  81.0%
B  0.414s  7994  87.3%
B  0.444s  8443  92.2%
B  0.474s  8790  96.0%
B  0.505s  9024  98.6%
B  0.535s  9143  99.9%
B  0.565s  9162 100.0%
sleeping 3 sec between shutter moves...
A  0.020s  9150 100.0%
A  0.050s  9065  99.0%
A  0.081s  8774  95.8%
A  0.111s  8415  91.9%
A  0.141s  7969  87.1%
A  0.172s  7374  80.6%
A  0.202s  6692  73.1%
A  0.232s  5898  64.4%
A  0.262s  4983  54.4%
A  0.293s  4044  44.2%
A  0.323s  3165  34.6%
A  0.353s  2379  26.0%
A  0.384s  1708  18.7%
A  0.414s  1147  12.5%
A  0.444s   698   7.6%
A  0.475s   354   3.9%
A  0.505s   127   1.4%
A  0.535s    13   0.1%
A  0.566s    -8   0.0%
B  0.020s  9145  99.9%
B  0.050s  9034  98.7%
B  0.081s  8772  95.8%
B  0.111s  8425  92.0%
B  0.141s  7959  86.9%
B  0.171s  7379  80.6%
B  0.202s  6694  73.1%
B  0.232s  5898  64.4%
B  0.262s  4984  54.4%
B  0.292s  4051  44.3%
B  0.323s  3162  34.5%
B  0.353s  2384  26.0%
B  0.383s  1713  18.7%
B  0.414s  1141  12.5%
B  0.444s   697   7.6%
B  0.474s   354   3.9%
B  0.504s   127   1.4%
B  0.535s    11   0.1%
B  0.565s    -6   0.0%
A     1  0.0mm   15 N F R | B    -5 -0.2mm   15 N F R
---------------------------------------------------
Evaluating shutter behavior:
Shutter A opened to 100.0 percent in 0.536 seconds.
Shutter B opened to 100.0 percent in 0.565 seconds.
Shutter A closed to 0.0 percent in 0.566 seconds.
Shutter B closed to 0.0 percent in 0.565 seconds.
Problems: 0
---------------------------------------------------
Attempting audio conversion ... size=     228kB time=00:00:07.29 bitrate= 256.1kbits/s speed=1.07e+03x    
audio converted to shutter.wav.
Copying shutter audio to dashboards ... done

DFM (FOCUS)
  changing to best focus...  dfm: UNINITIALIZED filter=[1] foc=0 IDLE

Wed Jun 17 20:23:48 SAST 2026 [elapsed=269 sec] IMAGING (acam+auxcam)
  status...OK
  cam: main=[idle] aux=[off] N=0/0 spi=38.8 IDLE
  taking preflight exposures:
  acam...OK
  acam image:
    73M Jun 17 20:24 /atlas/obs/03a/61209/03a61209e0001.arch.fz
  AUXCAM: Skipping (noauxcam=1).

Wed Jun 17 20:24:11 SAST 2026 [elapsed=292 sec] DEHUMIDIFIER
  OTA humidity exists...OK
  OTA humidity is current... OK (22 seconds old)
  OTA humidity level...OK (8%)
  Dome humidity exists...OK
  Dome humidity is current... OK

Wed Jun 17 20:24:11 SAST 2026 [elapsed=292 sec] SCHEDULE
  Status...OK
  Checking scheduled filter...FAIL


Wed Jun 17 20:24:11 SAST 2026 [elapsed=292 sec] Parking the mount.
Parking ... mount: task <-parksouth> accepted
Waiting ... Mount is parked.

Wed Jun 17 20:24:37 SAST 2026 [elapsed=318 sec] Disabling mount tracking.
  Waiting on the mount ... done.
  Turning tracking off ... done.
  Waiting on the mount ... done.
  Recheck mount status ... done.
doorpsw  3 OFF redLED
doorpsw  2 OFF whiteLED
Wed Jun 17 20:24:38 SAST 2026 [elapsed=319 sec] NOTICES: det_dzdx=-885 is outside range +-250 from -558; det_dzdy=2314 is outside range +-250 from -1915; 

Preflight failed:
Filter changer insert/withdraw axis is reporting is;

Wed Jun 17 20:24:38 SAST 2026 [elapsed=319 sec] Preflight took 319 seconds to complete.


Wed Jun 17 20:24:38 SAST 2026 [elapsed=319 sec] STARTING CLEANUP
Turning off LEDs
doorpsw  3 OFF redLED
doorpsw  2 OFF whiteLED
Releasing TCS lock [preflight]
OK TCS lock released by preflight-STH

  
Azimuth check

Preflight image

Dome audio


Shutter audio

Shutter acutation data:

A  0.020s     3   0.0%
A  0.051s   107   1.2%
A  0.081s   376   4.1%
A  0.111s   724   7.9%
A  0.141s  1180  12.9%
A  0.172s  1762  19.2%
A  0.202s  2443  26.7%
A  0.232s  3238  35.4%
A  0.262s  4144  45.3%
A  0.293s  5074  55.4%
A  0.323s  5970  65.2%
A  0.353s  6753  73.8%
A  0.383s  7413  81.0%
A  0.413s  7990  87.3%
A  0.451s  8443  92.2%
A  0.481s  8846  96.6%
A  0.511s  9058  99.0%
A  0.541s  9146  99.9%
B  0.020s    -1  -0.0%
B  0.050s   104   1.1%
B  0.081s   368   4.0%
B  0.111s   712   7.8%
B  0.141s  1183  12.9%
B  0.172s  1751  19.1%
B  0.202s  2438  26.6%
B  0.232s  3241  35.4%
B  0.262s  4136  45.2%
B  0.292s  5071  55.4%
B  0.323s  5958  65.1%
B  0.353s  6748  73.7%
B  0.383s  7420  81.1%
B  0.413s  7985  87.2%
B  0.443s  8443  92.2%
B  0.474s  8785  96.0%
B  0.504s  9015  98.5%
B  0.534s  9135  99.8%
B  0.564s  9154 100.0%
sleeping 3 sec between shutter moves...
A  0.020s  9154 100.0%
A  0.050s  9044  98.8%
A  0.081s  8771  95.8%
A  0.111s  8410  91.9%
A  0.141s  7944  86.8%
A  0.171s  7363  80.4%
A  0.202s  6695  73.1%
A  0.232s  5898  64.4%
A  0.262s  4985  54.5%
A  0.292s  4042  44.2%
A  0.323s  3169  34.6%
A  0.353s  2385  26.1%
A  0.383s  1709  18.7%
A  0.413s  1149  12.6%
A  0.444s   693   7.6%
A  0.474s   357   3.9%
A  0.504s   126   1.4%
A  0.535s    13   0.1%
A  0.565s     2   0.0%
B  0.020s  9142  99.9%
B  0.050s  9011  98.4%
B  0.081s  8761  95.7%
B  0.111s  8408  91.9%
B  0.141s  7936  86.7%
B  0.171s  7358  80.4%
B  0.201s  6665  72.8%
B  0.232s  5891  64.4%
B  0.262s  4981  54.4%
B  0.292s  4054  44.3%
B  0.322s  3164  34.6%
B  0.352s  2391  26.1%
B  0.383s  1703  18.6%
B  0.413s  1147  12.5%
B  0.443s   700   7.6%
B  0.473s   357   3.9%
B  0.503s   138   1.5%
B  0.534s    15   0.2%
B  0.564s     0   0.0%
A     1  0.0mm   15 N F R | B     0  0.0mm   15 N F R