•On Sol 54, TEGA was running a slightly tweaked version of the Checkout-1 block run on Sol 3 and Sol 28.

–Changed multiplier voltage AFTER checking all TAs. Never reachedthis command on Sol 54.

–Running under new EGA mode table

•Data rates were similar to previous mode table versions

•50 minutes into the run communication between the 386 and the S/C stopped

–TEGA had completed checking TAs 0 –4 and was partway through checking TA5

–All data was completely nominal right up to the last byte received

–No corrupted packets received

•20 seconds after the last heartbeat was received by the S/C declared a heartbeat loss and issued a TEGA RESET

•TEGA rebooted and reestablished communication with the S/C

•The S/C ran the TEGA safingblock

–S/C sent start TEGA FSW command

–TEGA closed all valves

–TEGA sent health check engineering showing TEGA was healthy

–S/C powered off TEGA and declared TEGA safed

Bus Current Timeline

Possible Causes

•TEGA stopped sending data

–DMA buffer lockout

–Interrupts disabled and not reenabled

–MASTER_RUN bit cleared

–386 software just went off into the weeds

–Current spike caused momentary brownout

•TEFSW stopped listening

–Telemetry task starvation

–?????

What It Wasn’t

•This was not a migration of the short circuit to a high-side location

–S/C power subsystem reported no anomalies

•This does not appear to be internal watchdog reset

–Timing appears inconsistent. We should have seen 2 sets of loader messages

–We have identified possible cases where this could be consistent

•Possible, but unlikely (20%?). Timing would have to be perfect

•This was not a repeat of the interrupt problem seen on Sol 47

–All data types stopped, Sol 47 issue would stop just one data type

–TA_DONE global variable was set by OVEN_CLOSE command. We have verified that this is sent from the background, NOT the interrupt.

Possible Causes

•DMA buffer lockout

–Would require that we somehow miss 2 S/C transmit interrupts

–This mechanism is used for every packet we send

•Interrupts disabled and not reenabled

–We often have to interrupt-protect background code

–We have reviewed the code we believe was executing at the time and found no suspects

Possible Causes

•MASTER_RUN bit cleared

–This bit is set once at the startup of the code, never touched after that

–386 FSW would have to be seriously compromised to write to this bit

–This bit is not SEU hardened (2MeV/mg/cm^2 –look it up)

•386 software just went off into the weeds

–Catch all

•Current spike caused momentary brownout

–S/C did not see any power anomalies

•Task Starvation

•TEGA has two tasks running on the S/C processor, command task and telemetry task. Command task is higher priority than telemetry task. Telemetry task receives data from instrument, including heartbeat messages, it increments a heartbeat counter upon receipt of heartbeat. Every 15 seconds Commanding task looks at heartbeat counter, if it has not incremented from last time it was looked at, TEGA is reset, then safed.

•If both tasks are starved, commanding task is activated first and if the heartbeat check is done, it could start the safingprocess even if the instrument had sent heartbeats during the starve period since the telemetry task had not yet been allowed to run.