← AI Foundry lab dashboard › Worklog
Covers Tue 6 Oct 2026 (week 41), 15:54–16:50 PDT: aifoundry2 in the AI Foundry lab at Studio 45. Written Tue 6 Oct 2026, about 16:45 PDT, from the lab's live collector (one record every 5 s), the test logs and photos of the BIOS screens.
Update, Fri 9 Oct 2026 (week 41): back in service. The stalling fan under the card was replaced by a two-fan 92 mm bracket. The idle card holds at 51 °C, and the heaviest workload settles at 75 °C instead of reaching the 90 °C stop: the 9 October page.
Update, Wed 7 Oct 2026 (week 41), 17:50 PDT: the card ran away again. Idle and untouched all day, it crept from 64 °C to 92 °C and fell off the PCIe bus at 16:46, at 138 °C and 133 W, the same failure as 2 October. A reset of its slot at 17:01 did not bring it back. The fans bought margin, not a cure.
Below: what happened, why it runs away, a simulator to try it, the model against the record, and why nothing on the card stops it. The 6 October report follows unchanged. The lab's own page for the day, with its data: What happened on 7 October.
7 October: the doom loop came back
Idle all day, aifoundry2 crept up while the other cards stayed flat
Mean die temperature, 5-minute means; nobody ran anything on aifoundry2 after Tue 16:35. Hover or tap for values.
Nothing in the card changed; its cooling did. At 03:55, with the host idle and nobody in the lab, the card's cooling got worse in one step: 64 °C became 78 °C by 05:00. A second, slower loss came in the afternoon, and by 15:30 the card sat at 89–90 °C, its tipping point. It crawled there for an hour, then ran away: 92 °C at 16:26, 100 °C at 16:39, 138 °C at 16:46:35.
The card's power matched the 2 October leakage curve within 0.2 W all day, so it made no extra heat. The room didn't change either: the other three cards in the lab and aifoundry2's own drive, network-chip and CPU sensors stayed flat. The BIOS fan setting can't have reverted, because the host never rebooted, and Linux never touched the fans. That leaves the airflow at the card: a fan near it slowing, stopping or knocked out of line would look exactly like this. Nobody can see the fans from Linux, so someone has to look on site.
Why it runs away: a loop with a tipping point
Hot silicon leaks more, and leaking heats it more. An idle card draws about 14 W that doesn't depend on temperature, plus leakage, the current that flows through transistors that are switched off. Leakage doubles every 22.5 °C. The air carries heat away in proportion to how much hotter the card is than the air. Two heat stores, the die and its heatsink, fit both runaways:
C2 dTh/dt = (T − Th)/R1 − (Th − Tair)/R2
P(T) = 14.4 W + 21.6 W · 2(T − 80 °C)/22.5 °C T is the die (the mean of 34 sensors), Th the heatsink. Fitted on the 2 October runaway: C1 = 78 J/°C, R1 = 0.88 °C/W, C2 = 858 J/°C, R2 = 0.21 °C/W. At rest, the total resistance R = R1 + R2 = 1.09 °C/W is what counts.
The card rests where the heating curve crosses the cooling line, and it can stop crossing. A line can cross an exponential twice. The lower crossing is where the card rests (64 °C on 6 October). The upper one is a point of no return, because above it heating outruns cooling. Warmer air, or less of it, slides the line toward the curve until the two crossings meet and vanish. That happens where the leakage's slope equals the cooling's, dP/dT = 1/R: at about 90 °C on the die and 30 W of leakage. Beyond it the card has nowhere to rest. Each degree of heating then brings more than a degree more, and the climb ends only when the card falls off the bus.
That is why the curve looks funny. It rises fast for ten minutes, then crawls for half an hour near 85–92 °C, where heating barely beats cooling: the ghost of a resting point that has just vanished. Then it explodes, because leakage feeds on itself: 100 °C to the bus drop took 7.1 minutes on 2 October and 6.7 on 7 October. Near the tip, small changes in the air move the card a long way. At 64 °C a degree warmer air raises the card 1.8 °C; at 90 °C it raises it without limit. So the margin that matters is how much warmer the air can get, not how cool the die looks.
The chip heats the same way on both days
Idle board power at each whole-degree die reading (median), 2 Oct and 6–7 Oct, and the one law fitted to all of it.
Simulate it
Drop the card at any temperature and watch where it goes
Pick a day, then move the sliders. Tap the left chart to start the card at that temperature. The model and its fitted constants are in the next section.
Heating against cooling
Board power the idle card draws at each die temperature (orange), and what the air carries away at that temperature (blue). Where they cross, the card can rest.
Die temperature over time
The model (blue); the measured run for that day (grey), where there is one.
More: the chip's leakage, a burst of work, how long to run
The cooling resistance is what the fans set: lower means more air through the heatsink. The burst starts at minute 5. The card falls off the PCIe bus near 138 °C, as it did both times; the model stops there.
The model against the record
Five checks, run on aifoundry2's live-collector records (one every 5 s) and the 6 October heavy run's 10 Hz telemetry. The searches ran on Modal, in parallel.
1. The chip heats the same way on both days
Fitted separately, the two runaways give the same law: leakage doubling every 22.3 °C (2 Oct) and 22.6 °C (7 Oct). One law fits all 18,414 idle records from 45 to 138 °C to 0.21 W, with a bias under 0.1 W on each day (the chart above). The heating side is the chip's own, and it didn't change.
2. Past the tipping point, both runaways are the same
Past the tipping point, both runaways run on the same clock
How fast the die heated at each temperature (median over 3 °C bins): 2 Oct from boot, 7 Oct from 13:30. Hover or tap.
Above about 95 °C the two days' heating rates lie on one curve: 2 °C/min at 104 °C, 7 at 116 °C, 16 at 124 °C. From 110 °C to the bus drop took 2.4 minutes on 2 October and 2.5 on 7 October. Once over the edge, the air no longer matters.
3. The model reproduces both days
Run forward from the first reading, with nothing measured fed in after it, the two-stage model fits the whole 2 October run to 0.50 °C rms and leaves the bus at minute 68.5, the observed minute. A one-stage model can't: the hour-long crawl runs on the heatsink's large heat capacity and the blow-up on the die's small one. For 7 October the card's constants were held at the 2 October values and only the air was fitted; the model tracks all 24 hours to 0.37 °C rms. Fitted on its own, 7 October gives the same card: a die store of 74 J/°C against 78, and 0.79 against 0.88 °C/W to the heatsink.
The air reaching the card warmed in steps until it hit the tipping value
Effective air temperature implied by the die reading and the board power (model resistance 1.09 °C/W). Hover or tap.
4. The margin, hour by hour, and what a forecast could have said
Fitting the air on data up to each moment only, then holding it at its last half-hour's mean:
| Data up to | The air could warm by | The model then said |
|---|---|---|
| 12:00 | +2.9 °C | it rests: no runaway |
| 14:00 | +1.5 °C | it rests: no runaway |
| 15:00 | +0.20 °C | it rests: no runaway |
| 16:00 | +0.07 °C | it rests: no runaway |
| 16:20 | −0.01 °C (at the edge) | it crawls at the edge for 3 hours or more |
| 16:30 | −0.23 °C (over the edge) | off the bus at 16:59 |
| 16:36 | −0.45 °C (over the edge) | off the bus at 16:48 |
| 16:40 | −0.58 °C (over the edge) | off the bus at 16:46 |
| what happened | off the bus at 16:46:35 |
The margin fell from 2.9 °C at noon to 0.2 °C at 15:00, and stayed within 0.2 °C of zero for an hour and a half. The model saw the card at the edge long before; it could name the minute only once the card was over it. That is the nature of a tipping point: the last tenth of a degree decides when.
5. A different kind of test: the 6 October heavy run
Driven by the heavy run's measured power (random data on every minion, 80 °C to the 90 °C cap in 37 s, then idle), with the card's constants from 2 October, the model tracks the die to 1.7 °C rms over the 4 minutes. It reads 89.1 °C at the cap (measured 89) and 74.8 °C three and a half minutes later (measured 75). The air it needs, 36.7 °C, matches the 36.5 °C that evening's idle readings imply. One prediction follows. With the fans as fixed on 6 October, an extra 40 W, about the heaviest all-minion load, held for more than about 1¾ minutes takes the card past its point of no return; 30 W, about 2½ minutes. The 90 °C cap on long runs stays necessary.
Why nothing on the card stops it: the firmware
Roman Shaposhnik, in the AI Foundry Discord: on 2 October, after that day's runaway, it maybe the same Firmware doom loop that @Gianluca identified long time ago
; on 6 October in #thermals, a bit of a doom-loop bug in firmware
. The same day a community member reported 100 °C on a card in a ten-year-old desktop, and 47 °C at most with a large case fan pointed at it: hot silicon leaks more current and it keeps feeding itself
.
The open firmware source shows why nothing on the card stops the loop. The cards run release 1.3.1 (service-processor firmware BL2 0.20.0, closest public source et-platform ffca4cbb4, May 2024):
| Guard | On the cards (1.3.1) | In the newest open source (836a4ab) |
|---|---|---|
| Thermal throttle above 65 °C | Runs on an idle card too, but it can only step the clock down, and an idle card is already at the bottom (600 MHz, 0.517 V). It loops every 0.4 s, changing nothing. | Skipped entirely while the card is idle (get_mm_state() != MM_STATE_IDLE, thermal_pwr_mgmt.c:880) |
| Emergency “safe state” at the PMIC's 75 °C or 75 W alarm (300 MHz) | An inverted test skips the clock change when the voltage lookup succeeds; the voltage change is commented out. Only a bookkeeping register moves. | Fixed in e024210b (5 Sep 2024, BL2 0.21.0 of 25 Sep 2024): really sets 300 MHz and a lower voltage |
| The 75 °C alarm's temperature | The PMIC's own sensor, which the firmware's comment says reports 0 | The same TODO: PMIC is currently reporting system temperature as 0 |
| Power-off when too hot | pmic_force_shutdown() exists, and nothing calls it | The same |
| The card's fan | The board's MAX6660, a temperature sensor and fan-speed regulator, is read only by a test command in the open PMIC firmware (aifoundry-org/et-boards, IoTask.c), and the handler for its over-temperature alert prints handleTEMP_ALERT_N not implemented | |
Both are true: the airflow set it off, and the firmware let it finish. The firmware adds no heat; the card drew exactly its leakage all day. What it lacks is a guard that cuts leakage on an idle card: lower voltage, power gating, or switching the card off. The upstream fixes of September 2024, one release after the cards' firmware, repair the emergency state. Even they leave an idle card unthrottled, and the 75 °C alarm reads a sensor that reports 0. On the way up, the board passes 75 W at about 114 °C, so a working 300 MHz state at a lower voltage might have caught both runaways there. Nobody has measured whether it would. No public issue or document addresses any of this. The ET-SoC-1 preliminary datasheet's ratings and package thermal sections each read This information will be included in a future release of this document.
What would stop it
- On site, before the card comes back: look at the fan on the card and the fans that blow across it. In the BIOS, Smart Fan 6 shows each fan's speed; on 6 October System 5 read 2,368 RPM. A fan aimed straight at the card adds the most margin.
- Then the full cold reset from 2 October: a warm reboot or a slot reset doesn't bring the card back, as 17:01 showed. Watch the first hours: idle, it should settle below about 70 °C. Above 85 °C idle, or 40 W of idle power, the margin is nearly gone.
- A warning on the dashboard: the margin can be computed live from the die reading and the board power, as in the simulator. Alarm well before it reaches zero.
- Firmware, on Requests for Roman: a release with the September 2024 safe-state fix, and a real idle trip, lowering the minion voltage or switching the card off at about 105 °C.
- Heavy jobs: keep the 90 °C cap. With the fans fixed, the model puts the point of no return under two minutes of full load.
- More fixes, ranked and checked, and the three machines' power management side by side (written after aifoundry2 failed to come back from the 18:16 power cut): power management on aifoundry1–3.
6 October: the fan fix (the original report)
The card is fixed. Idle, it now holds at 64 °C (27.4 W). On 2 October the same idle card climbed past 95 °C within the hour and fell off the PCIe bus at 138 °C. Under the 8-minute load test it reached 76 °C, well inside its limits. Only the heaviest load still drives it to its 90 °C cap within a minute, more slowly than on 21 September.
Idle: the same card now cools where it used to heat
Same idle card: it ran away on 2 Oct, and it holds steady now
aifoundry2's card, mean die temperature, from the moment it came back on the PCIe bus. Hover or tap for values.
- Five minutes after coming back on the bus, both runs read 64–65 °C at about 28 W. On 2 October the card was heating there by about 2 °C a minute; today it was cooling.
- At 64 °C the cooling now carries away all 27 W the card makes. On 2 October, by that day's heat fit, it carried away about 14 W at that temperature, and the rest heated the card.
- In that model's terms, the air reaching the card is now about 12 °C cooler (about 39 °C instead of 51 °C), and the idle runaway point moves from about 97 °C to about 122 °C.
- Today's trace starts at 67 °C because the card sat powered while the BIOS screens were open.
Under load: the test from aifoundry1's fan repair
The 8-minute load test: aifoundry2 stays well inside its limits
Mean die temperature after each burst of the same sgemm test (8 s on, 2.5 s off). aifoundry1's two cards ran it on 2 Oct, after card 0's fan was replaced.
- The same script as on aifoundry1 on 2 October: sgemm bursts, 8 s on and 2.5 s off, for 8 minutes, stopping at 85 °C (hottest 95 °C). It never stopped early, and every burst exited cleanly.
- The card went from 66 to 76 °C (33 W), still creeping up by about a degree every two minutes at the end; its hottest sensor peaked at 79 °C.
- aifoundry1's cards ran the same test at 54 °C (card 0) and 61 °C (card 1) at their warmest. So aifoundry2 runs warmer than aifoundry1's cards, but far from the runaway.
The spiky graph from 21 September
- That graph came from aifoundry2's long session on Mon 21 Sep. Each heavy run took the die from 80 °C to the 90 °C cap in 19–27 seconds, so the script stopped it, the card cooled, and the next run began. That's the sawtooth.
The run behind the spiky graph: 37 s to the 90 °C cap today, 19–27 s on 21 Sep
Mean die temperature from the moment a heavy run started (random data on every minion), same script and protocol: heat to 84 °C, idle to 80 °C, run up to 10 min, stop at 90 °C.
- Today the same run, from the same 80 °C start, reached the 90 °C cap after 37 seconds, against 19–27 seconds on 21 September, at the same power (about 71 W). The fans slow the spike to about half its speed but don't remove it: at full load on every minion, this card still needs the cap or short runs. It also recovers faster: three and a half minutes after the cap it was at 75 °C and still falling, where on 21 September it settled at 81–82 °C before the next run.
Reading the numbers
- "Die" is the mean of the 34 minion-shire sensors, read live. It is what every chart here shows.
- The "hottest" figure is a peak, not a live reading. The card reports its sensors' minimum and maximum since its statistics were last reset, and nothing has reset them since today's boot. So after the heavy run it reads 93 °C while the card idles in the 60s, and it will stay there until the next reset or reboot. (On 2 October it tracked the live temperature only because the card heated the whole time.)
What was done
- 15:54, the BIOS. Del at boot, Advanced Mode; F6 opens Smart Fan 6 (the lab notes said Smart Fan 5).
- 15:56, the cause on screen. Every fan header followed the CPU temperature (43 °C) on the Normal curve: 58% of full speed at 40 °C, 25% at 20 °C. The card's slot sensor (PCIEX16) read 57 °C, the hottest on the board. An idle host meant slow fans, while the idle card kept heating.
- About 16:00, the fix. CPU_FAN Speed Control set to Full Speed, then TUNE ALL to copy it to every header. Fan Stop stays Disabled and Control Mode stays Auto (at full speed the fan gets full power either way). System 5 read 2,368 RPM, and within five minutes PCIEX16 fell from 57 to 55 °C and PCIEX8 from 51 to 48 °C.
- About 16:03, saved. F10, save and exit; the machine booted (16:05). The card came up on the bus at 67 °C.
- About 16:06, a full-reset reboot, the one in the lab's card-behaviour notes that power-cycles the slot. The machine was up at 16:07.
- Checks. The card enumerated at 16 GT/s x8, the driver (et_soc1 0.20.0) loaded, the kernel logged no PCIe errors since boot, nobody held the card, and the live collector has read it every 5 s since 16:05.
What changes for people using the lab
- aifoundry2's card is back in service. The lab's dashboard links this page from its Worklog, under 6 October. Update 7 October: out of service again, after an idle runaway at 16:46 (above). Don't use it until its cooling is checked on site. Update 9 October: back in service, with a two-fan 92 mm bracket fitted at the card; it idles at about 51 °C at 23 W, and the heavy run settles at 75 °C (what was done on 9 October).
- Its fans now run at full speed all the time, so the machine is louder. If that's a problem, aifoundry3's setting is the fallback: Manual, with every point of the curve at 50% or more.
- The setting lives in the BIOS. A BIOS update or a CMOS reset puts the fans back on the Normal curve; set it again then. (Smart Fan 6's Save Fan Profile can keep a copy.)
Sources and method
- Today's records: the live collector on aifoundry2 (
~/live/history/2026-10-06.jsonl): mean minion-shire temperature ("die"), the hottest shire, board power; one record every 5 s. - 2 October's runaway and its heat fit:
docs/reports/data/2026-10-02-idle-runaway-aifoundry2/in et-soc1-prototyping (records.jsonl,fit.py: P(T) = 13.6 + 2.0·e^(0.030·T) W; R 0.90 °C/W; air 51.4 °C; time constant 4.4 min). Today's "air" and runaway point use that fit with today's steady temperature and power; they are model numbers, not measurements. - The load test:
burn.sh, the 2 October aifoundry1 test (docs/reports/data/2026-10-02-fan-fix-aifoundry1/) with stricter stops, run 16:22:25–16:30:30. - 21 September:
docs/reports/data/2026-09-21-horace-aifoundry2/long/(10 Hz telemetry, starts and ends of each run); the four heavy runs are runs 1, 4, 8 and 23. The graph is the one posted in #community-lab on Tue 22 Sep, 09:48 PDT. - References: aifoundry3's card and aifoundry1's two cards, idle medians over the 24 hours to 16:11 today, from each machine's live collector.
- 7 October: the live collectors of aifoundry2 (
~/live/history/2026-10-06.jsonl,2026-10-07.jsonl), aifoundry1 and aifoundry3; the 6 October heavy run at 10 Hz (~/claude/work/aifoundry2-fanfix-1006/long-1006/on aifoundry2). - The model: the leakage law fitted to 18,414 idle records (2, 6 and 7 Oct); the two-stage model simulated forward from the first reading and fitted by a global search (differential evolution), with a scan of the total resistance from 0.85 to 1.4 °C/W (good fits at 1.0–1.15, tipping points 87–94 °C); the 7 October air and the forecasts by gradient fits in JAX, the card held at the 2 October constants. These ran on Modal, in parallel. Air temperatures are the model's effective values: with the 0.90 °C/W fit above they read about 9 °C higher, and the margins barely change.
- Firmware: et-platform at
ffca4cbb4(closest to the cards' BL2 0.20.0) and836a4ab(newest),thermal_pwr_mgmt.c,bl2_pmic_controller.h, thedevice-bootloaderschangelog and commitse024210b,a3192849,60b40c10; et-boardsIoTask.c; the ET-SoC-1 Preliminary Datasheet Rev 1.0 (et-man). - The lab's 7 October record:
docs/reports/data/2026-10-07-idle-runaway-aifoundry2/in et-soc1-prototyping (commita66cd26), and its page. - Discord: the AI Foundry server,
#community-lab(2 Oct) and#thermals(6 Oct). - Photos: taken on the lab's monitor, 15:54–16:05, cropped.