Emulating the Carrera040
The MicroMac Carrera040 is a 68040 accelerator for a 68k Macintosh’s PDS slot. Thanks to the Ethernet riser board by Bolle, it can be even be wedged into a SE/30.
The Carrera040 is a very strange board, and isn’t compatible with some things - like A/UX - which would be frankly hilarious to have running on the SE/30.
With a lot of digging and decompiling, it’s possible to get QEMU to emulate the Carrerra040 inside a SE/30 - to at least a first order of approximation - including some of the quirks and device-breaking bugs.
A huge thanks to ZZJ for advice and pointers along the way, his suggestions really helped narrow down the search and backed up a lot of the discoveries.
Most of the patches here center around the Carrera040 inside a SE/30, and expect a 32-bit-clean ROM. I use the BMOW ROM SIMM, but in theory it shouldn’t matter.
All this builds on the work of others reverse engineering the Carrera040 to work in the SE/30 in the first place. Be sure to check out the work at Geekdot for how the SE/30 starting point was even possible in the first place!
QEMU patches are available at the end.
I do software and hardware engineering and reverse engineering for a living, but some of this work was done with bot assistance - if that’s a deal breaker for you, I get it. Frankly, the amount of free time I have to chase fun projects like this is limited, so being able to optimize some of the stages is the difference between doing it at all, or it staying an amusing idea I’d never have a chance to chase down.
The Carrera040
Some accelerator cards completely replace or bypass the system CPU. The Carrera040 doesn’t. It adds a 68040 next to it and switches between the two:
- The motherboard 68030 always boots the machine. The ROM’s power-on code, including the Slot Manager scan, only ever runs on the 030 on real hardware.
- Once Mac OS is up, the Carrera INIT copies a small bring-up stub to physical address 0 and writes 1 to the card’s control register at
$53000000. That halts the 030 and releases the 68040 from reset, and the 040 starts from the reset vectors the stub left at 0 and 4. - CPU use can be toggled: Writing 5 and then 0 to the control register hands control back to the 030. Later writes of 1 resume the parked 040 where it stopped. Neither CPU is reset by a switch: the other one is just held off the bus.
- Reading the register returns the jumper block in bits 0-2. The driver accepts 5, 6 or 7, and with 6 or 7 it loads its FPSP for a full 68040. On an SE/30 without the card the read bus-errors, which is how the driver detects the card.
- The driver sets up its own 68040 MMU tables, an identity map with 8 KB pages, and relocates the vector table. That last part cuts MacsBug off, so the real machine has no usable debugger - at least, no usable MacsBug. Post-Carrera-INIT debuggers can still be installed, like TMON, and getting proper MacsBug may be a future patching effort.
- Control panel 1.8 has four modes: 030 SLOW, 040 MED (write-through pages, the shipped default), 040 FAST (copyback) and AUTO.
The 68040 has no dynamic bus sizing, but the SE/30’s devices are mostly 8-bit ports. Two Xilinx XC3000 FPGAs on the card fill the gap. An XC3020 carries the 16 data bits between the two sides. An XC3042 is the bus sizer: it splits each 68040 access into the byte or word cycles the 68030 side expects and steers each byte lane onto the right data pins. That second chip comes up again below.
Why emulate it
The target was A/UX 3.1 on a real SE/30 with a Carrera040, a BMOW 32-bit-clean ROM, 64 MB of RMA and a nuCF CompactFlash card.
Without a working MacsBug, all feedback from the real machine came as phone photos of the screen.
An emulator that ran the same configuration meant kernel and boot-loader patches could be tested before anything went near the real disk.
No existing emulator modelled the card (gee, wonder why), so it made sense to add it to QEMU. I’ve been using Snow as my primary emulator, but given how focused they are on accurate emulation, it seemed wrong to jank-patch in guesswork about how the Carrera040 worked.
The emulator
It’s QEMU 11.1.2 with six patches for the SE/30 itself and one for the card:
| Patch | What it adds |
|---|---|
| 0001 target/m68k | the 68030’s on-chip PMMU, 68881/68882 FSAVE frame sizes, one opcode decode table per CPU model, and a M68K_WATCHPC debug aid |
| 0002 mac_via, swim | classic PRAM inside XPRAM like the real RTC, the SE/30 ROM’s ADB sequence, the IWM status registers the floppy probe reads |
| 0003 hw/scsi | the NCR 5380 with the Mac’s pseudo-DMA, DRQ and IRQ through VIA2 |
| 0004 hw/display | SE/30 built-in video: 512x342 mono, two buffers, VBL |
| 0005 tests/fp | skips a test suite the emulator doesn’t need |
| 0006 hw/m68k/macse30 | the machine itself, -M macse30 |
| 0101 hw/m68k/macse30 | the Carrera040: -M macse30,carrera=on |
The nuCF CompactFlash card is a separate patch set on the same SE/30 machine.
Two CPUs in one machine
Stock QEMU builds one decode table for the whole m68k target. A Carrera040 system has a 68030 and a 68040 share a bus, and each needs the instructions only it has (PMOVE on the 030, MOVE16 and the 040 MMU instructions on the 040). Patch 0001 builds one decode table per CPU feature set, and adds the 68030 PMMU, which QEMU never had: table walks, transparent translation, PTEST and PLOAD.
-M macse30,carrera=on creates both CPUs and holds the 68040 halted. A write of 1 to $53000000 switches CPUs:
- The first time, the 040 is reset and loads its SSP and PC from physical 0 and 4, just like the real card coming out of reset. After that, switches only halt one CPU and wake the other, so each resumes with its own registers intact.
- The GLUE’s interrupt lines go only to whichever CPU holds the bus.
- A RESET instruction executed by the 030 (the ROM’s restart path) resets the card too. The 040 is held in reset again, and the next write of 1 cold-starts it as at power-on.
Parking the outgoing CPU on time
The first version of the switch let the outgoing CPU run on to the end of its translation block before halting. QEMU runs guest code in blocks, and the write to the control register sits in the middle of one. On the real card the outgoing CPU stops on the very next bus cycle. In QEMU it ran a few more instructions and reloaded a stale context. The fix stops it right after the write: it restores the precise guest PC, steps past the store instruction (the driver only uses four forms of it), and leaves the CPU loop at once.
The quirks
The calibration that divides by zero
Loading the “Startup Carrera” extension bombs with a divide by zero error… but only in the emulator.
Turns out the Carrera software times a DBRA loop to calibrate itself, and then divides by the result. QEMU doesn’t model bus timing, so the count comes out zero.
This can be bypassed by patching out the Carrera extensions calibration routines; tools/emu_carrera_calib.py IMAGE replaces the multiply and divide with a fixed result. It finds the sequence 32380d00c2fc0fa082c53001 and replaces it with 303c7e004e714e714e714e71. This is for the emulator only. Use it on throwaway copies of disk images, never on one meant for the real machine.
This is one of the few patches needed to the emulated system; perhaps I’ll be able to update the QEMU patches to provide bus timing, and eliminate that so it will boot a pure image in the future.
The write-steering hole
The “fun” discovery was that as soon as a patched A/UX was finally able to begin booting on the physical machine, it immediately pureed its disk. Glad this was all on BlueSCSI images!
Mac OS never demonstrated this problem on the Carrera040 (obviously, or we’d have known long, long ago).
Digging into the disk images, every fourth byte written to disk landed as $00. That’s … not optimal…
After much digging, it turns out that A/UX’s scsi_out writes to the 5380’s pseudo-DMA port four bytes at a time with move.l. Mac OS and the ROM only ever write bytes to the I/O window, so the Mac side never showed it. Changing the kernel to write byte by byte (patch_scsiwrite) made the corruption go away.
Digging in the Carrera040 Mac OS extension shows that they also patch this behavior automatically, so something is definitely up with the behavior of move.l.
Actually finding the cause required decoding the configuration PROM and the Carrera040 bitstream. This involved taunting the Basilisk enough to get it to decode the Xilinx streams, and probably would never have happened without bot assistance.
The PROM image holds the bitstream that loads the two FPGAs at power-up: 45,612 bits of Xilinx serial configuration, an XC3020 then an XC3042 in a daisy chain.
The bot ran Arisotura’s xc3000 bitstream viewer headless over the dump, turned every used CLB into equations, and stepped the XC3042’s sequencer through each access size against an 8-bit port:
long -> 8-bit port, A=0 byte0 | constant 0 | byte2 | byte3
word -> 8-bit port, A=0 byte0 | constant 0
word -> 8-bit port, A=2 byte2 | constant 0
byte -> 8-bit port correct for A=0..3
long -> 16-bit port correct
reads from 8-bit ports correct
The D31-24 output mux has no term for byte 1 in the second cycle of a long or word write, so the card drives zero there. A move.l to an 8-bit port loses D23-16, and a move.w loses its low byte. That’s the “every fourth byte” from the disk.
In the emulator: carrera-quirks=on is the default. Word and long writes from the 68040 to the I/O window ($50000000, all 8-bit) write their second byte as $00. An unpatched A/UX kernel corrupts its disk in QEMU exactly as on the real machine. -d guest_errors logs the first 32 such writes with their PC, which makes it easy to check whether other code runs into it. carrera-quirks=off gives an ideal bus sizer.
Any code that runs on the Carrera’s 68040 and writes words or longs to SE/30 I/O ports gets wrong data. Byte writes and all reads are fine.
What the emulator can’t show
Some faults only happen on the real card:
- FAST mode memory corruption. In copyback mode the real machine corrupts memory: the cursor image, and the 128 bytes just before a freshly copied block. QEMU has no 68040 cache model, so this doesn’t reproduce. MED mode is fine. This is still unresolved.
- Boot loader hang. A/UX Startup hung at “Loading…” on the real 040 because the Carrera’s
GetPhysicalnever returned. It was found with an NMI-triggered debugger INIT that draws machine state in a big hex font for photographing. It’s fixed with patches to A/UX Startup. - A kernel freeze in low memory. On the real machine the kernel stops partway through the Mac low-memory setup in
initMacEnvironment. The same code runs fine in QEMU. To compare, the machine has a debug watch (SE30_LOWWATCH=1) that logs every access the A/UX kernel makes to physical$2B0-$DD7on the 040. This turned out to be a Carrera bus quirk; see A/UX on the Carrera040.
Download
qemu-se30-carrera040.tar.gz contains:
patches/- the SE/30 machine (0001-0006) and the Carrera040 (0101), against QEMU v11.1.2build.sh- clones QEMU v11.1.2 (or uses a checkout given with--src), applies the patches and buildsqemu-system-m68ktools/-emu_carrera_calib.py, a 32-bit-mode PRAM image, andqserve.py/qc.py/qdrive.pyfor driving a headless session through the monitor (screenshots, typing, mouse)README.md- build requirements and the full option list
tar xzf qemu-se30-carrera040.tar.gz
cd qemu-se30-carrera040
./build.sh # headless; --gui for a display window
To also get the nuCF card, add its patch set. The two sets share the SE/30 patches, and build.sh applies each patch only once:
tar xzf qemu-se30-carrera040.tar.gz
tar xzf qemu-se30-nucf.tar.gz
cp qemu-se30-nucf/patches/*.patch qemu-se30-carrera040/patches/
cd qemu-se30-carrera040 && ./build.sh
Or, if one set is already built, run the other’s build.sh --src on the same QEMU checkout. Patches it already has are skipped.
You need to supply the ROMs yourself: an SE/30 ROM (or the BMOW ROM) for -bios, and the SE/30 video declaration ROM se30-video.rom in a directory given with -L. A typical run:
Run emu_carrera_calib.py on a copy of the disk image with the Carrera software first.
cp tools/pram32.bin pram.bin
qemu-build/qemu-system-m68k -M macse30,carrera=on -m 64 \
-bios bmow.rom -L romdir -display none \
-monitor unix:mon.sock,server,nowait \
-drive file=disk.hda,format=raw,if=scsi,unit=0 \
-drive if=mtd,format=raw,file=pram.bin \
-rtc base=1995-06-01T12:00:00
With both sets built, add nucf-rom=nucf.rom to the machine options and -drive if=ide,index=0,format=raw,file=cf.img for the card. In the monitor, cpu 1 selects the Carrera’s 68040 for info registers and x; CPU 0 is the motherboard 68030.