Spike 00 — Build PyMOL from this tree so import pymol works (macOS arm64)
Status: SUCCESS. PyMOL 3.2.0a0 builds and imports on darwin 24.6.0 / arm64, headless,
with no Qt, no PySide, no GLUT, no display, and with MMTF/BCIF support enabled.
Everything below was executed on this machine. Every transcript is real output, copied verbatim.
Follow-on doc: docs/build-and-tooling.md (the prior analysis this spike executes and
corrects).
STATUS — read this before §6.1
The recipe shipped.scripts/bootstrap.shIS this document, and cites it in six comments, by section: §1, §2.1 (whycatch2is deliberately absent), §2.2 (the vendored mmtf-cpp headers), §3, §3.1 twice (--no-build-isolation, and the keg-only libxml2) and §4.3 (copying_cmd*.sointo the source tree).scripts/dev-bridge.shcites §3 andscripts/doctor.mjscites §5.2’s 877-byte MMTF round trip. The venv it builds ispackages/bridge/.venv— substitute that for every<SCRATCH>/venv/bin/pythonbelow; the scratchpad interpreter in §0/§3/§8 is gone.§6.1 is AMENDED, not overturned — and getting this backwards costs a wave
“_cmd._draw()SEGFAULTS without a GL context” is true as stated but reads as “never call_draw”, and the architecture was written that way for a while. The precise rule, from04-picking.md§1 anddocs/code-ownership.md:103-107:§6.2 (C
_cmd._drawsegfaults only whenoptions.no_gui == 0(⇒HaveGUI = pmgui = 1) and no GL context is current — at theglGetStringonpackages/engine/layer5/PyMOL.cpp:2307.- With a current GL context it is not merely safe, it is mandatory.
ExecutiveDrawNowis the only caller ofOrthoExecDeferred, andPyMOL_GetIdleAndReadyonly becomes true afterPyMOL_Drawhas setDrawnFlagthree times. A bridge that never draws drains no deferred work at all: no clicks, no drags, no deferredcmd.png, no deferred ray.cmd.refresh()is not a substitute —CmdRefreshnever setsDrawnFlag.- §6.1’s own remedy paragraph (“the rasterisation path for a headless backend is
cmd.ray()+cmd.png()… not PyMOL’s GL renderer”) is therefore true only of a context-free bridge. The bridge that shipped creates a context first and drives_drawevery tick.exit()skipsatexitandPy_FinalizeEx) and §6.3 (_cmd._refreshdoes not exist) are unchanged and are cited fromtenmol_bridge/dispatch.py:354andpolicy/base.py:175. §7’s four recommendations are all done; their paths werewebclient/…and are corrected in place below.
0. TL;DR
Three findings that change the bridge design are in §6. Read those.
1. Machine state (measured, not assumed)
python3 on PATH is a pyenv shim (/Users/amirangel/.pyenv/shims/python3) and
/Users/amirangel/anaconda3 is also on PATH. setup.py:288-294 (is_conda_env()) changes
prefix search when sys.prefix looks conda-ish. Always build from the explicit venv below,
never from the ambient interpreter.
2. Native dependencies
2.1 Already installed via Homebrew — nothing had to be installed
brew install was NOT needed. If starting from a bare machine, the equivalent is:
catch2 is intentionally not installed: it is only used by --testing=true
(packages/engine/layerCTest/Test.h:14 wants the Catch2 v2 umbrella header <catch2/catch.hpp>, and brew
only ships v3). We do not build layerCTest.
2.2 The one dependency no package manager ships: mmtf-cpp
packages/engine/layer3/MoleculeExporter.cpp:16 does #include <mmtf.hpp>. It is header-only, it is not in
Homebrew, and it is not vendored in this tree (packages/engine/include/ has only pymol/ and tnt/).
Vendored out of tree (so packages/engine/include/ stays pristine — hard rule: do not touch upstream files):
setup.py scans each PREFIX_PATH entry for an packages/engine/include/ subdir (setup.py:783-796), so
adding …/deps/mmtf-cpp to PREFIX_PATH is enough — no -I hacking, no source edits.
This is a strict improvement on build-and-tooling.md’s recommended
--config-settings use-msgpackc=no workaround, which loses MMTF and BCIF I/O and causes
5 test errors. With the vendored headers we get full MMTF/BCIF and 4 of those 5 errors disappear
(§5).
3. The exact reproducible build
$SCRATCH/build-real-1.log):
3.1 Why each argument
3.2 Actual compile line produced (from the log, one object)
_PYMOL_NO_MSGPACKC is absent — msgpack is on. _PYMOL_NO_MAIN is present —
no GLUT main. 254 .o files (find build -name '*.o' \| wc -l → 254).
4. Errors hit, and how they were resolved
4.1 'mmtf.hpp' file not found — the only genuine build failure
Reproduced deliberately with the naive command (default use-msgpackc=guess, no mmtf-cpp on
PREFIX_PATH), i.e. what a new dev would type first:
options.use_msgpackc = "guess" (setup.py:198) → guess_msgpackc()
(setup.py:297-312) sees brew’s /opt/homebrew/include/msgpack/version_master.hpp with
MSGPACK_VERSION_MAJOR > 1 → returns "c++11" → _PYMOL_NO_MSGPACKC is not defined →
MoleculeExporter.cpp:15’s #ifndef _PYMOL_NO_MSGPACKC block is live → needs mmtf.hpp.
Resolution: vendor mmtf-cpp headers out of tree and put them on PREFIX_PATH (§2.2).
(The alternative, --config-settings use-msgpackc=no, also builds but costs MMTF and BCIF.)
4.2 Non-errors that look like errors
catch2missing — expected; only--testing=trueneeds it, and brew’s v3 would not work anyway (packages/engine/layerCTest/Test.h:14wants the v2 header). Not a blocker.build/appears in the repo (99 MB) —.gitignore:4already ignoresbuild, and.gitignore:3ignoresgenerated. Left in place so incremental rebuilds are ~1–2 s.packages/engine/modules/pymol.egg-info/appears in the repo and is NOT gitignored — this does dirtygit status. Removed manually after each build (§3 step 4). See §7 for the recommended permanent fix.packages/engine/testing/timings.tabappears after running the test suite — also not gitignored. Removed.
4.3 Why non-editable install
pip install -e . drops a 10 MB packages/engine/modules/pymol/_cmd.cpython-313-darwin.so into the source tree,
and .gitignore does not ignore *.so (.gitignore:1-5 is only *.pyc, *.d, generated,
build, .vscode). That violates “the repo must stay clean”. A plain pip install . copies
everything into site-packages and leaves packages/engine/modules/ untouched.
Consequence for downstream agents: edits to packages/engine/modules/pymol/*.py in the repo will NOT be picked
up until you re-run the install command in §3 (~16 s). If you need a live-editable PyMOL, use
-e and add packages/engine/modules/pymol/_cmd*.so + packages/engine/modules/pymol.egg-info/ to .git/info/exclude first
(that file is local-only and git never merges it).
5. Verification — real transcripts
5.1 The required one-liner
PYTHONUNBUFFERED=1 (or an explicit flush) that exact command prints NOTHING and still
exits 0. That is not a build defect — see §6.1. It is the single most likely thing to make
another agent believe the build is broken when it is not.
Note finish_launching’s docstring says “THIS IS NOT SUPPORTED ON macOS”
(packages/engine/modules/pymol/__init__.py:435). It nevertheless works here, in the -cq (no-GUI) mode. The
supported entry point for the bridge is pymol2.PyMOL(), verified next.
5.2 pymol2 headless, full pipeline including MMTF round-trip
5.3 pymol2.SingletonPyMOL and the _cmd bridge symbols
_cmd._refresh does not exist and never has. The complete set of underscore-prefixed
“raw hook” entry points in _cmd is exactly:
_cmd functions that take the _COb
handle: _cmd.refresh(_COb), _cmd.refresh_now(_COb), _cmd.refresh_later(_COb) — all present,
and used by packages/engine/modules/pymol/internal.py:551-558. The bridge should call cmd.refresh() /
cmd.refresh_now(), not a nonexistent _cmd._refresh.
Also newly surfaced and relevant to the bridge: _cmd._pushValidContext /
_cmd._popValidContext (not mentioned in any prior doc) — these are how PyMOL is told a GL
context is current.
5.4 Companion modules and data files
$PYMOL_DATA/shaders — that is where the @tenmol/viewer shader port should read them from.
setting_help.csv is there too (settings panel parity).
5.5 Full upstream test suite
testglTF— needs the externalcollada2gltfbinary; no Homebrew formula exists. Environmental, not a build defect. Only affectscmd.save('*.gltf').symop_py.test_commands— a ray-traced image byte-diff; ray-traced images are not bit-stable on darwin/arm64. Do not gate CI on image-diff tests on macOS.
build-and-tooling.md §2.6.5, which reported failures=1, errors=5 using
use-msgpackc=no. The 4 MMTF errors (testMMTF, testMMTFExportEmpty, testMMTFExportSele,
testSave_symmetry__mmtf) and the BCIF pytest failures are gone because we vendored
mmtf-cpp. That is the concrete payoff of §2.2.
Also correcting build-and-tooling.md §2.6.6: pymol -ckq … --run all returned EXIT=2, not 0,
when tests failed. Exit codes are not reliably meaningless — but still parse the output, don’t
trust the code.
5.6 Repo cleanliness
6. Findings that change the bridge design — READ THESE
6.1 _cmd._draw() SEGFAULTS without a GL context
This is the big one. p.button(), p.drag(), p.reshape(), p.idle(), p.getRedisplay() are
all safe headless. p.draw() is not.
~/Library/Logs/DiagnosticReports/Python-2026-07-30-173634.ips):
_cmd._draw /
SingletonPyMOL.draw(). The rasterisation path for a headless backend is cmd.ray() +
cmd.png() (verified working, 6 ms for 120×90 in §5.2), or geometry extraction — not PyMOL’s
GL renderer. Any design doc that proposes “call _draw and scrape the framebuffer” is proposing
a guaranteed segfault unless it first creates and makes-current a real GL context (and then
_cmd._pushValidContext). Note this crash takes down the whole process — there is no exception
to catch — so a single bad RPC would kill the user’s session.
6.2 PyMOL tears the process down with C exit(), skipping Python shutdown
atexit handlers do not run and buffered stdout is discarded:
END OF SCRIPT printed, so the script ran to completion — but 10 (buffered stdout) and
ATEXIT RAN were both lost. sys.stdout is not replaced (verified: sys.stdout is so → True);
the process is simply terminated from PyMOL’s thread via C exit() (packages/engine/layer5/main.cpp:221,
packages/engine/layer1/P.cpp:359/369/1488), which skips Py_FinalizeEx.
Implications: (a) the bridge daemon cannot rely on atexit or finally for cleanup —
flush/persist eagerly; (b) any log line the bridge writes must be flushed or written to an
unbuffered/line-buffered stream; (c) pymol.finish_launching is the wrong entry point for a
long-lived server — use pymol2.PyMOL(), which does not install that teardown path (§5.2 ran to
completion and returned control normally).
6.3 _cmd._refresh does not exist
Detailed in §5.3. Use cmd.refresh() / cmd.refresh_now() / cmd.refresh_later().
7. Recommendations for files I do not own
These are reported, not applied.-
.git/info/exclude(local, never merged, not a repo file) should gain:Without this, everypip installleavespackages/engine/modules/pymol.egg-info/as untracked andgit statusis never clean. Whoever owns dev-environment setup should do this inscripts/bootstrap.sh. -
scripts/bootstrap.sh(owner: build/tooling agent) should use--config-settings use-msgpackc=c++11with an out-of-tree vendoredmmtf-cpponPREFIX_PATH, notuse-msgpackc=no. Thenopath silently disables MMTF and BCIF file I/O, which are rows infeature-parity.md. -
docs/architecture.md(owner: architecture agent) — if it describes driving_cmd._drawfrom the bridge, that is unimplementable headless (§6.1) and needs revising. -
CI: do not gate on ray-traced image-diff tests on macOS/arm64 (§5.5), and do not use
--testing=trueon a brew-only mac (needs Catch2 v2).
8. Rebuild cheat-sheet
build-real-1.log (successful build),
build-default-fail.log (the mmtf.hpp failure), testsuite.log, draw-crash.log.