Skip to main content

Spike 02 — Headless feedback capture

STATUS — re-verified on 2026-08-02: STILL TRUE, IN FULL

This is the spike that has aged best. §1’s two rules are implemented verbatim in packages/bridge/tenmol_bridge/engine.py, with this file’s reasoning in the comments beside them: options.no_gui = 0 at :125 (with -c explicitly rejected at :121-123), options.internal_gui = 0 / internal_feedback = 0 at :129-130, pymol2.SingletonPyMOL() at :142 under a comment that repeats §1 rule 2, and _install_pcatch() at :162. §9’s ordering trap — uninstall pcatch before p.stop(), because SingletonPyMOLGlobals is nulled first — is at :404-407 and cites “spike 02 §9”. §9’s FeedbackCapture was a reference implementation; the shipped code is tenmol_bridge/feedback.py + engine.py, not a copy of it. The behaviours §4 and §8 pin — destructive single-consumer read, None (not []) on a lock miss, the unbounded queue, the ~1018-char hard split, the line-prefix classification table — are the reason that module has the shape it has. Two citation fixes, made in place below: §11 items 1 and 2 pointed at an "action item (3)" of 00-build.md that has never existed; the finding they mean is 00-build.md §5.1/§5.3 and its recommendations are §7.
Status: BLOCKER RESOLVED. Full PyMOL console parity is achievable headless, with zero C++ changes, provided the bridge does two specific things. Both are non-obvious and both are the opposite of what the current build/architecture docs assume.

1. The answer in two rules

Rule 1 — never pass -c. OrthoFeedbackIn() is gated on G->Option->pmgui (packages/engine/layer1/Ortho.cpp:492-499). pmgui is set from !options.no_gui (packages/engine/layer1/P.cpp:1820), and -c sets no_gui=1 (packages/engine/modules/pymol/invocation.py:401). Start with no_gui = 0 and suppress the GUI by simply never creating a GL context and never calling _draw. Rule 2 — use pymol2.SingletonPyMOL, not pymol2.PyMOL. The pcatch stdout hook writes through the file-scope SingletonPyMOLGlobals pointer (packages/engine/layer1/P.cpp:2667). With a non-singleton instance that pointer is nullptr, so pcatch silently discards every print() — worse than not installing it. pmg_qt already does the right thing: packages/engine/modules/pmg_qt/pymol_gl_widget.py:3 is from pymol2 import SingletonPyMOL as PyMOL. HaveGUI is only assigned from pmgui inside PyMOL_DrawWithoutLock() (packages/engine/layer5/PyMOL.cpp:2244-2248), which a headless bridge never calls. So no_gui=0 costs nothing: ray, png, refresh, idle, scenes, movies and reshape all still work with no GL context (verified, §6).

2. Route-by-route results

Every row below was executed. “user-visible” = the exact text the Qt feedback pane shows.

2a. The -cq blocker, reproduced (e3_cq.py)

Text appears on the terminal; _get_feedback() is empty every single time. pcatch was installed here and did not help — the gate is upstream of it, in OrthoFeedbackIn.

2b. The working route (e2_pcatch.py)

Note the iterate line: the user’s print output (N ALA) is correctly interleaved between the prompt echo and the C-level summary. That interleaving is the reason to use pcatch rather than a separate Python-side tee — both streams pass through the same OrthoAddOutput line buffer.

2c. The non-singleton trap (e4_nonsingleton.py)

Those strings appear neither in the feedback queue nor on the terminal. They are dropped by PCatchWrite’s if(SingletonPyMOLGlobals) guard.

3. How the Qt GUI does it today

  • packages/engine/modules/pmg_qt/pymol_gl_widget.py:3from pymol2 import SingletonPyMOL as PyMOL
  • packages/engine/modules/pmg_qt/pymol_gl_widget.py:99-105self.pymol.start() then import pcatch; pcatch._install()
  • packages/engine/modules/pmg_qt/pymol_qt_gui.py:391-394 — a QTimer (single-shot, 100 ms initially)
  • packages/engine/modules/pmg_qt/pymol_qt_gui.py:941-958update_feedback(): feedback = self.cmd._get_feedback()colorprinting.text2html('\n'.join(feedback))browser.appendHtml(...)feedback_timer.start(500)
  • packages/engine/modules/pmg_qt/pymol_qt_gui.py:964 — after each typed command, feedback_timer.start(0) for an immediate drain.
Every part of this path is available headless. Nothing in it touches Qt, GL, or _draw. The web bridge is a 1:1 translation: replace QTimer with a poller thread / asyncio task, replace text2html + appendHtml with an SSE or WebSocket stream of JSON line records. colorprinting.error/warning/suggest are plain print in open source (packages/engine/modules/pymol/colorprinting.py:29-32), and colored_feedback reports " Setting-Warning: colored_feedback is not supported in Open-Source version of PyMOL" — so feedback lines contain no ANSI escapes and the React client needs no ANSI parser. (Belt and braces: OrthoFeedbackOut strips ANSI when colored_feedback is off, packages/engine/layer1/Ortho.cpp:502-516.)

4. Consume-once semantics — CONFIRMED DESTRUCTIVE

packages/engine/modules/pymol/internal.py:593-606 loops _cmd.get_feedback() until it returns empty; packages/engine/layer4/Cmd.cpp:3892 calls OrthoFeedbackOut, which does front() + pop() (packages/engine/layer1/Ortho.cpp:502-516). Measured (e5b.py):
Consequences the bridge design must honour:
  1. Exactly one consumer per process. If the HTTP handler and a background poller both call _get_feedback(), lines are split randomly between them. Route everything through one owner.
  2. PyMOL keeps no scrollback for you. (I->Line[] is a 256-entry ring, OrthoSaveLines 0xFF at packages/engine/layer1/Ortho.cpp:62, but it is not readable from Python.) The bridge must own the ring buffer so a reloading browser tab can replay history.
  3. The queue is unbounded until drained. 5000 cmd.do('print(...)') calls with no drain produced a single _get_feedback() returning 10000 lines with nothing dropped (e5_semantics.py). Good for correctness, but it is a memory leak if the poller ever stops. Keep polling.
  4. It can return None, not []. internal._get_feedback returns None when lock_attempt fails. Observed 1 None in ~100 polls while the main thread was rendering (e6_thread.py). if not fb: treats None and [] identically and is fine for skipping, but never do lines.extend(fb) without the None check.

5. cmd.feedback() interaction — a genuine trade-off

The terminal printf in OrthoNewLine (packages/engine/layer1/Ortho.cpp:1160-1169) is gated on Feedback(G, FB_Python, FB_Output). So is PCatchWrite (packages/engine/layer1/P.cpp:2668). Measured (e14_mute.py), after cmd.feedback("disable","python","output"):
and the terminal printed nothing at all.
  • Muting stops the duplicate echo on the launching terminal and keeps C-origin lines in the queue…
  • …but it kills Python-origin capture entirely.
Recommendation: leave python/output enabled and accept that the terminal PyMOL was launched from also shows the console text. For a local desktop-replacement that is a feature, not a bug. Per-module verbosity control (cmd.feedback("disable","executive","actions") etc.) works normally and affects the queue as expected — verified in e5_semantics.py.

6. no_gui=0 does not break headless operation

e9_pipeline.py / e10_opts.py, SingletonPyMOL with no_gui=0, no GL context anywhere:
Gotcha: with no_gui=0, internal_gui and internal_feedback default to 1 and PyMOL reserves screen real estate for its own overlay — reshape(800,600) yielded a viewport of (580, 582), not (800, 600). Setting internal_gui = 0 and internal_feedback = 0 before start() restores the exact viewport without affecting feedback capture (e10_opts.py):
OrthoFeedbackIn only reads pmgui; internal_feedback is used solely to decide whether to mark the ortho layer dirty (packages/engine/layer1/Ortho.cpp:1119-1122).
e8_fd.py — pipe over fd 1 + reader thread, under -cq:
It works, and it is the only route that works under -c. Reject it anyway: it needs a dedicated reader thread or the 64 KiB pipe buffer will deadlock PyMOL mid-render; it swallows the process’s real stdout so server logs become indistinguishable from console text; and it gives up the pmgui-gated queue’s clean line framing. Keep it in the back pocket only if a future PyMOL build must run with -c.

8. Behavioural details the web client must handle

All measured in e11_edge.py / e12_long.py / e13_classes.py.
  • Synchronous availability. 500 iterations of cmd.do(...) + immediate _get_feedback(): 0/500 drains missed their own line. On a single thread, feedback for a command is complete by the time cmd.do returns. No settling delay needed.
  • Cross-thread polling works. A poller thread drained 407 lines including all 200 T-* lines while the main thread ran fragment/show surface/ray; 1 None return (e6_thread.py).
  • Direct API calls are quiet; cmd.do is not. cmd.fragment('ala') produced [], while cmd.do('fragment ala') produced ' Executive: object "ala" created.'. Route user-typed commands through cmd.do() — that is where console parity lives. Programmatic calls made by the React UI (button clicks) should call the typed API and will correctly stay silent, or use cmd.do if you want them echoed like the Qt command line does.
  • echo=0 (cmd.do(x, echo=0)) suppresses the PyMOL> echo line but keeps the output.
  • Long lines are hard-split at ~1018 chars regardless of wrap_output (default off); OrthoLineLength fail-safe at packages/engine/layer1/Ortho.cpp:1097-1104. A 20000-char write became 20 feedback entries totalling exactly 20000 chars — no loss, but no 1:1 line mapping. Do not assume one feedback entry == one logical line for huge output.
  • Partial writes are buffered until a newline. sys.stdout.write("PARTIAL") yields []; the subsequent write("-CONTINUED\n") yields ['PARTIAL-CONTINUED'].
  • stderr is captured toopcatch._install() sets sys.stderr = sys.stdout = pcatch.
  • Full multi-line tracebacks come through, one feedback entry per traceback line, including the Python 3.13 ~~~~^^^^ caret lines (e6.out).
  • UTF-8 is fine: ['PyMOL>print("ångström Å 你好")', 'ångström Å 你好'].
  • Progress bars are NOT in the feedback stream. pmg_qt polls cmd.get_progress() separately (pymol_qt_gui.py:942update_progress). cmd.get_progress() returns -1.0 when idle. The web client needs its own progress channel.
  • cmd.log_open() is not a capture route. Verified contents for a .pml log: fragment ala\ncount_atoms all\nprint("logged?") — input echo only, zero output. .py logs give cmd.do('''fragment gly'''). Useful for session replay, useless for the console pane.

Observed line prefixes for client-side classification (e13.out)

Note the caret continuation lines ("( ( ( (<--", "nonexistent_object<--") that follow selection errors — they must be rendered with the preceding error line to be intelligible.

9. Deliverable — runnable capture module

Verified working end-to-end (§10). Drop this in the bridge package (owner’s choice of path; this spike creates no code files).

Wiring notes for the bridge owner

  • FeedbackCapture must be the only thing in the process that calls cmd._get_feedback().
  • Serve the console over SSE/WebSocket by looping wait_for(last_seq) and emitting the returned entries; the client sends back the highest seq it has, so a page reload replays cleanly from the ring buffer.
  • Use fc.server_stdout for uvicorn/logging output, or configure logging handlers before fc.start(); anything written to sys.stdout after start() lands in the PyMOL console pane.
  • fc.stop() restores sys.stdout before pymol.stop(). Reversing that order makes all subsequent prints disappear.

10. Verification transcript of the deliverable

test_feedback_capture.py, exit 0:
and on the real terminal, correctly not swallowed:

11. Required changes to other owners’ documents

These are reported, not applied — the files belong to other agents.
  1. docs/spikes/build.md and docs/build-and-tooling.md — the canonical smoke test pymol.finish_launching(['pymol','-cq']) and any bridge startup using -c/-cq must not be carried into the bridge. -c permanently disables console feedback. Keep -cq for CI smoke tests only.
  2. docs/spikes/build.md §5.1 and §5.3 (this item used to cite an “action item (3)” that does not exist in that file) — “must use pymol2.PyMOL(), not pymol.finish_launching()” is half right. It must be pymol2.SingletonPyMOL(). pymol2.PyMOL() breaks pcatch and silently destroys all Python console output.
  3. docs/architecture.md — the bridge must set no_gui=0, internal_gui=0, internal_feedback=0, external_gui=0 on pymol.invocation.options before start(). Setting them afterwards has no effect (options are copied into CPyMOLOptions at _cmd._new).
  4. docs/architecture.md / internal-gui.md — the console feature rows should record that _get_feedback() is a destructive single-consumer read and that scrollback, sequence numbers and replay are the bridge’s responsibility, not PyMOL’s.
  5. docs/internal-gui.md — progress reporting is a separate channel (cmd.get_progress()), not part of the feedback stream.
  6. Headless feedback capture — the blocker this spike was written to answer can be marked resolved; no C++ change required.