Skip to main content

Spike 08 — Native (C++) wave 2: change counters, pick data, missing instances

Status: IMPLEMENTED AND VERIFIED. Plan code-ownership.md §4 Task 6 (ReprVersion), §4 Task 3 (pick data) and the instance half of defect D6 are done. Every number, transcript and table below was produced on this machine by running the scripts named in §7. Nothing is inferred from source reading alone. Where something is not verified it says so in those words.

STATUS — re-verified against the C++ on 2026-08-02

packages/engine/layer4/CmdWebGeometry.cpp has been extended twice since this spike was written, so §0’s diffstat is stale. What is still exactly true, and what moved: STILL TRUE, re-measured today.
  • The merge-surface guarantee, to the line. Counting lines inside tenmol web client -- BEGIN … END blocks today: Executive.cpp 10 blocks / 64 lines, ExecutiveDef.h 1 block / 19 lines, Cmd.cpp 2 blocks / 10 lines = 93 guarded lines, zero unterminated blocks — the same three numbers §6 recorded. BEGIN and END counts match in every file. Zero upstream lines removed.
  • _cmd.web_get_versions payload shape (§2). Live keys today: changed, counters, objects, recomputed, rehashed, serial, walks; per object enabled, n_atom, n_state, reps, type, version. Exactly §2’s contract.
  • _cmd.web_resolve_pick payload shape (§3.1). Live keys today: atom, bond, bond_atoms, coord, describe, index, message, object, ok, pick_kind, selection, state, status. resolve(1tii, index=10, bond=cPickableAtom)status ok, selection u11, describe /u///HOH11/O, pick_kind atom.
  • §5.2’s status table. Re-run on 1tii with every rep shown: labels → unsupported; slice/volume/callback/cgonot-built; cellok from CoordSet::UnitCellCGO; extentok from CObject::ExtentMin/ExtentMax; ellipsoidsempty on a structure with no ANISOU; everything else ok.
CHANGED — §0 and §6 understate the file.
  • CmdWebGeometry.cpp is now 2,660 lines, not the “+1012” of §0 (spike 06 created it at 1,451). The three upstream files are untouched beyond the 93 guarded lines above, so the growth is entirely inside the file upstream does not have.
  • A whole object type was added that §5 does not mention: ObjectMesh — the isomesh/isodot OBJECTS the density wizard builds. They own no CoordSet and no Rep, so nothing in §5’s molecular path could reach them. There are now two arms: extractObjectMesh() reads ObjectMeshState::{N,V,VC} and reuses extractMesh’s kind: "mesh" envelope, adding source: "ObjectMeshState::V", level and map_name; and a matching arm in the object signature so web_get_versions lists a mesh|<state> row for them. Measured today on pept.pdb + map_new + isomesh/isodot:
  • Width is now hashed in TWO signature arms, and the second one is a bug this spike’s design would have shipped. §2.1 says the signature is a content hash of the rep’s CPU geometry — but mesh_width changes no vertex, so web_get_versions never bumped mesh|<state> and the client’s version poll, which gates every geometry pull, never refetched. Measured in the code comment at CmdWebGeometry.cpp:1776: Mode P went 22,780 → 36,444 ink pixels while Mode G stayed at 22,771 with geometryFrames stuck at 1. Both arms now hash it — repSignature’s cRepMesh arm hashes RepMesh::Width, and the ObjectMesh arm hashes the mesh_width / dot_width setting (an object-level setting, not state). Confirmed live today: set mesh_width, 3, msh takes mesh|0 from {version: 1} to {version: 2} with changed=True. This is the same class of hole §5.1 already recorded for dash_radius/dash_width; treat “a setting that changes appearance but no vertex” as the standing failure mode of a pure content hash.
  • §6’s mirror table lists one mirror; there are ten, all in namespace mirror at the top of the file: RepSurface, RepCartoon, RepCylBond, RepWireBond, RepRibbon, RepNonbonded, RepNonbondedSphere, RepEllipsoid, RepMesh (all from spike 06 §8.1) plus this spike’s RepDistLines. extractObjectMesh needs no mirror: ObjectMesh.h is a public header.
  • §7’s t*.py probe scripts lived in a session scratchpad and are gone. The behaviour they proved is pinned in packages/bridge/tests/ instead — test_p11_geom.py is the isomesh / ObjectMesh suite.
STALE PATHS IN THE C++, reported not applied (packages/engine/ is upstream and not this file’s to edit): CmdWebGeometry.cpp:8, :38, :2087 and :2421 still cite docs/webclient/spikes/…, the pre-monorepo path. The files are at docs/spikes/….

0. TL;DR

The CmdWebGeometry.cpp row below is the size at the time of this spike. The file is 2,660 lines today; the three upstream files are unchanged. See the STATUS block above.
GL-free scorecard. The backend needed a GL context for (a) Mode P rasterising and (b) the pick pass. This spike removes the need for (b): the geometry payload already carries stable (atom index, bond index) per vertex/instance, and §3 proves that resolving those client-side reproduces the backend pick exactly, on the same screen positions, for both an instance rep and a triangle-mesh rep. Nothing in web_get_versions or web_resolve_pick touches GL — both were run under pymol2.PyMOL() with no_gui=1 and no context at all.

1. What was added, and where

1.1 struct CExecutive (packages/engine/layer3/ExecutiveDef.h)

Four monotonic unsigned counters. They are hints, not content hashes:

1.2 Bump sites (packages/engine/layer3/Executive.cpp, 10 sentinel blocks, 0 lines removed)

The plan named five sites. Three of them are the right ones; two would have missed the single most common change of all. What actually landed, and why:

1.3 SettingRec::setChanged() was deliberately NOT patched

The plan offered it as optional. It is not needed and it is the worst place to touch: Setting.h is included by ~everything, so a field there is a full-tree rebuild at every upstream merge, and setChanged() fires for every setting including bg_rgb, ray_trace_mode and the movie panel — i.e. it would manufacture false wake-ups. Instead, the 113 ExecutiveInvalidateRep call sites inside packages/engine/layer1/Setting.cpp are what actually rebuild a rep, and the single bump at ExecutiveInvalidateRep covers all of them. Measured:

1.4 packages/engine/layer4/Cmd.cpp — 4 lines, both inside the sentinel blocks WP-26 already created


2. Task 6 — _cmd.web_get_versions()

A rep entry appears the first time it is ever active; when it goes away it stays in the map with active: false and a bumped version. That is precisely the signal D1 needs.

2.1 The counters are cheap and exact

counters alone is a hint. The versions are derived from a 64-bit FNV-1a content signature of each rep’s CPU geometry, computed only when a counter moved. The signature is deliberately pointer-free: Rep* is recycled by the allocator across a rebuild, and a CGO’s out-of-line floatdata blocks move, so hashing either would produce phantom changes. For CGO_DRAW_ARRAYS the signature hashes mode, arraybits, nverts and the out-of-line data block — never the pointer that holds it.

2.2 D1, reproduced and fixed

The bridge’s invalidation can now be: “for every (object, rep, state) whose version differs from the one I cached, refetch; for every one that is active: false, drop.” No content hash, no get_vis()/get_state() fingerprint, no self-declared inexactness.

2.3 No false positives — 700 idle polls, zero

Read-only commands do not fake a change (they can move the panel counter, which costs a walk, but changed stays false):
And a no-op recolour bumps a counter (work was queued) but not a version:

2.4 The change the old poll cannot see

get_vis() returns the same dict before and after color red, resi 1-20; get_state() is unchanged. This is the single row that makes a ReprVersion worth having, exactly as the plan says.

2.5 An optimisation that was measured, then REJECTED

Full disclosure, because it is the only place this design is knowingly slower than it could be. The panel counter moves on every SelectorTmp construction — count_atoms, iterate, select, get_model all bump it — so any of those costs one 47 ms walk on a structure that size. The obvious fix is to gate the expensive per-Rep hashing on counters 1..3 only (enable / name / rep) and let a panel-only move do just the cheap object-list pass. It was implemented, and then a differential harness killed it: two PyMOL instances, the identical 400-command random sequence, one polling gated and one polling force=1 (which always re-hashes).
Every divergence was create <name> onto an existing object name, whose atom count changed with no counter but the panel one moving. That is a D1-class stale-geometry hole, so the conservative rule stands: any counter moving triggers a full re-hash. rehashed is therefore always equal to recomputed; it is kept in the payload so the decision stays observable. With the fast path reverted, the same harness over 800 random commands:
One hardening did survive the revert: n_atom/n_state are now read and compared before the rep walk and are part of the object-level change test in their own right, because remove can drop an atom that no built rep was drawing, leaving every rep signature identical.
For the bridge: poll web_get_versions at whatever rate you like when idle (1.5 µs), but treat a recomputed: true as costing up to ~50 ms on a very large structure. That is the same moment the client is about to refetch geometry anyway.

3. Task 3 — pick data, and why the backend no longer needs a pick pass

3.1 What is shipped, and what is deliberately not

Not shipped: the pick colour. PickColorManager::colorNext (packages/engine/layer1/Picking.cpp:150-186) is a per-frame draw-order counter whose reverse map holds raw CObject* and is invalidated on every rebuild. It is meaningless outside the frame that produced it. Shipped: the identifiers behind CGO_PICK_COLOR (packages/engine/layer1/CGO.h:150-151) — a 0-based atom index inside the object plus a bond index or a cPickable_t sentinel, aligned per vertex and per instance with the geometry buffers WP-26 already emits (spheres.pick, cylinders.pick1/pick2, cones.pick, ellipsoids.pick, lines.pick1/pick2, crosses.pick, draw_arrays[].pick, and surface.atom from RepSurface::AT). _cmd.web_resolve_pick(_self._COb, object, index, bond, state=-1) turns one back into a selection, with no GL context:
u11is byte-for-byte whatpackages/engine/layer1/SceneMouse.cpp:245 builds for a real backend pick (”%s%d" % (obj->Name, index + 1)). Half-bond picks additionally get bond_atoms and a ready bond_selection:
Sentinels behave:

3.2 The round trip, against a REAL GL pick

Ground truth is a genuine ScenePicking.cpp pick-colour render + glReadPixels on the headless CGL + FBO context from spikes/picking.md §3 (GL: 2.1 Metal - 89.4 / Apple M4 Max), read out through _cmd.get_click_string. The client-side answer is produced in pure Python from only the web_get_rep_geometry buffers plus packages/viewport/src/camera.ts’s view maths — it never looks at the GL result.
18/18 and 15/15. They did not agree at first, and the two reasons are the important part of this section — a client that ignores either will disagree with the backend a few percent of the time.

3.3 Disagreement #1 — PyMOL’s pick has a ±7 px snap radius

A naive exact ray-sphere test scored 16/18. Both misses were pixels where the ray passed just outside the sphere the GL pick returned:
A pixel-centre offset sweep from −1.5 to +1.5 px in 0.25 px steps never reached 18/18, so it is not a convention error. The cause is in the source:
SceneRenderPickingSinglePick reads a 15×15 px window around the click and walks outward in square rings, taking the first non-zero pick index. So a PyMOL click snaps to anything within ~7 px (scaled by DIP2PIXEL). Porting that ring scan client-side:
For an isolated 16.18 px-radius sphere the silhouettes are otherwise identical to the pixel — a horizontal scan found the last GL hit and the last analytic hit both at +16 px.
For the WebGL agent: implement the outward ring scan with cRange = 7 * devicePixelRatio, in the same order (rings d = 0..6, row-major within a ring). Without it, clicks near a silhouette or on a thin rep will differ from Mode P.

3.4 Disagreement #2 — the pick pass is FLAT-shaded: use the LAST triangle corner

A naive “nearest barycentric corner” surface picker scored 10/15, and the disagreements were always a neighbouring atom (360 vs 361, 31 vs 32, 345 vs 346). The pick pass is flat-shaded: SceneSetupGLPicking does glShadeModel(GL_FLAT) (packages/engine/layer1/Scene.cpp:5186), the pick colour travels through gl_FrontColor precisely so that glShadeModel applies to it (packages/engine/data/shaders/default.vs:15-16, “using the built-in allows to use glShadeModel”), and ScenePicking.cpp:229-234 restores GL_SMOOTH afterwards with the comment “Picking changes the Shading model to GL_FLAT” (cSetting_pick_shading defaults to 0, so the restore is to GL_SMOOTH). A flat-shaded triangle takes the colour of its provoking vertex, which for GL_TRIANGLES is the last one. Measured over 72 GL picks on the 1UBQ surface:
third = 72/72. So the rule is: for a triangle hit, report atom[index[3*t + 2]], not the nearest corner. With that rule plus the ±7 px snap, the surface round trip is 15/15.

3.5 A finding the client must know: the backend cannot pick a surface by default

cSetting_pick_surface defaults to 0 (packages/engine/layer1/SettingInfo.h:812), and RepSurface.cpp:1565 does I->shaderCGO->no_pick = !pick_surface. Verified: with the default, clicking dead centre of an opaque 1UBQ surface returns type=none. The pick_surface, on setting had to be switched on to obtain any ground truth at all in §3.2. Client-side picking has no such limit — it can pick the surface whether or not the setting is on. That is a behavioural difference from Mode P, and the client should honour pick_surface to stay consistent, or the same click will select an atom in Mode G and nothing in Mode P.

3.6 What is still NOT verified

  • mesh, dots, ribbon, nonbonded and nb_spheres were probed for backend pickability at a single centre pixel and all returned type=none; that is most likely because those reps are sparse/thin and the pixel missed, not a proven “unpickable”. Not investigated further, and therefore not verified either way.
  • Multi-object and multi-state scenes were not part of the pick comparison. web_resolve_pick takes an explicit object name and state and was exercised with state=-1; other states are unverified.
  • Orthoscopic projection was not exercised in the pick comparison (the view was perspective, view[17] = -20). The ortho branch of the ray construction is unverified.

4. Task 3 (D6) — cone and ellipsoid instances

Both are decoded into flat typed instance buffers. Nothing is tessellated.

4.1 Cones

Note cone[2]: CGO_CYLINDER and every *_CYLINDER* variant land in the same cones bucket with r1 == r2. A client that implements the truncated-cone instance gets cylinders for free. Per-instance layout (cones bucket):

4.2 Ellipsoids

1EJG, 367 ANISOU records — the rep every text exporter drops (spike 03 §4.1: .wrl 234 bytes, .pov 0, .obj 0 faces, .dae 0 geometry nodes):
axes is the 3×3 from CGOSimpleEllipsoid (packages/engine/layer1/CGO.cpp:4535): three non-unit row vectors whose lengths are the semi-axes. Draw as a unit sphere transformed by that matrix, translated to xyzr[0:3]. xyzr[3] is the ellipsoid scale factor PyMOL passes alongside.
axes=3303 is 9 × 367 = 3303 floats. The buffer is correct; the odd-looking number is just 9 per instance.

5. The rest of D6 — audit of the previously unsupported reps

5.1 Now supported

  • cell comes out of CoordSet::UnitCellCGO, which is a CGO_DRAW_ARRAYS with mode = GL_LINES and 24 vertices (12 edges). It is not a lines bucket — the client reads it from draw_arrays, which it already supports. Measured on 1UBQ the box spans (0,0,0) .. (50.840, 42.770, 28.950) Å, i.e. it is the crystal cell with its origin corner at the model-space origin, and the extents equal cmd.get_symmetry()[0:3] exactly.
  • extent is synthesised from CObject::ExtentMin/ExtentMax, falling back to ExecutiveGetExtent because ObjectMolecule never latches CObject::ExtentFlag. 12 segments in the lines bucket. Verified equal to cmd.get_extent().
  • dashes / angles / dihedrals read the raw float* V of RepDistDash / RepAngle / RepDihedral (one mirror covers all three; RepAngle uses pymol::vla<float>, which holds exactly one float*, packages/engine/layer0/vla.h:42). The dash pattern is already baked into V by the rep builder, so the client draws the segments verbatim — no dash_gap arithmetic client-side. The strongest validator available is I->ds == ds; it must equal the DistSet the rep was reached through, or the call returns layout-mismatch.
  • cgo (standalone ObjectCGO) is harvested from ObjectCGO::State[s].origCGO.
One real trap, found by running it. RepDistDash::radius and ::linewidth are only assigned inside render() (packages/engine/layer2/RepDistDash.cpp:340, RepAngle.cpp:70, RepDihedral.cpp:75), so on a never-rendered rep — which is exactly the GL-free case this accessor exists for — they read 0. The accessor now reports the two settings those lines read from (dash_radius, dash_width) under radius/linewidth, and the raw members under rep_radius/rep_linewidth so a layout drift stays visible. The signature hashes the settings for the same reason, so set dash_radius bumps the version.

5.2 Full status table (1UBQ, one object, one state)

5.3 Still unsupported, and why — these are not laziness

  • labels — a label is not geometry. RepLabel holds text plus a cSetting_label_* font stack; drawing it means rasterising a font, and the on-screen result is a screen-space billboard whose size does not scale with the model. The right answer is DOM/canvas text in the client driven by cmd.get_model()’s label field, not a vertex buffer. Shipping a vertex buffer here would be actively wrong.
  • volume — a 3-D texture plus a transfer function. There is no CPU triangle geometry to extract at all; the whole rep is a ray-marching shader over a GL_TEXTURE_3D. Porting it means porting a volume renderer to WebGL2 and shipping the map, which is a work package of its own.
  • slice — a textured quad sampled from a map, regenerated per frame from the slice plane. Same problem as volume: the payload is a texture, not vertices.
  • callback — by definition arbitrary user C/Python code that issues GL calls at render time. There is nothing to serialise.
Everything else the plan listed is now supported.

6. Upstream-merge surface

The CmdWebGeometry.cpp line in this diffstat is historical (see STATUS). The three upstream rows and the 93-guarded-line audit below were re-run on 2026-08-02 and are exact.
Audited programmatically:
Zero lines removed from any upstream file, and every added line sits inside a tenmol web client -- BEGIN … END block. No build-file change was needed — setup.py:808-816 globs packages/engine/layer4/*.cpp. packages/engine/layer4/CmdWebGeometry.cpp gained one more layout mirror, which is the only thing that can drift: Run t3_instances.py at each merge; a layout-mismatch names the file that changed.

7. Reproducing

Build (unchanged from spikes/build.md):
Scripts, all in <scratch>/nat/: Test suites:
The single failure (symop_py.TestBondSymOp.test_commands) and the single error (exporting_py.TestExporting.testglTF) are byte-identical to the spike-00 baseline and predate this work. requests and biopython must be present in the venv or the suite reports 3–4 extra import errors that have nothing to do with these changes.

8. What this leaves for the other agents

  1. Bridge — replace the 4 Hz get_vis()+get_state()+content-hash fingerprint with web_get_versions(). Cache per (object, rep, state) version; refetch only on a bump; drop on active: false. That is D1, done properly.
  2. Bridge / protocol — plumb spheres.pick, cylinders.pick1/pick2, cones.pick, ellipsoids.pick, lines.pick1/pick2, crosses.pick, draw_arrays[].pick and surface.atom through the wire format; add a resolve_pick RPC over _cmd.web_resolve_pick.
  3. WebGL — add the cone and ellipsoid instance draws (packages/viewport/src/modeG/instances.ts currently lists both as “NOT DRAWN — reported as a fallback reason”). The buffer layouts are §4. Then implement picking with both rules from §3.3/§3.4 — the cRange = 7 outward ring scan and the last-triangle-corner convention — or clicks will disagree with Mode P near silhouettes.
  4. WebGLcell arrives as a draw_arrays block with mode = GL_LINES, not in the lines bucket; extent, dashes, angles, dihedrals all arrive in the lines bucket.
  5. Product — decide whether Mode G should honour pick_surface (§3.5). Client-side picking is strictly more capable than the backend here, which is a parity difference, not a parity gap.