Distributed model coordination (development preview)

Grid is developing a way for models on separate hosts to share live values, run Processes on another host, and serve one model to more than one membership. Today these capabilities exist only as development profiles. Each profile is selected explicitly when a host is set up, keeps its state in private local files rather than a public service, and makes no claim of production-grade custody, public hosting or measured performance. When no profile is selected, ordinary models, startup and evaluation are unchanged.

What the development profiles cover

• Coordination authority. A local authority records which participant owns each claim, can hand a claim to an enrolled successor, and returns its original answers after a restart or a loss of quorum.
• Value delivery. Participants exchange published values either in order (Ordered) or as the newest value (Latest) over private HTTPS. Deliveries carry signed receipts and recover after interruption. One publication can fan out to up to 1,024 destinations, and secure Latest and Ordered traffic each keep their own capacity.
• Shared memberships. One model can serve two independently certified memberships while each keeps its own output capacity and receipts. Leaving a membership completes only after its original delivery, publication, Process and grant obligations close.
• Remote Processes. A Process can run on another host through a direct service or an encrypted mailbox. Requests and outcomes survive lost replies and crashes, and the receiving host's grant accounting is preserved.
• Native missions. A native owner can quiesce its model and reserve a bounded mission allocation through the original authority. Retries preserve capacity, and expiry does not reopen the prior writer. The selected native Process profile consumes that allocation with the authority unreachable and preserves original outcomes and receipts through restart and hard process termination. New work stops at the signed limits. Authenticated return joins the original consumption, effect outcome and issuer settlement; unknown outcomes must first be reconciled by their existing owner. The selected local native profile passes its return/crash checks and 40 valid paired cost samples. Cross-node complete transfer remains separate.
• Never-issued mission recovery. If the original issuer proves that a reserved mission never received execution rights, it closes that reservation once and signs the retained closure. The quiesced native installation can then recover its complete original owners into the successor named at departure. Missing or issued reservations cannot use this path. Earlier uncertain effects and outstanding deliveries remain obligations of their original owners.
• Ledger/Process outcomes. The selected local profile joins a terminal Ledger submission with its original Process wait and execution-grant charge. Lost replies and cancellation can recover the original finalized effect while the Process remains stopped. Unknown effects keep capacity unavailable; receipt reconciliation never creates another submission or successful Source acceptance. Recipient surrender and issuer settlement remain separate original-owner steps. A stopped wait with durable proof of no dispatch can close its unused charge; any preexisting ledger reservation remains a separate domain obligation.
• Wallet observations. The selected native wallet reads exact Ledger money and execution-grant allocation, reservation, consumption and surrender from their original owners. Issuer settlement is inspected separately from the actual retained issuer record. Unknown outcomes keep execution units reserved; money cannot replenish them. Copy/Fork observations carry no executable grant admission. Native owner transfer and recovery retain the original history and predecessor fence. See the local wallet example.
• Archive wallet profile. A separate native development selection connects archive metadata and the five wallet@1.1.0 Process operations to the same custody, Ledger and grant owners. The selected local native checks pass through original receipt inspection, coherent archive snapshots, observed-only copied portfolios and cold inspection without the private signing key. The older installed-payment catalog keeps its ABI. A manifest grants no rights, a copied portfolio cannot spend, and a proposal does not submit a payment. No edition enables this profile automatically. Wallet metadata does not consume a metered workload; actual domain execution retains its original grant/result coupling.
• OPC UA source composition. An explicitly admitted numeric tag can feed an Ordered encrypted route through the original source journal and native model. Lost replies retain the same observation and publication. Good finite values are projected; unsupported quality or types refuse before capture. A Process reading the calculated observation requires its own execution grant. Unknown application or operation outcomes retain their original obligations. See the local OPC UA example.
• Local installation transfer. An explicitly selected private-file profile moves one complete native model installation between two local directories on the same filesystem. Model, Process, key custody, grant accounting and pending deliveries keep their original identities. Preparation retires the source; restart resumes the same transfer and only the verified destination can execute. An uncertain effect still holds its capacity. Existing Keychain-enrolled or governed installations require their own migration adapters.
• Cross-domain exports. A gateway passes a value from one domain to another only with separate disclosure and publication rights, and keeps the original receipts through recovery and revocation.
• Storage and custody. Authority state can be checkpointed and retired generations reclaimed. On Apple silicon, an opt-in profile keeps credentials in the macOS Keychain. The selected native model-owner profile preserves the existing model identity and delivery history through custody enrollment and renewal; locked or revoked custody refuses model bridge admission and signing. Cloud storage adapters for DynamoDB and S3 are prepared and tested locally, but no cloud service is installed.
• Selected peer recovery. An additional native authority capability can retain a bounded peer snapshot and retire its covered active replication prefix. An admitted voter can catch up while original receipts remain recoverable. This preserves the receiving voter's own identity and hard state; it does not delete historical outcomes or guarantee physical space savings. It is off by default, and full native realtime coexistence assessment remains in progress.

Limits

• Profiles are opt-in and development-only. A selected profile that cannot run refuses instead of falling back to a weaker mode.
• Most profiles carry finite numbers only.
• No public hosting, production custody, live cloud deployment or measured performance is claimed.
• gridctl exposes the routes a participant advertises, and nothing retries an uncertain operation automatically.

Related guides: Execution grants and Security and execution.

The selected local MQTT/Sparkplug profile connects one admitted numeric metric to Ordered encrypted model delivery through the existing source journal and native owners. Restart preserves journaled observations; Sparkplug birth aliases are rebuilt explicitly. Bad-quality death notifications retain their source debt and refuse scalar delivery. This does not promise replay of QoS-0 messages lost before local journal commit. See the Sparkplug embedding example.

The selected local Modbus TCP profile similarly joins one admitted numeric tag to encrypted Ordered model delivery. Recovery uses journaled observations and original receipts, without re-reading the device as a substitute. Device exceptions remain source debt and refuse scalar projection. The legacy device connection is unauthenticated; V3 authentication begins at the Grid participant. See the Modbus TCP embedding example.

The selected local industrial examples now also cover BACnet/IP and passive CAN/J1939. They preserve journaled observations through encrypted Ordered delivery, native model recovery and separately authorized Process reads. Confirmed-COV release, passive receive completion and Process outcomes remain separate original-owner transitions. External stacks/devices and production deployment remain separate; consult each linked investigation record for executed coverage and cost status.

An explicitly selected Unix private-file development profile now retains native issuer and recipient keys through rotation and revocation. Revoked or retired keys cannot start fresh work. Original operations can still reconcile and close against their retained grants and receipts; key rotation never creates capacity or changes an existing signed grant. Missing, replaced or weakened key storage refuses, including after restart. Production credentials and providers remain separate installations.

Native storage replacement keeps the full original owner bundle and retained handoff history, fences its predecessor before activating a successor, and preserves uncertain effects and outstanding delivery obligations. Active missions must settle and rejoin first; exhausted semantic epochs and unsupported storage formats refuse rather than reset ownership. Native embedders follow a separate enrollment, exact-retry and recovery procedure.

Native operator and evidence status

An explicitly selected local native embedding can inspect the original grant owner's exact capacity, unresolved operations and public installed-key status. Surrender still goes through its original fenced owner and refuses unresolved effects. The service transport retains exact original requests after reply loss; a refused retry does not make the earlier outcome known. In Network's Control plane, an original native document can explicitly open Document operators, read its existing selections, and inspect a Wallet allocation or Process operation. Opening a panel does not query an owner. Wallet surrender is explicit and guarded; Process inspection cannot resume or replace work. Notes and original selections survive navigation. A reply arriving after a panel closes leaves its observation uncertain until explicit inspection of the original.

The eight R1 presentations and Network navigation R3 are approved. The launcher also offers Consensus Authority, Authority Decision, Reference Bindings, Reference Inspector, Decision Panel and Cluster Memberships. Their scopes come from installed native owners or current certified document reference routes. Explicit controls retain the original signed operation or Process revision. Ledger money, recipient capacity and issuer settlement are separate observations; a missing issuer receipt remains unobserved. Service availability depends on that installation's admitted native owners. Trusted native setup can register borrowed DecisionOwner and WalletOwner handles alongside authority handles in Services::install, bound to the exact authenticated principal/document/model Scope, before constructing the Host with native_operator_services. The operator dispatcher uses those original loans as well as any existing document Process provider. Opening a panel never installs a Ledger, creates a Decision Process or authorizes a payment. The optional private GRID_BACKBONE_OPERATOR_CONFIG file reopens existing authority installations; it cannot describe new Ledger accounts or Decision workflows. The closeout ledger records the source and scope of the native integration checks.

Consensus inspection reports the installed fault/custody/storage profile and performs an explicit read barrier through the original Raft owner. A retained leader metric alone is not live quorum evidence; a failed barrier stays unknown. Decision inspection shows the original Open and applied workflow receipts, including retained policy, fixed roster, input provenance, ballots and human gate when those originals are present. These are historical applied observations, not a fresh global Decision read or permission to continue a workflow. The business approval threshold remains separate from the authority quorum.

Reference inspection can authenticate original retained source/disclosure records after a cold restart without reopening signing custody or activating new Registry epochs. It marks current authority unknown and withholds fresh controls until the current certified scope is available. Original target custody and document/model identity still apply; historical evidence never supplies fresh permission.

The Core-local lifecycle/recovery and paired-cost assessment is complete for the named native profiles. Records retain their exact source, machine and observation scope. The original CAN and storage overruns remain recorded alongside their separately measured passing corrections. A diagnostic assessment does not establish an optimized product budget or another platform's behavior. The bounded Linux native owner, application and accepted-state reader checks pass on the retained source in a local x86_64/Rosetta environment; they do not establish physical-x86 Performance. Optimized Mac timing and resource assessments retain their distinct providers and exact native source. Resource observations cover successful Rust allocation requests/requested bytes, native root high-water RSS and a conservative native cohort thread bound. Exact simultaneous thread peaks, total native/GPU heap, all logical locks/clones and hard hardware reservations are not established. Production providers, payment services, external SDK rollout and stronger or portable custody remain separate tracks.

Selected local diagnostics can count raw TLS stream bytes in both directions, including handshake, reply and retry traffic. The grid-backbone/wire-observation and native authority transport-observation capabilities are off by default and separate from timing observation. The Host forwards them with backbone-wire-observer. Blocking scopes belong to one transport thread; async authority counters belong to one installed transport lifetime. Reports must retain every relevant peer/lifetime and mark overflow or incomplete coverage unavailable. Count each socket's written bytes once for traffic amplification; adding both the sender's writes and receiver's reads double counts that traffic. Async poll counts are not syscall counts. These counters exclude IP/TCP headers, kernel retransmission and traffic outside the observed owners, and establish no physical network reservation or realtime acceptance.

Native shared memberships also accept a declared scalar-model admission ceiling of at most 65,536 counted operations/expression occurrences. Existing callers selecting 128 retain their limit. This admits no function, range opcode, descriptor, effect or cyclic/iterative work and grants no additional execution capacity. The 4,096-cell example passes the bounded same-model A/B native recovery journey, including all-cell checks, peer snapshot/purge composition and cold semantic reload. Numerical coexistence acceptance remains pending.

The same admitted membership model can follow verified native reference bookkeeping and capture commits. Its caller refreshes the current input/stamp admission at those original WAL boundaries; it does not recompile the model, replace its evaluator/producer or increase reservations. Stale capture evidence, an uncertain application, changed custody/fence or a replaced resident image refuses. Source capture/signing can run outside the model writer after the refresh-start boundary; the source and target retain their existing authority and provenance checks.

Selected two-membership callers can separate journal ingress and publication from the shared native writer using explicit tickets. Inputs, semantic state and the frozen original producer envelope share the existing native WAL. Recovered publication uses that same position/value even after another membership updates the model. A caller must release the Host lock around the journal phases; apply_next is a synchronous convenience. This bounded scalar profile refuses other selected document owners and unresolved applications without a matching native original. It adds neither another evaluator nor additional capacity; bounded native writer-stall/original-recovery checks pass, while sustained comparative assessment remains tracked in the implementation ledger.

The separate backbone-v3-model-fanout capability joins the native scalar model and original producer to a frozen recipient cohort. One recipient's receipt cannot release the full audience's source debt. Native retirement also retains the same semantic root's obligations and publisher fence. The capability is off by default; the complete topology and comparative assessment remain in progress. Private bulk mappings retain their separate carrier contract and cannot be substituted for scalar model inputs.

Selected private mappings can decode a separately certified numeric matrix using the existing GRDX dense format: finite, row-major f64 values, two dimensions, no missing values and at most 16,384 cells. Decryption must complete before any typed data is returned. This returns staged data; model input permission, geometry, atomic native application and retirement remain separate owners. The native bulk transport/writer join remains in progress. Scalar routes and opaque mappings do not acquire matrix semantics automatically.