Extension-defined literals

Native formats in 0.75.0

GEDCOM is a native Standard (Lite) parser, inherited by Pro:

family = gedcom"""0 HEAD
1 GEDC
2 VERS 7.0
0 @I1@ INDI
1 NAME Ada /Example/
0 TRLR
"""

Do not import pkg:grid/gedcom: native tags are selected by the engine assembly and require no extension installation or WASM engine. The implementation is in grid-genealogy-core; see the native example and format catalog and edition contracts. PARSE_GEDCOM(text) invokes the same native reader. Keep GEDCOM level numbers at the start of each line. Native constants run without the parser service; recompilation needs the selected native binding. Native provenance acceptance is tracked separately from the extension contract below.

Package-defined tags

General-purpose extensions remain separate from the native format program. For an installed package that declares the non-built-in tag custom:

USE "pkg:example/records@1.0.0" EXPOSING (custom)
records = custom"payload"

custom"...", custom"""...""", and <custom>...</custom> select the same parser. Ordinary quotes decode Grid escapes. Triple quotes and angle blocks retain their payload verbatim. Angle blocks end at the first matching closing tag, ignoring case. Custom literals have no implicit interpolation. The explicit template tag parameter selects a separate template compiler when the package supplies one; see Reactive templates.

USE "pkg:example/records@1.0.0" AS records exposes records.custom. Installation alone does not activate a package tag. Conflicting imports and built-in/native tag replacements are refused. Compilation never downloads a missing package. Existing built-in literal semantics remain unchanged.

The builtin executable lua tag is provided by the runtime and needs no package import. It returns a checked scalar through extension execution; Executable language tags describes that separate invocation contract. Qualified imported tags retain this guide's explicit resolution rules.

Package literal compilation and distribution

Parsing, formatting, and syntax-tree inspection retain opaque literal source; they do not invoke the package. Explicit compilation invokes the installed, verified parser before semantic analysis and lowers the result to native constants. Compilation fails if the package or execution service is unavailable.

A compiled model runs without its literal parser package. Editing and recompiling the literal requires the exact package again. If the model also calls ordinary extension functions, those calls retain their normal runtime dependency. The compiled model carries inert parser and expansion provenance, including the manifest, artifact, source and result digests.

Expansion caches use the verified parser identity and decoded payload. The host currently retains the lazy parser service for one compilation; repeated payloads within that compilation share results. Embedders may retain an injected service across compilations. Results are never reused for a different admitted contract or artifact identity.

Package contract

For reactive templates, add "templateExport": "compile_template" to the literal declaration. This is a separate scalar export with the same pure, exact, capability-free, one-string contract as the constant parser. Its result signature is result_type(grid_extension_core::templates::plan_type()).

Declare format-specific delimiter defaults alongside the export:

{"name":"card","parserExport":"parse","templateExport":"compile_template",
 "templateDelimiters":{"open":"[[","close":"]]"}}

With this declaration, <card template>Hello [[Name]]</card> uses the package's defaults. The start and end tag attributes override either default for one literal. Without templateDelimiters, the defaults are {{ and }}. Both strings must contain 1–64 UTF-8 bytes and no control characters. Declaring defaults requires templateExport. This metadata is part of the manifest's identity and is available before parsing expression holes; reading it does not execute the package. Syntax inspection of imported implicit defaults needs the package declaration. Compilation refuses a syntax tree parsed using different defaults; reparse with the declaration or spell both attributes explicitly.

The input string is JSON: {"version":1,"chunks":["Hello ","!"]}. Slot zero lies between chunks zero and one. Expressions and delimiters are not sent to the package. For each request, return one TemplatePlan inside the normal values result envelope:

{
  "version": 1,
  "nodes": [
    {"kind":"constant","text":"\"Hello \"","slot":-1,"children":[],"keys":[]},
    {"kind":"slot","text":"","slot":0,"children":[],"keys":[]},
    {"kind":"constant","text":"\"!\"","slot":-1,"children":[],"keys":[]},
    {"kind":"text","text":"","slot":-1,"children":[0,1,2],"keys":[]}
  ],
  "root": 3
}

constant holds a JSON scalar; slot references a numbered expression; text concatenates children; array constructs a sequence; object pairs static keys with children; tag applies ordinary scalar TYPE_TAG semantics; document materializes a JSON-compatible native value with the document tag in text, including custom format tags. Both have one child. Document construction preserves arrays and records, propagates errors, and rejects values requiring an explicit projection, such as native trees. Every child precedes its parent, every non-root node has exactly one parent, and the root is the last node. Every input slot must appear. Repeat a slot by emitting another slot node, not by sharing an intermediate subtree. Unused fields must be empty, with slot set to -1 outside slot nodes.

Plans allow at most 4,096 slots, 128 construction levels, one million nodes, and 64 MiB of plan data. The existing package execution, source and result budgets apply as well. Unknown operations, cycles, detached nodes, missing slots, duplicate object keys and excess resources fail compilation. No node may emit Grid source or request an arbitrary runtime function. Template diagnostics are checked against the encoded request and currently reported at the enclosing template span.

Cache identity includes the admitted template export and encoded static request. Expansion provenance records package, artifact, request, and result digests. Formatting and syntax inspection never call either package export.

Manifest schema 3 adds an optional declaration:

"literals": [{ "name": "custom", "parserExport": "parse" }]

Schema 1 and 2 packages continue to work. Omitted/empty literal declarations do not change their manifest digest. Literal names are identifiers; qualification comes from USE, not from the package's public tag name.

Manifest schema 4 adds optional editor metadata to a literal:

"literals": [{
  "name": "custom",
  "parserExport": "parse",
  "editor": { "languageId": "gedcom", "tokensExport": "tokens" }
}]

languageId names an editor language the payload may be delegated to for highlighting; an id the editor does not know renders plain. tokensExport names a pure, exactly deterministic, capability-free export that takes the decoded payload and returns { version: 1, tokens: [{ start, end, category }] } with sorted, non-overlapping half-open UTF-8 byte spans and categories from the language service's semantic-token legend. Editor metadata never changes parsing, formatting or execution, and an omitted editor object leaves the manifest digest unchanged. A language server may run the admitted parserExport and tokensExport only in a trusted workspace, under the package's existing budgets; analysis in an untrusted workspace runs no guest.

The referenced scalar export must be pure and exactly deterministic, require no capabilities, and accept one required string. Its result signature uses the schema constructed by grid_extension_core::literals::result_type(T):

{
  version: 1,
  values: [parsedValue],
  diagnostics: [{ severity, code, message, start, end }]
}

values has exactly one item on success, including success with warnings; it is empty when an error diagnostic exists. start and end are half-open UTF-8 byte offsets into the decoded payload. They must lie on character boundaries. The compiler maps them through the enclosing Grid literal's escapes.

T can contain strings, booleans, numbers, integers, blanks through nullable fields, arrays and closed objects. Handles, Frames, and ambient capabilities are refused. Existing GRDX array encoding is version 1.2; generated native bindings support borrowed arrays of objects without introducing a guest ABI.

The existing runtime enforces fuel, timeout, memory and output budgets. Input is capped at 16 MiB and expansion at one million value nodes, subject to the stricter existing schema depth and export output limits. A module also caps cumulative expansions at one million nodes and 64 MiB of encoded native values. The compiler rechecks results from injected services. Execution errors fail compilation rather than leaving executable parser calls in the model. Embedders can inject CancellableLiteralCompiler; core-WASM execution observes cancellation on the existing engine epoch, and cancellation becomes a compilation diagnostic.

Native GEDCOM result

The native reader imports UTF-8 GEDCOM 5.5.1 and 7.0. family contains people, families, names, events, sources, citations, notes, links, nodes, edges, and the complete ordered records index. Each raw record retains its ID, parent, tag, value, original line and source offsets. Unknown/custom tags remain in that index even when they have no specialized projection.

Dates such as ABT 1900 remain expressions, not fabricated exact dates. Relationship edges retain the asserted GEDCOM kind and pedigree qualifier; families remain graph entities. No biological parentage is inferred. Isolated people remain in nodes. Dangling references produce warnings and retained links but no unusable graph edge. The warning highlights the original pointer, including in quoted literals. Up to 64 reference warnings appear individually. Above that limit the first 63 remain, followed by a warning with the number omitted. All authored links stay available for inspection.

Revision 2 accepts LF, CRLF, and CR record endings and keeps original byte offsets. In GEDCOM 7, @VOID@ explicitly records an absent target and produces no dangling-reference warning. A citation’s texts array retains each DATA.TEXT passage, continuations, media type and language in order. text exposes the first passage. Header SOUR identifies the producing software and stays in the raw records. GEDCOM 7 specification

The scope excludes GEDCOM export, legacy encoding conversion, GEDZIP, media retrieval and pedigree rendering. This is a record importer, not a claim of complete GEDCOM validation or byte-identical export.

Construct a native graph

The native reader emits one node per person and family, even when no edge references that entity. Every resolved relationship assertion contributes an edge. Family nodes retain multiple unions, and the pedigree property preserves explicit child relationship qualifiers. The status, adoption, and eventId fields retain the asserted status, adopting parent designation, and owning event record. Revision 3 projects family pointers under an individual’s birth (BIRT), christening (CHR), adoption (ADOP) and child-sealing ordinance (SLGC). For these event-scoped links, eventId identifies the row in events. The editor resolves these pointers within the same literal. Oppositely directed assertions are separate records; they do not imply a biological relationship.

GRAPH accepts a closed literal object with nodes and edges arrays of records. Node records use id; edge records use id, from, and to. Other scalar fields become properties. The constructor builds a directed graph. An empty edge array is valid, so a document containing only isolated people still produces a graph.

graph ancestry = GRAPH(gedcom"""0 HEAD
1 GEDC
2 VERS 7.0
0 @I1@ INDI
1 NAME Isolated /Example/
0 TRLR
""")
node_count = ancestry.NODE_COUNT()

Pass the literal directly to this constructor. Named or computed record objects are not constant-folded into a graph: that would discard their live dependency. The existing Frame/range constructor remains the route for changing sources. For record queries, bind the literal separately and use ordinary accessors, for example family.people[1].name. Repeated native payloads can reuse the frozen binding's cache when the parser contract and limits match.

Native literal compilation selects the linked reader on first use. A source containing only native literals does not initialize the package compiler. Rust embedders can return a request-owned cancellation flag from ModuleResolver::native_literal_cancellation. A cancelled native expansion reports GRID_PARSER_CANCELLED, including on a cached read. Separate compile requests can share the same cache and use different flags. Parser warnings keep their payload-relative byte spans so they map to each occurrence in source. A malformed parser response is rejected before cache insertion.

See examples/native-formats/gedcom-family.grid for the complete model. The older examples/extensions/gedcom directory is a historical WASM prototype.