Images, history, and live AI input

The upcoming image interface lets you ask about a chart, photograph, screenshot, or document page using the same AI setup as your workbook. It requires a new Grid build. Installing a model package alone does not add this interface to an older app.

Ask about an image

In Chat or Grid Agent, choose Add source → Images, paste an image, or drop it into the message box. Review the thumbnail, write your question, and send. Remove an attachment with its × button. Sent images remain visible with your message; retrying uses the same images. Device-local conversation history also retains images in unfinished drafts and sent messages, subject to its storage limits. Changing workbooks leaves that draft with its original conversation.

Choose a model that accepts images. OpenAI, Claude, and Vercel AI Gateway receive attachments through their existing configured provider; the selected model must support vision. Local models use the existing signed installation and workbook AI runtime. A text-only local package explains that an image-capable release is needed. Grid never substitutes a cloud provider for a local model.

PNG and JPEG are supported: up to eight images, 4 MiB per image, 8 MiB combined, and four million pixels per image. Resize larger images before attaching them. For a PDF, export the pages you need as images first. Audio and direct PDF input are not included in this interface.

Prepared local packages include Gemma 4 E2B Instruct 1.1.0 and Qwen 3 VL 4B Instruct 1.0.0. Their source packages live in grid-capabilities. They remain unsigned and unpublished, and their vision inference has not been run in Grid. Earlier text packages remain unchanged. Availability in the app depends on a compatible build and actual registry publication.

Show the Agent a window, camera, or video

Choose Add source beside the message box, then choose a source:

• Grid view attaches the visible Grid webview on macOS Desktop. It captures the calling window, including visible conversation content, not other apps or a hidden App layer. Native child browser and rendering surfaces may be absent from a webview snapshot; review the thumbnail before sending.
• Share window opens your browser's screen/window picker where supported. Choose what to share, then choose Attach frame when the preview shows the content you want. This control appears only where screen sharing is available.
• Camera opens a camera preview after permission is granted. Choose Attach frame to add a photograph from the preview. Availability depends on the browser or desktop webview and system camera permission.
• Video file opens a local recording supported by your browser. Play or seek to a moment, then choose Attach frame, or use Compare a time range to gather several moments together. Each attachment shows its clip position.

Review the attached images, write your question, and send. Previews are local; opening a source does not send a continuous feed to the model. Stop preview releases the source. Closing capture, hiding Chat, or changing conversations also releases it. A permission request that finishes after you leave is discarded. Attached frames remain ordinary conversation attachments, subject to Chat's existing local history limits.

Captured video frames keep their color and scale to at most 1920 pixels on the longest edge. Video sound is not included. This is analysis of selected images, not native audio/video inference, and it may miss events between selected frames. If window capture is unavailable, attach an operating-system screenshot using Add source → Images. There is no silent switch to another source or AI provider.

Compare part of a recording

Choose Add source → Video file, then open a recording supported by your browser. Under Compare a time range, enter From and To positions using seconds, m:ss, or h:mm:ss, and choose two to eight frames. For example, choose 0:10 to 0:30 and four frames to sample 12.5, 17.5, 22.5 and 27.5 seconds.

Choose Add range frames. Grid prepares evenly spaced color images inside the selected range without moving the visible preview. Progress shows the current frame; Cancel sampling, closing the picker, or stopping the preview cancels the operation. A failed or cancelled batch leaves your existing draft unchanged. Sampling does not send the recording or make model calls.

Review the thumbnails and their clip positions, close the picker, write your question, and send. For example: “What changed across these moments, and when did it first appear?” The same selected AI connection handles the image request. Grid respects the remaining image slots and attachment size allowance, reducing frame resolution when necessary. Existing attachments remain in the draft. The recording itself stays on your device; sent frames follow the normal conversation history limits.

These are selected images, not native video or audio inference. Sampling can miss events between frames. Clip positions are not calendar dates; Grid does not infer a recording date from a filename or file-modification time. Recordings without a reported duration can still use Attach frame. Keep the original recording for later analysis of other intervals.

Observe a live source

Choose Add source → Camera or Share window. Once the preview is ready, write a question under Observe this source, such as “What is changing near the entrance?”, then choose Start observation. This explicitly sends sampled images to the AI connection selected when the session starts. A vision-capable model is required. The session shows its model, call count, and latest answer's capture time.

Observation limits controls capture frequency (2, 5, or 10 seconds) and the call limit (6, 12, 24, or 32). Defaults are five seconds and twelve calls; every session also stops within ten minutes. Each request contains at most three recent frames within the shared 8 MiB limit, giving the model a small window for visual comparisons. One inference runs at a time. A newer pending window replaces an older one while the model is busy; a slow upload delays the next capture instead of building a queue. Images and answers must remain within a one-minute freshness window. A slow answer is rejected, and an answer that ages out is labeled Earlier observation.

Choose Stop observation to stop model calls while keeping the preview, or Stop preview to release the camera/window too. Closing the source picker, hiding Chat, or changing conversations stops observation. Starting again is explicit; errors and exhausted budgets never trigger automatic restarts. Cancellation cannot guarantee that an already-sent hosted request is unbilled.

For a connector or App image input, select its loaded value in Data or App, then choose Add source → Observe selected input. The producer must already be updating a grid-ai-input/1 image value with known capture times and distinct image identities. The picker rereads that selection; it does not start a device, fetch a network stream, or resolve another workbook value. Changing the selected input or its source ends observation. Unchanged images do not spend repeat model calls; an idle source ends the session. Numerical camera matrices still require a color-image adapter.

Answers and source/time metadata are available in Tasks → Activity, including after stopping. These observation runs are temporary: they do not resume after a runtime restart and do not save original images or recordings. This is read-only analysis; it does not change workbook values or execute Agent tools. Local video files use Attach frame or Compare a time range, because clip position alone does not establish a live capture time. Native audio/video understanding is not included.

Reuse a selected or earlier image

Add source → Selected images adds the images held by your current Data or App selection. For example, select the image input written by your camera connector, open Chat, then choose Selected images. That value must contain a grid-ai-input/1 observation object; a numerical camera matrix is not an image attachment. The picker reads the loaded selection without starting the connector or resolving another model value. If it has no images, load/select its input in Data or App first. Ordinary selected values continue to travel in the Agent's existing context.

Expand Earlier in this conversation to reuse a recent attachment. Up to 24 distinct images are shown, most recent messages first. Each retains its original source, capture time when known, and video position. Added dates describe arrival when capture time is unknown. Select the images you need, write your new question, and send. Re-selecting an image already attached does not use another slot; exceeding the shared limits leaves your draft intact.

These actions attach snapshots to your new question. They do not import a producer's earlier prompt or timing mode, subscribe to a live feed, search other conversations, or create a shared archive. Workbook historical/realtime requests still use the explicit timing contract below. Close the picker or press Escape to return to the message box controls.

Use the same input from a workbook or connector

The canonical workbook accepts an observation object through Grid's existing input API. PREDICT receives one grid-ai-input/1 object. Supply the input before requesting the output; until then the empty prompt cannot run inference. Keep PREDICT at the formula root so Grid can schedule its job. The object contains:

• A question (prompt) and ordered image observations (observations).
• Each observation's stable ID, source kind and source ID, arrival time, content hash, media type and encoded bytes.
• Capture time and sequence number when the producer actually knows them.
• A clip position (mediaTimeMs) for a frame from a recording, when known.
• A timing mode: snapshot, historical, or realtime.

Source kinds are app, connector, peripheral, and archive. These identify where data came from; they do not authorize access. The app or connector obtains authorized data first. The AI runtime accepts the supplied bytes, never a path or URL that it should fetch. It checks hashes and decodes image pixels at the AI boundary. Encoded image bytes are not inserted into generated formula source.

Connector and device authors use the same observation object with ordinary input writes; they do not need a second model API. A connector can submit a selected image this way. Supplying the object does not capture a camera image or automatically subscribe an existing connector to a model.

Analyze an earlier period

Use historical with startMs and endMs. Every image must have a known capture time inside that inclusive window. A photograph's file-modification time is not its capture time, so imported still images initially use snapshot. Likewise, a frame's position within a video is not a calendar timestamp. The video picker records clip position without inventing a capture date. An archive adapter can supply a known recording date and frame time for a historical query.

Keep original assets in the source's authorized archive. Retain the exact input object, asset hashes, model release, prompt and generation settings if you need to rerun an analysis. The timing checks use the requested historical period, not today's date. Grid does not silently replace an archived image with a live one. Repeating an LLM request is not a promise of identical wording.

The Chat Agent run record keeps attachment identities, times and hashes, not image bytes. This record alone cannot replay images; the source archive must retain them. The Chat UI retains attachments in device-local conversation history, not a shared media archive. Images have their own allowance: 4 MiB per image, 128 distinct images and 32 MiB per conversation, and 128 MiB across the local store. Conversation text has separate limits of 2 Mi characters each and 16 Mi characters total. Repeated images share storage; deleting the last saved conversation that uses an image removes its saved bytes. Export includes images. Chat shows save failures and offers export instead of silently dropping images. Keep original files separately for long-term replay.

Share images with managed tasks

Open Tasks to submit attached Chat images or select PNG/JPEG files in the task form. A draft/review team explicitly shares the original images with each member. Every selected model needs image support. Task details show source metadata, and new dependent tasks can explicitly reuse an existing image source. The API and CLI also accept historical windows and direct observationSource references. Select snapshot or a bounded historical window in the task plan.

Managed tasks retain original images in memory. If you select durable recovery, the private local checkpoint retains their exact bytes until you remove the run. These recovery files are not encrypted. Public task responses and activity show metadata only. Keep long-term originals in your authorized archive. This recovery option is separate from live sessions, which discard completed input windows.

Analyze live observations

Use realtime with maxAgeMs and an absolute deadlineAtMs. Capture timestamps must share the host's time reference. Unknown, future-dated, stale or expired observations fail explicitly. Grid checks freshness before inference, again after local model loading, and after the answer arrives. This bounds acceptable answer age; it does not guarantee a model will finish on time.

For live input, keep images in capture-time order within the eight-image, 8 MiB limit (base64 transport adds roughly one third). Discard the oldest images when full and remove stale images before submitting a snapshot. A delayed older image should not displace a newer image from a full window.

A live connector should allow one inference in flight and replace pending input with its newest bounded window. Do not queue every camera frame behind a slow model. Save originals separately when the application requires an archive. Apply a completed answer only to its matching observation generation, and show its capture time and deadline beside the result.

Workbook cells retain their completed values. A result is not automatically cleared when wall-clock time passes its deadline; the live consumer must check that deadline before acting or displaying it as current. AI descriptions are asynchronous observations, not a substitute for a deterministic control loop.

Add other modalities

A document adapter can emit selected page images with page IDs. A video adapter can emit a bounded sequence of timestamped frames through this image contract; that is sampled image analysis, not native video understanding. Audio needs a new typed segment contract and an engine/model that accepts it. Preserve event time, source identity, bounded input, retention choices and model admission as those adapters are added. Do not hide unsupported media inside a text prompt.

Connect app data and peripherals

Use structured model values for numbers, tables, selections and connector events; use images for visual information such as a chart, camera view or scanned page. The source chooses the relevant records or frames before handing them to AI. Avoid flattening every source into screenshots or sending an entire device stream with every question.

In the upcoming 0.66.8 interface, open Connectors → Camera and video and choose Images for AI under Use this source for. Choose an available camera, or enter a video file path or network stream address. Start the source; your current workbook stays open. In Chat → Add source → Connected sources, choose Attach latest image, write your question, and send with your existing image-capable AI connection. You can also open Connectors directly from this picker to set up a source. Open image workbook remains available in the connector controls when you want to inspect its input.

The picker lists sources without reading their images. Attaching reads only the chosen stored image; it does not evaluate workbook formulas or start capture. Stopped sources are labeled as last-image snapshots. Refresh updates the list. If capture restarts, refresh and wait for an image from the new session. Closing the picker cancels a pending read, leaving your draft intact. Images keep their original source and arrival time, including when attached more than once; the same image does not occupy another attachment slot.

Starting the connector captures images only; it does not call a model. The original Spreadsheet values option retains its numerical grayscale output. A workbook's declared source controls its output; open a new workbook when you want a separate image source. Only one camera/video connector runs at a time.

Image capture uses the installed FFmpeg decoder. It defaults to one frame per second within 960 × 540 pixels, preserving the source aspect ratio. Authored sources can select up to five frames per second within 1280 × 720 pixels. The producer keeps one latest pending frame and skips superseded frames while delivery is busy. Stop releases capture. A delivery failure or thirty seconds without frames stops the image source; correct the problem and start again. A finished recording ends without looping.

These connector images retain source identity, sequence, arrival time and a content hash. FFmpeg output does not establish the original sensor capture time or a recording date. Therefore use Connected sources or Selected images for snapshots; Observe selected input still requires a producer with known capture times. No original recording, searchable archive, audio track or native video inference is added. A selected image follows the normal conversation history rules after you send it. Source permissions and original recordings stay with the app or connector owner.