AI Scene Composition Pipeline

The AI pipeline is Yugma's core differentiator. It transforms natural language into structured 3D scenes through a multi-stage architecture.

Data Flow

User: "Build a sci-fi room with neon lighting"
  │
  ▼
[1] Reference Resolver (client)
    "that" → selected obj, "the red sphere" → id
  │
  ▼
[2] Spatial Preprocessor (client)
    "6 cubes in a circle" → precomputed [x,y,z] positions
  │
  ▼
[3] Style Fingerprint (client)
    Analyzes scene palette → "[STYLE] industrial metallic..."
  │
  ▼
[4] YSL Serializer (client)
    Scene → compact text at ~60 tokens/object (measured)
  │
  ▼
[5] Cloud Function: aiCompose / aiComposeStream (server)
    └── composeCore — the shared brain (agentic loop)
        ├── Default 3 iterations, clamped 0–5 (AI_MAX_ITERATIONS)
        ├── AI calls tools → SimScene executes them
        ├── Tool results (incl. created IDs) sent back
        └── AI sees results → calls more tools or stops
    (Session persisted to Firestore client-side, non-blocking)
  │
  ▼
[6] Tool Dispatch (client)
    Each tool call → Zustand store action → Three.js re-renders

Stage 1: Reference Resolution

File: src/utils/referenceResolver.ts

Resolves natural language references to scene object IDs before sending to the AI:

• "that" / "it" → currently selected object
• "the red sphere" → search by type + color
• "everything on the left" → filter by position

Stage 2: Spatial Preprocessor

File: src/utils/spatialPreprocessor.ts

LLMs hallucinate trigonometry. The preprocessor detects arrangement patterns and computes exact positions client-side:

Pattern	Trigger words	Computation
Circle	"in a circle", "ring", "circular"	x = r*cos(θ), z = r*sin(θ)
Grid	"in a grid", "3x3", "matrix"	Row/col with centered offset
Stack	"stacked", "tower", "pile"	Y increments
Spiral	"spiral", "helix"	Circular path + Y ramp
Line	"in a row", "line of"	X spacing, centered
Scatter	"scattered", "randomly placed"	Deterministic pseudo-random (LCG)

Positions are injected as a [SPATIAL_PREPROCESSOR] hint in the message — visible to AI but not to the user's chat history.

Handles NxM grid parsing (e.g., "3x3 grid" → 9 positions), strips radius N and spacing N from count detection to avoid misparse.

Stage 3: Style Fingerprint

File: src/utils/styleFingerprint.ts

Analyzes existing scene materials and produces a compact summary:

• Top 3 dominant colors (quantized bucketing)
• Average roughness, metalness, emissive intensity
• Heuristic style tag: neon, glass-heavy, metallic, industrial, natural wood, matte, minimalist

Injected as [STYLE] header in the system prompt so the AI matches existing aesthetics.

Stage 4: YSL Serializer

File: src/utils/aiSerializer.ts

The Yugma Scene Language (YSL) is a custom compact format designed for LLM context windows:

[★ hero_sphere] sphere id=abc pos=[2,0.5,0] rot=[0,0,0] s=[1,1,1]
  mat=#ff0000 r=0.3 m=0.8 e=#000000 ei=0 op=1 tags=[hero,accent]
  role=hero nextTo=[def] supports=[]

~60 tokens/object (measured) vs USD's ~262 (fair parametric) / ~850 (mesh) tokens/object. A 50-object scene fits in ~3,000 tokens.

Tier 3 output includes: id, name, type, position, rotation, scale, full material, tags, semantic role, and relationships.

Stage 5: The Agentic Loop (Server)

File: packages/yugma-functions/src/ai/composeCore.ts — the shared brain for both aiCompose (callable) and aiComposeStream (SSE).

The former two-pass planner (3D-GPT pattern) was removed in the composeCore refactor. There is no separate planning call — the agentic loop itself handles multi-step composition.

Executor (Tool Loop)

AI → tool_use blocks → executeToolSim(SimScene) → tool_result → AI → ...

• SimScene: lightweight in-memory object tracking IDs and positions (NOT Firestore)
• MAX_ITERATIONS: AI_MAX_ITERATIONS env var, default 3, clamped 0–5 (0 = single turn, no loop)
• Scene audit feedback: after each iteration, a pure-math validator (sceneAudit.ts) derives AABBs over the SimScene and appends a compact SCENE AUDIT: line to that iteration's tool results — heavy overlaps, floating/sunken objects, runaway distances, oversized props — so the model self-corrects on its next turn within the existing iteration budget
• 17 tools: add_object, update_object, remove_object, set_environment, clear_scene, animate_object, duplicate_object, align_objects, distribute_objects, focus_camera, apply_material_preset, search_select, set_tags, create_group, import_from_sketchfab, set_animation, play_animation — set_animation also accepts jointTargets (joint-space animation for kinematic models); play_animation plays embedded GLB clips

Dual-Write

After each successful turn, the client persists the session to Firestore via saveAISession (non-blocking, fire-and-forget). On next mount, the most recent session is loaded via loadLastSession.

Stage 6: Client Tool Dispatch

File: src/panels/AIPanel/index.tsx → TOOL_DISPATCH

Each tool call returned by aiCompose is dispatched to the appropriate Zustand store action:

Tool	Store	Action
add_object	useSceneStore	addObject() + updateObject()
update_object	useSceneStore	updateObject()
remove_object	useSceneStore	removeObject()
set_environment	useSceneStore	setEnvironment()
focus_camera	useSceneStore	setCameraTarget() (smooth tween)
animate_object	useAnimationStore	addKeyframe()
apply_material_preset	useSceneStore	lookup MATERIAL_PRESETS → updateObject()
create_group	useSceneStore	addObject('box', invisible) + reparentObject()

Configuration

Setting	Value	Notes
Model	gemini-3.5-flash (default)	Default provider + model are server-configured (Admin → Providers, sysconfig/providers); cerebras:* / claude-* remain as server-side overrides. No user-facing model picker (removed 2026-07-08)
Max tokens	AI_MAX_TOKENS env, default 8192	LLM completion budget
Gemini thinking	GEMINI_THINKING_LEVEL env, default medium	low / medium / high; any other value (e.g. off) omits thinkingConfig — required for gemini-2.5-* model overrides
Max iterations	AI_MAX_ITERATIONS env, default 3 (clamp 0–5)	Agentic loop cap
Temperature	0.7 (creative) / 0.2 (precise)	User toggle
Rate limit	30 req/hr per user	Firestore counter, clientRequestId dedupe
Function timeout	180s (aiCompose) / 300s (aiComposeStream)	Accommodates the agentic loop