Live snapshot of the first ever WallSpace.Studio Liver performance with the app at Gray Area Noise Pop show in SF

Continuation of https://app.daydream.live/creators/AEYESTUDIOS/crt-wall-manager project page focused on AI Video Cohort 3. Will be posting daily releases here for beta group members as well as release notes on new features and bug fixes. What's New

Current Latest Build:

3/11/2026

https://github.com/jackmo650/crt-wall-controller/releases/tag/v2.2.8

WallSpace.Studio v2.2.8

Major release introducing Metal GPU compute effects, live audio transcription via whisper.cpp, deep architecture refactoring (R3 Domain Contexts), a new Scope effects node plugin, and significant UI cleanup across the application.

Metal GPU Compute Effects (webgpuEffects.ts)

• New WebGPU compute shader pipeline for real-time per-pixel effects using Apple Metal backend
• Two rendering APIs: Canvas API (async, for preview) and Texture API (sync, for compositor pipeline)
• 16 GPU-accelerated effects: chromakey, edge detect, threshold, invert, RGB gain, scanlines, brightness, contrast, saturation, blur, glow, color quantize, Canny, lineart, softedge, depth
• Effects run entirely on-GPU with zero CPU readback — chained via the WebGPU compositor ping-pong buffer system
• Automatic device/pipeline caching with context-loss recovery
• Integrated into the existing effects.ts and compositor.ts pipelines

Live Audio Transcription System

Whisper Bridge (whisperBridge.ts)

• New main-process child process manager for whisper.cpp (C++ inference)
• Auto-downloads Whisper models from HuggingFace on first use: tiny.en (~75MB), base.en (~142MB), small.en (~466MB)
• Manages whisper.cpp subprocess lifecycle (spawn, kill, restart)
• PCM → WAV conversion for whisper.cpp input
• Relays transcripts to renderer via IPC bridge

Transcription Service (transcriptionService.ts)

• AudioWorklet-based PCM capture — avoids MediaRecorder crashes in Electron
• Captures Float32 PCM from any audio source, resamples to 16kHz mono
• Configurable chunk duration and accumulation window
• Sends audio chunks to Whisper via IPC, receives transcript text back
• Three audio source modes: system audio (loopback), microphone, layer audio

Caption Renderer (captionRenderer.ts)

• Renders live transcript text onto a cached HTMLCanvasElement (compositable as a layer source)
• Full styling support: font family/size, text color, outline, background, positioning
• Word-wrap with configurable maxLines limit
• Performance-optimized: only re-renders when text or style actually changes

macOS Audio Permissions

• Info.plist patched at postinstall for NSAudioCaptureUsageDescription, NSMicrophoneUsageDescription, NSScreenCaptureUsageDescription
• Electron builder extendInfo config for production builds

Architecture: R3 Domain Contexts Refactoring

Major refactoring of App.tsx (reduced from ~8,471 to ~7,466 lines) by extracting state into 4 React domain contexts:

SelectionContext

• Tracks user selections across all panels (CRTs, scenes, layers, slices, projectors)
• Separated from scene data to avoid unnecessary re-renders during high-frequency selection changes
• Manages drag state for slices and corner pins

SceneDataContext

• Single source of truth for all persistent scene graph data
• Owns: spaces, walls, CRTs, scenes, projectors, active IDs, canvas/wall config
• Provides derived values: activeSpace, activeWall, layers, enabledLayers, etc.
• initialState.ts extracted for independent provider initialization

OutputContext

• Manages output window state, display configurations, compositor refs
• Absorbs useOutputState hook into a shareable context
• Owns canvas refs, quality settings, output window lifecycle

ScopeContext

• Manages Scope/Daydream AI pipeline state
• RunPod GPU state, plugin serverless service configuration
• Phase A: state variables and refs in context; complex handlers remain in App.tsx hooks

Provider tree: SceneDataProvider > SelectionProvider > OutputProvider > ScopeProvider > App20 components now consume contexts directly via hooks (useSelection, useSceneData, useOutput, useScopeContext), eliminating deep prop drilling.

Scope WallSpace Effects Node (New Plugin)

• New standalone Scope plugin: scope-wallspace-effects
• GPU-accelerated effects deployable as preprocessor or postprocessor in the Daydream Scope pipeline
• Two pipeline configurations: wallspace-effects-pre and wallspace-effects-post
• 16+ real-time effects powered by PyTorch: brightness, contrast, saturation, chromakey, edge detect, blur, glow, scanlines, Canny, lineart, softedge, depth, color quantize, and more
• Python 3.12+, no additional model downloads required
• Complementary scope-wallspace-upscaler plugin for AI upscaling in the Scope pipeline

Centralized Logging System (from v2.2.7+)

• LogAggregator in main process: ring buffer, JSONL file output, port registry, IPC push
• createLogger(source, service) for main process, createAppLogger(source, service) for renderer
• Console interceptor captures React warnings + unmigrated console calls → Logs panel
• Port registry with 60s orphan detection (PID-based cleanup)
• Scope Local server auto-stops when browser window closes
• 459 console calls migrated across 13 files
• LogPanel in Zone 2 "Logs" tab with filter tabs (All/Errors/Ports/Scope/Perf)

Compositor & Rendering Improvements

• Compositor pipeline updated to route effects through WebGPU compute when available, with Canvas2D fallback
• perspectiveWarp.ts: WebGL context caching with size-based invalidation and context-loss detection (fixes WebGL context exhaustion at 30fps)
• crtHelpers.ts: Updated slice rendering and batch CRT optimizations
• webgpuCompositor.ts: Ping-pong buffer refinements for effect chaining
• Performance template types updated for new GPU profiling metrics

UI Cleanup & Component Updates

• LayerEditor: +188 lines — transcription controls UI, caption layer source type, audio source selection, Whisper model picker
• HierarchyItemRow: New caption/transcription indicators in the hierarchy tree
• BatchCRTEditPanel / CRTEditPanel / LayerListPanel: Minor type alignment fixes
• WebRTC Import Service: Extended for audio stream capture alongside video
• Audio Engine: +106 lines — system audio loopback capture, AudioWorklet integration, PCM extraction for transcription
• ControlNet Sync: Extended sync capabilities for new effect parameters
• Plugin params types: +80 lines — new effect parameter definitions for Scope effects node

Type System Updates

• pluginParams.ts: New effect parameter interfaces for all 16 GPU effects
• scope.ts: +36 lines — Scope pipeline types for effects node integration
• electron.d.ts: +13 lines — IPC type definitions for Whisper bridge
• gpuConsole.ts: Updated GPU performance metric types
• index.ts: Extended layer/source types for caption layers

Testing & Tooling

• Storybook: Vitest setup for portable story testing (.storybook/vitest.setup.ts)
• Vitest shims: Global type shims for test environment (vitest.shims.d.ts)
• Render profiling infrastructure from R3 refactoring

Previous Builds

3/10/2025

https://github.com/jackmo650/crt-wall-controller/releases/tag/v2.2.7

v2.2.7 — Architecture Refactoring, GPU Pipeline, WebCodecs Transport & UI Polish

Latest jackmo650released this2 minutes ago v2.2.7 c246543

v2.2.7 — Architecture Refactoring, GPU Pipeline, WebCodecs Transport & UI Polish

This release represents a major multi-session effort using Claude Code (everything-claude-code workflow) to systematically refactor the application architecture, optimize the GPU rendering pipeline, introduce hardware-accelerated frame transport, and polish the canvas UI.

Architecture Refactoring (R1/R2)

Significant structural improvements to the monolithic App.tsx, reducing it from 8,770 → 8,143 lines (−627 lines, −7.2%):

• IPC Bridge (utils/ipcBridge.ts): 48 fire-and-forget window.electronAPI.* calls migrated to a typed ipc.* wrapper organized by domain (output, preview, projector, mapping, gpu, config, media, camera, recording, etc.). Provides compile-time safety and cleaner call sites throughout the app.
• Layer Builder (utils/layerBuilder.ts): Extracted buildLayers(), buildLayersFromList(), and compositeExtraSources() from the render loop into a standalone utility module with a LayerBuildContext interface (~300 lines). Eliminates closure dependencies and enables future testability.
• useOutputState Hook (hooks/useOutputState.ts): 14 output state vars, 6 3D preview vars, 2 output configs, and 9 compositor refs extracted into a dedicated hook. Accepts initialOutputDisplayConfigs parameter.
• ServiceRegistry (services/serviceRegistry.ts): Centralized lazy service lifecycle management for 6 services (Syphon/Spout, NDI, Plugin Serverless, Scope Controller, WebRTC Import, RunPod). Services are created on-demand with proper disposal on unmount.
• useUIState & useLayerManager Hooks: 32 UI state variables + 7 effects and 12 layer management handlers fully wired into App.tsx.
• R3 Domain Contexts Plan (docs/R3_DOMAIN_CONTEXTS.md): Documented future architecture plan for 5 React Contexts (Selection, SceneData, Compositor, Output, Scope) to replace 80+ prop drilling on CanvasView. Includes Storybook setup strategy and render profiling plan.

GPU Rendering Pipeline (Phases 1–3)

Complete output performance stabilization from plan through implementation:

• Phase 1 — Quick Wins: Fixed BeatDetector/AudioAnalyzer startup race (competing RAF loops), staggered output encodes (even→output1, odd→output2 for 50% CPU reduction), output FPS feedback via IPC (cyan badge in OutputsTab), RAF timing stabilization with EWMA smoothing.
• Phase 2 — Pipeline Optimization: Half-resolution OffscreenCanvas encode (~4× faster than full-res JPEG), encode timing instrumentation with EWMA metrics, lazy service initialization (Syphon/Spout, NDI, Scope, Plugin Serverless only init when needed).
• Phase 3B — WebGPU Compositor: Ping-pong render technique with WGSL shaders, 13 blend modes (multiply, screen, overlay, darken, lighten, etc.), per-pixel RGB gains/gamma/black-white point/zoom/pan/flip. Metal backend on macOS for full M3 GPU utilization. Feature-detects navigator.gpu, falls back to Canvas 2D.
• Phase 3C — Direct Frame Transport: MessageChannelMain for direct renderer-to-renderer communication (zero-copy ArrayBuffer transfer, bypasses main process). Falls back to JPEG IPC path if frame port not established.
• Feature Onboarding: First-run modal for feature selection (inputs, audio, outputs, AI, MIDI). Stored in localStorage, gates service initialization for simpler setups.

WebCodecs Frame Transport (New in v2.2.7)

Hardware-accelerated video compression for cross-window frame delivery:

• WebCodecs VideoEncoder integration with H.264/VP8 codec support (8 Mbps bitrate, realtime latency mode, prefer-hardware)
• EWMA timing instrumentation for readback, transfer, and total frame metrics via EWMATracker utility
• FrameSender.sendFrameEncoded() with codec-init handshake, keyframe interval (every 30 frames), and automatic codec configuration
• Backward-compatible: FrameReceiver handles both encoded (codec) and raw RGBA messages — falls back gracefully when WebCodecs unavailable
• Compositor integration: useWebCodecsTransport flag in compositor defaults to true when VideoEncoder is available

WallSpace Upscaler Plugin

• New Scope plugin for AI-powered upscaling of CRT wall output
• Serverless GPU Docker container with enhancement pipeline
• Scope template bumped to v0.1.7

UI Fixes & Polish

• Render Engine panel now auto-collapses when navigating to the Outputs tab (starts collapsed on mount)
• Quick Add sidebar auto-collapses when a CRT or projector is selected in the canvas, reducing panel clutter
• Bottom panel (DetailPanelRouter): Removed CRT Details and Projector/Slice Details from the bottom bar — those panels are already available in the canvas right sidebar. The Layer Editor now stays visible when CRT or projector is selected, so you can adjust layers without switching selections.
• Compact Wall/Space selector: Redesigned from two rows into a single compact bar — wall tabs on the left, space selector on the right. Inline badges for 3D transform, source region, and scene assignment. "All Walls" view automatically hidden when only 1 wall exists in the active space.
• SelectionTagBar component: Unified tag display for wall, CRT, projector, and scene selections in the toolbar
• ScenePlaybackQueue component: Timeline-based scene sequencing

Claude Code Workflow

This release was developed across multiple sessions using the everything-claude-code methodology:

• Architectural analysis with dependency mapping and risk assessment
• Phased refactoring plan (R1 hooks → R2 services/IPC → R3 contexts) with incremental verification
• Performance plan with 11 prioritized tasks across 3 phases, each verified against output FPS metrics
• Persistent memory files for cross-session continuity (render-pipeline.md, refactoring.md)
• Type-checked after every change (npx tsc --noEmit) with zero regressions

3/09/2026

https://github.com/jackmo650/crt-wall-controller/releases/tag/v2.2.6

WallSpace.Studio v2.2.6

 jackmo650released this4 days ago· 9 commits to main since this release v2.2.6 faf5740

What's New

• Scenes & Layers panel stays open — The panel now defaults to expanded when the app loads, no longer auto-collapses when switching tabs
• Windows support — First portable .exe release for PC users
• Windows disk space check — Fixed low-storage warning to work on Windows (uses wmic instead of Unix-only df)
• Build improvements — macOS native helpers (Syphon, NDI) properly packaged as extra resources; native files excluded from Windows builds

Downloads

FilePlatformWallSpace.Studio-2.2.6.dmgmacOS installerWallSpace.Studio-2.2.6-mac.zipmacOS zipWallSpace.Studio 2.2.6.exeWindows portable (no install needed)

Notes

• macOS: If you get a Gatekeeper warning, right-click → Open, or go to System Settings → Privacy & Security → Open Anyway
• Windows: The portable exe runs directly — no installation required. Your GPU is used automatically for WebGL rendering.