EgoLens

Visualize point clouds, cameras, and 3D annotations in your browser
— straight from the most widely used autonomous driving datasets.
No conversion, no preprocessing.

   

Live Demo · Teachable Lens (WebMCP) · URL Loading · Share View · Dev Setup

_{Waymo Open Dataset — 3D segmentation, keypoints, and camera views. Timeline markers indicate frames with annotations (cyan = LiDAR seg, lime = 3D keypoints, light-cyan = 2D keypoints, magenta = camera seg).}

What It Does

One tool for the three most widely used AV datasets. Drop a folder or paste a URL — auto-detected, zero setup.

• LiDAR point clouds with multiple colormap modes (intensity, height, range, segmentation, camera projection)
• 3D bounding boxes as wireframes or 3D models with color-coded tracking
• Synchronized camera views with POV switching — click a camera to jump into its viewpoint
• Trajectory trails showing object movement over past frames
• Semantic segmentation overlays (LiDAR and camera)
• Timeline with play/pause, frame scrubber, and buffer progress bars

nuScenes — 32-class LiDAR segmentation projected onto camera views via LiDAR overlay	Argoverse 2 — camera colormap mode with 7-camera POV switching

Supported Datasets

Feature	Waymo v2	nuScenes	Argoverse 2
LiDAR point cloud	✓ (5 sensors)	✓ (1 sensor + 5 radar)	✓ (2 sensors)
Camera images	✓ (5 cams)	✓ (6 cams)	✓ (7 cams)
3D bounding boxes	✓	✓	✓
2D camera boxes	✓	—	—
Cross-modal hover linking	✓	—	—
Trajectory trails	✓	✓	✓
3D human keypoints	✓	—	—
2D camera keypoints	✓	—	—
LiDAR segmentation	✓ (23-class)	✓ (32-class)	—
Camera panoptic seg	✓ (29-class)	—	—
POV camera switching	✓	✓	✓
Local (drag & drop)	✓	✓	✓
URL loading	✓	✓	✓

Dataset format is auto-detected from folder structure.

Teachable Lens (WebMCP)

WebMCP tools usually let agents use an app. EgoLens flips it: agents extend the app. Drop a folder EgoLens does not recognize and the page becomes a WebMCP tool provider. Your agent teaches EgoLens the format, you approve the render, and the app's capabilities keep growing. Files never leave the browser.

Built during the WebMCP Challenge (Aug 29 – Sep 3, 2026). EgoLens existed before as a browser viewer for three built-in datasets with remote loading and share links. Everything in this section is submission-period work: the recipe language and runtime, the five document.modelContext tools, the authoring session and review UI, the three-host compatibility, and the sample release. The commit-by-commit split is in docs/WEBMCP_CHALLENGE.md (PRs #23–#135).

Why WebMCP fits this use case

Driving logs are large and often confidential, so the data has to stay in the browser. The expertise needed to read a new format is in the agent; the judgment of whether a render is right is in the person looking at it. WebMCP puts the tools where the data already is and gives both sides the same page: the agent gets typed tools, the person gets the viewer. Any WebMCP host works with zero agent-specific code.

How it creates a better user experience

• Adding a dataset format no longer means converting your data or forking the project. You drop the folder, say one sentence, and review renders.
• Every validated revision renders in the production viewer, docked next to the review panel, so problems are found by looking, not by reading JSON.
• Review is per capability with named issues; verdicts reach the agent through get_state, so you never copy error text into a chat.
• Drag-and-drop and Select Folder both work in in-app browsers (ChatGPT, Codex) and in Chrome; no File System Access API is required.
• A taught format is remembered: drop that layout again and it renders in one click, or share it as a link.

What people and agents can now do together

Before, an engineer wrote a loader for one dataset and it died with the notebook. Now a person and an agent author a reusable adapter in minutes, with the person's eyes as the acceptance test. During the challenge week three unknown datasets were taught this way: A2D2 (3 revisions), KITTI Raw (7), PandaSet (2–16 depending on the run, one human rejection each time). The agent reads files and writes bindings; the person looks at the scene and says what is wrong. Each does what it is best at, and the viewer keeps growing.

How we implemented WebMCP

Five tools are registered with document.modelContext.registerTool while an unknown folder is open and removed when the session ends (see src/teachable/authoring/webMcp.ts). Tool descriptions are self-describing steps, inputs are normalized whether they arrive as objects or JSON strings, and results are strings, so ChatGPT's in-app browser, Chrome's native WebMCP, and the Codex app behave identically. Recipes are declarative JSON executed in an isolated graph runtime with resource limits; a wrong recipe renders wrong rather than running anything. Playwright probes drive Chrome's real WebMCP against the live site end to end.

_{An agent's third revision of an adapter for the sample driving log, rendered in the production viewer. Each capability is accepted or rejected by looking, and the verdict flows back to the agent.}

How it works

1. Drop an unknown folder. EgoLens detects the sensor layout (cameras, LiDARs, radars). You confirm it and name the dataset. That layout is a contract the recipe must meet.
2. Tell your agent: "Teach EgoLens this dataset." Five tools are registered on document.modelContext:
   • egolens_teachable_inspect — inventory, metadata, bounded text/JSON samples, table schemas (Parquet, Arrow, pandas pickles). Raw bytes stay in the tab.
   • egolens_teachable_get_contract — the recipe JSON schema, the reader/operator vocabulary with JSON-schema params, an authoring guide, and a placeholder skeleton. No dataset examples.
   • egolens_teachable_apply_revision — submit a complete recipe. EgoLens compiles it, binds files, renders sample frames, and returns diagnostics that name the failing input.
   • egolens_teachable_get_state — validation state, the sensor contract, your latest review, and the next step.
   • egolens_teachable_finalize — seal the recipe with recipe, format, and operator-set hashes.
3. Review in the real viewer. Every validated revision loads into the production renderer docked beside the review panel: orbit, scrub, toggle boxes and segmentation, switch camera POVs. Accept or reject each capability with a named issue.
4. Finalize. The sealed recipe can be exported, imported, and shared. Drop the same layout again and EgoLens recognizes the trained format and renders it in one click.

Recipes are declarative JSON, never code, executed in an isolated graph runtime with resource limits. The same tools work unchanged in ChatGPT's in-app browser, the Codex app, and Chrome 146+ with chrome://flags/#enable-webmcp-testing.

Try it

1. Download and unzip the sample driving log: 80 frames, 439 MB or the 6-frame version, 32 MB (six cameras, one LiDAR, 3D cuboids, per-point labels; CC BY 4.0, attribution in the release notes).
2. Open egolens.org in a WebMCP-enabled browser and drop the folder.
3. Say "Teach EgoLens this dataset." and review each capability as it renders.

Datasets taught this way so far: A2D2, KITTI Raw, PandaSet. Judge notes, the prior-vs-new work split, and run logs are in docs/WEBMCP_CHALLENGE.md. The WebMCP entry point is src/teachable/authoring/webMcp.ts.

Quick Start

Local files (drag & drop)

1. Open the live demo
2. Drag & drop your dataset folder into the browser
3. Done — browse frames, toggle sensors, play the timeline

URL Loading

Load data directly from S3 or any static file server by providing a URL.

Two modes:

• URL only — auto-discovers all segments/scenes in the directory
• URL + Segment ID — loads a specific segment directly (works with any static file server)

https://egolens.org/?dataset=argoverse2&data=https://your-server.com/av2/sensor/val/
https://egolens.org/?dataset=nuscenes&data=https://your-server.com/nuscenes/
https://egolens.org/?dataset=waymo&data=https://your-server.com/waymo_data/&scene=SEGMENT_ID

The URL should point to a directory containing the dataset's standard folder structure. Works with S3 buckets, any HTTP server, or localhost.

Note: Waymo's license prohibits data redistribution, so no hosted demo data is available. You'll need to host your own copy after accepting the Waymo Open Dataset License.

Share View

Point your team to exactly what you see. When data is loaded via URL, click Share View to get a link that encodes your exact view — frame, colormap, camera angle, overlays, everything. Paste it in Slack or a PR comment and your teammate lands on the same frame, same angle, same overlays. No screenshots, no "go to frame 142 and turn on segmentation."

_{Share what you see — one link captures frame, camera angle, overlays, and all settings.}

Dev Setup

git clone https://github.com/egolens/egolens.git
cd egolens
npm install
npm run dev

npm run build   # Type-check + production build
npm run lint    # ESLint
npm test        # Vitest

Built With

React 19 · TypeScript · Three.js · React Three Fiber · Vite · Zustand · hyparquet · Web Workers

Browser Support

Chrome / Edge recommended. Safari may crash on large datasets due to WebKit memory limits. Firefox works but lacks the folder picker API.

Feedback

Found a bug? Have a feature idea? Want support for another dataset? Open an issue — all feedback is welcome.

See CONTRIBUTING.md for development guidelines · Changelog

Citation

If you use EgoLens in your research, please cite it via its Zenodo DOI:

@software{kim_egolens_2026,
  author    = {Kim, Heejae},
  title     = {{EgoLens: A browser-based viewer for autonomous-driving datasets}},
  year      = {2026},
  publisher = {Zenodo},
  version   = {1.0.0},
  doi       = {10.5281/zenodo.20460188},
  url       = {https://github.com/egolens/egolens}
}

Or use the Cite this repository button at the top of the GitHub page (powered by CITATION.cff).

License

MIT · Built by Heejae Kim