Spectral World Renderer

Unified 3D Hyperspectral Unmixing

  • 1University at Albany, State University of New York
  • 2National Yang Ming Chiao Tung University
  • 3National Cheng Kung University

Preprint · under review

Paper Supplement Code Dataset Video BibTeX

Query any material in hyperspectral 3D worlds

A camera records how a surface looks; Spectral World Renderer shows what it is made of. One framework generates worlds of known composition and unmixes inside rendering, so a single render gives spectra and material abundances. Query any surface by name, click, spectrum, band or combination, and verify the answer against the known composition. Try the material queries →

From hyperspectral worlds to queryable material layers. Input: multi-view hyperspectral images of worlds with known material composition, produced by the SWR world generator (left). Output: a Gaussian scene whose primitives carry material abundances and spectra. Query: a user selects a material by name, a click (crosshair), a reference spectrum, a band index or a combination, and SWR colours the retrieved surfaces in any view; each tile shows one query on a test view.

SWR-Bench worlds

SWR-Bench: eight indoor and outdoor worlds with measured spectra, rendered with 64 bands from 420 to 980 nm together with material labels and area proportions.

Abstract

Cameras and the foundation models built on them recognize what a 3D scene contains, but not what it is made of: surfaces of the same colour can be different materials. Hyperspectral unmixing explains each spectrum as a mixture of material signatures, yet in 3D it is applied to rendered images after the views are composited, or tied to slow implicit fields, and no multi-view data with known composition exist to check the result. We present the Spectral World Renderer (SWR), which unifies world generation and unmixing. A coding agent builds executable 3D worlds that are rendered with measured spectra, so every hyperspectral image comes with exact labels and sub-pixel proportions; the resulting SWR-Bench provides eight indoor and outdoor worlds in which decompositions can be verified. SWR then unmixes inside rendering: each Gaussian of a compact 3D representation is decoded into material abundances before compositing, so one render yields spectra and material maps, and any surface can be queried by material name, a clicked pixel or a reference spectrum. Without material labels, SWR identifies materials best in all eight worlds, has the best mean PSNR on real captures, trains 3.7–12.9× faster than unmixing baselines, renders 34.9 labelled views/s and separates same-coloured materials.

8 / 8worlds with the best material identification (mIoU)
6 min 37 scomplete City pipeline, from RGB anchor to decoder
34.9labelled views per second at 960 × 540 × 64
3.7–12.9×faster training than the unmixing baselines on City

Video

Overview video frame: a mosaic of SWR-Bench worlds with their material layers in many colours, captioned Any world. Every material.

Playing the video loads it from YouTube (privacy-enhanced mode).

Pipeline

SWR world generator: worlds with known composition

Real multi-view hyperspectral datasets are tabletop or turntable captures without material labels or sub-pixel proportions, which cannot be annotated reliably by hand. The SWR world generator therefore creates worlds rather than annotating captures.

The SWR world generator. From reference photographs and a short brief, a coding agent writes an executable world whose every surface carries a library-measured spectrum. One shared sub-pixel rendering produces the hyperspectral and RGB observations together with the material of every pixel, so the ground truth is exact for the rendered world; checks on layout and rendering return flawed worlds for revision before release.

Generate the world

A coding agent turns reference images and text into an executable scene program that fixes geometry, materials, illumination and cameras.

Ground it in measurements

Every material is assigned measured spectra, with specimen variation and changes in spectral shape between sunlight and shade.

Render aligned evidence

Shared sub-pixel samples render RGB and hyperspectral observations together with material labels and area proportions.

SWR: unmixing inside rendering

Unmixing a rendered image afterwards acts on spectra that compositing has already mixed across surfaces. SWR instead unmixes inside rendering: at every ray intersection, a shared decoder turns the local feature into abundances and an intensity, and compositing accumulates both, so one render yields spectra and abundances. One Gaussian representation, composited with one set of visibility weights, renders spectra, abundances and material queries at any band count.

Overview of SWR. An RGB anchor fixes the geometry, and each Gaussian carries harmonic features whose size does not depend on the number of bands. At every ray intersection, a shared decoder maps the interpolated feature to abundances and an intensity, which mix scene-wide endmembers before compositing; the same visibility weights render the spectrum and the abundances. Clusters of multi-view appearance initialize the endmembers and regularize their assignment, so no material label enters training.

Band-independent storage

An RGB-trained anchor provides compact geometry, and each Gaussian stores a feature instead of a spectrum, so storage does not grow with the number of bands.

Decode before compositing

A shared decoder maps the feature at every ray intersection to abundances over a scene-wide endmember dictionary, so one render yields spectra and abundances and every Gaussian carries queryable abundances.

Intensity and appearance regularization

A separate intensity absorbs shading, and a multi-view appearance regularization assigns an over-complete set of endmembers to distinct appearances.

Multi-view appearance regularization

Because the intensity absorbs brightness, a single endmember can explain two spectrally similar materials: in Forbidden City, white marble is four times brighter than grey masonry but only 3.8° apart in spectral shape.

Clustering the pixels of all training views by spectral shape and brightness separates such appearances, and every view shares the same clusters. Each cluster initializes its own endmember, and the assignment is retained as a weak regularizer during training, which gives marble its own layer.

Simulated hyperspectral observations

Narrow bands share one exposure per world, so brightness differences between bands follow the radiance; the colour-infrared composite (R, G, B = 847, 651, 553 nm) renders vegetation red. Outdoor apparent reflectance reproduces the library reflectance, including chlorophyll absorption near 670 nm and the red edge; path-traced interiors deviate through interreflection and residual Monte Carlo variance. Bottom: clean and noisy crops of the 900 nm band.

Results

Material queries

Every Gaussian carries abundances and a spectrum, so a query is a selection over primitives, consistent across views. Pick a world below, choose a query and drag the divider to compare the RGB rendering (left) with the result (right).

  • Name a material, e.g. fired clay or water
  • Combination of names, e.g. stone ∨ travertine
  • Click a shaded pixel: the sunlit parts of the same material are retrieved too
  • Spectrum from a library, e.g. a living leaf
  • Band or spectral index, e.g. near-infrared
RGBQuery

Try different query modes and worlds.

Abundances come from SWR models trained without material labels. A name query sums the abundances of the endmembers assigned to the class through a frozen training-view map, on a fixed 0–1 scale, and a combination sums two classes. Click, spectrum and index queries compare the spectrum of every Gaussian with the clicked spectrum, a library reflectance or a band index under a fixed threshold. These clips are illustrative; quantitative results are reported below and in the paper.

Seeing beyond RGB

Living leaves differ from materials of the same colour beyond the visible range: chlorophyll absorbs red light and leaf tissue reflects near-infrared light, producing a red edge near 700 nm that minerals and dry leaves lack. In City, foliage-coloured mineral leaves are only 2° from foliage in RGB but 42° apart in spectrum, so they receive endmembers of their own.

Because abundances are stored on explicit, rasterizable Gaussians, a material query is a selection over primitives that is consistent across views and requires neither labels nor a per-image classifier. The mineral-leaf query retrieves no ordinary tree, whereas the equivalent RGB query retrieves eleven.

A measured spectrum can also serve as the query. With the sun and sky illumination factored out, a pixel clicked in shadow retrieves 77 % of the sunlit part of its material, against 10 % for an RGB colour query (mean over four outdoor worlds), in about 3 ms per query.

Material identification and spectral fidelity on SWR-Bench

SWR has the best mIoU in every world and learns the endmembers closest to the true spectra. Known composition exposes what fidelity hides: in Market and Study, 3DGS with post-hoc unmixing reaches a lower SAM but a far lower mIoU. Within SWR itself, decoding after compositing raises Market PSNR from 33.72 to 34.37 dB yet lowers mIoU from 0.47 to 0.37, the same gap.

WorldUnMix-NeRF3DGS + VCA / NNLSSWR
Material layers across the eight worlds. Each row tints one material by its abundance in a test view; IoU uses summed abundance > 0.5 against the true mask. Unmixing rendered spectra afterwards (3DGS + VCA) or inside a radiance field (UnMix-NeRF) spreads the material over unrelated surfaces, whereas SWR recovers the layer closest to the ground truth.

Efficiency

The City pipeline covers all four stages, from RGB anchor to decoder. Throughput counts views per second with spectra and label maps at 960 × 540 × 64.

City training time (lower is better)
Mean rendering throughput (higher is better)

SWR uses an optimized fused fp16 decoder that skips hits with weight below 10−4 (99.97 % identical labels); baselines use their reference code. Throughput is the mean over Market, Botanical, Study and Karst.

Real captures

Without material ground truth, real captures test spectral fidelity. SWR has the best mean PSNR on each real source; on HS-NeRF, 3DGS reaches a lower mean and 95th-percentile angle.

SourcePSNR ↑
baseline · ours		SAM ↓
baseline · ours
NeSpoF21 bands · HyperGS28.36 · 29.394.58 · 3.86
HSI-SC apple204 bands · HyperGS27.95 · 30.671.64 · 1.35
HS-NeRF128 / 141 bands · 3DGS26.32 · 26.515.71 · 5.97
204-band real capture. On the HSI-SC apple, SWR renders the most faithful spectra in this view. Means over 4 NeSpoF and 5 HS-NeRF scenes; the baseline is the best PSNR of 3DGS, HyperGS and DD-HGS; bold marks the better value of each pair.

Analysis

What the decoder learns

Intensity carries shading; endmembers carry material and illumination mode. Test views of Market (K = 24), City (K = 32) and Office (K = 32). The composited intensity follows the direct share of the illumination, and each material occupies distinct endmembers in sun and in shadow, so the over-complete dictionary represents illumination modes within a class.
  • Within one specimen, the logarithm of the composited intensity correlates with that of the true radiance (r = 0.97 on Market, 0.87 on City).
  • The sun or shadow state explains 84 % and 67 % of the within-class variance of the intensity, but only 3 % and 15 % of the variance of the class mass.
  • Abundances change little with the viewing direction on the Lambertian worlds (total variation at most 0.04 within 45°).

Primitives as material carriers

Each Gaussian is enclosed by the regular tetrahedron of its canonical frame, scaled, rotated and translated with the Gaussian. The clips dissolve from these bounding tetrahedra into the rendered view along the camera path of the material queries.

Bounding tetrahedra on test views. Columns show the rendering, the tetrahedra weighted by opacity and visibility, the tetrahedra coloured by the dominant class decoded at each Gaussian centre, and the rendered material map. The colouring reproduces the class layout of the rendered map: the median visible Gaussian places more than 98 % of its centre abundance on one class.

Citation

@article{lee2026swr,
  title   = {Spectral World Renderer: Unified 3D Hyperspectral Unmixing},
  author  = {Lee, Chia-Ming and Hsiao, Yu-Jou and Chang, Ming-Ching and
             Li, Xin and Liu, Yu-Lun and Hsu, Chih-Chung},
  journal = {Preprint, under review},
  year    = {2026},
  url     = {https://ming053l.github.io/Spectral-World-Renderer/}
}

Acknowledgements

To be added.