Alex Carlier

3D Gaussian Steganography, Alex Carlier (X | LinkedIn), Sept. 2023 (X post) Open-source code: https://github.com/ReshotAI/gaussian-painters

What am I looking at?

This artwork is a 3D Gaussian Splat (a new Radiance Field technique introduced in the SIGGRAPH 2023 paper titled "3D Gaussian Splatting for Real-Time Radiance Field Rendering" by Kerbl et al.).   While it is normally used for novel-view synthesis of a 3D scene, this experiment explores its artistic use in 2D planes.

Caption: The same 3D Gaussian point cloud, visualized in Blender with my experimental add-on https://github.com/ReshotAI/gaussian-splatting-blender-addon

What are 3D Gaussian Splats?

I have written a beginner friendly introduction to 3D Gaussian Splats in this blog post: https://www.reshot.ai/3d-gaussian-splatting Given a set of pictures from a 3D scene or object, a 3D Gaussian Splatting model optimizes a point cloud of 3D gaussians (think of ellipsoids) with different sizes, opacities, and (view-dependent) colors. Here's an example of a capture I made. As input, I used 750 images from a plush toy, that I recorded with my phone from different angles. Once trained, here is the point cloud visualized as simple points. When blended together, here's the visualization of the full model, rendered from ANY angle. As you can see, 3D Gaussian Splatting captures extremely well the fuzzy and soft nature of the plush toy, something that photogrammetry-based methods struggle to do. Here are some more 3D Gaussian Splats I trained. Compared to photogrammetry scans, they better capture fine details and view-dependent effects, like shiny reflections.

Let's move to 2D (Gaussian Painters)

Now what if we provide a single picture for the training of 3D Gaussian Splats?    As explained in my Tweet, we obtain (with some additional constraints so that the Gaussians stay on a 2D plane) "Gaussian Painters", where overlapping gaussians are optimized to fit a given picture (here, the painting Girl with a Pearl Earring).  Not perfectly fitting the target image somehow creates a beautiful painterly effect.

Gaussian Steganography

Inspired by the YES/NO optical illusion sculpture, by Markus Raetz, at Art Basel 2010, what if we optimize a 3D Gaussian Splatting model with a synthetic scene consisting of two orthogonal images. Here is the COLMAP setup for this idea. After training the scene, we obtain the following 3D Gaussian point cloud, where two different pictures can be observed at different angles. Here's another 3D Gaussian scene optimized with 3 different pictures at different angles. As a result, we get a new kind of steganographic art where 3 famous painting from Monet are hidden in a fuzzy point cloud.

In AR/VR?

Now imagine being able to observe this artwork in AR/VR! How fun would it be to walk around it and discover various hidden pictures, visible from certain angles only. I tried to do exactly this, by making 3D Gaussian Steganography avaialble for everyone to try as an AR filter on Instagram. Here's my attempt developing it in Meta Spark Studio with custom shaders. However, because of alpha blending limitations, I never published it. Should I give it another try? 😄 If you liked this artwork / creative experiment with AI, feel free to share your thoughts on the original post on X, or by writing me on social media. Thanks!