Mixed Reality + Diffusion prototype as a tool for exploring concepts, styles, and moods by transforming real-world surroundings into alternate realities.
#MixedReality #MR #AI #StableDiffusion #Quest3
226752156.5K
Hugues Bruyère✓
@smallfly
Try / Share / Iterate — Creative Technologist / Chief of Innovation @bonjourdpt
1.6K Following8.8K Followers5.1K Posts
- Hugues BruyèreRT @smallfly: Vitre Experiment #6… my favorite one so far! Same hallway, new portal effect opening onto a completely different world. Here again, I used Atlas and Marble 2 from @theworldlabs to reimagine the hallway from my original 3D capture. I’m having a lot of fun exploring what these tools can do, and I’m really impressed with the results. As with my previous experiments, the idea behind Vitre is to make the monitor disappear by aligning a virtual scene with the physical world. And of course… everything is 3DGS, rendered in real time. #3DGS #WorldModels #GaussianSplatting036027.3K
- Hugues BruyèreRT @smallfly: I love the point cloud and the detailed renders generated by Atlas... Here's a bit of the process to create the alternate version of the hallway... I used Atlas from @theworldlabs to create the look and generate a series of consistent views, then trained my own 3DGS from those views.Hugues Bruyère @smallflyVitre Experiment #5: Opening a portal… I’ve been experimenting with Atlas and Marble 2 from @theworldlabs , and seriously, the results are impressive! I generated an alternate version of the hallway I’ve been using for my experiments, then used Vitre to align it with the physical space and create the illusion of a portal opening into this alternate reality. #GaussianSplatting #3DGS #WorldModels0705.5K
- Hugues BruyèreRT @smallfly: Vitre Experiment #5: Opening a portal… I’ve been experimenting with Atlas and Marble 2 from @theworldlabs , and seriously, the results are impressive! I generated an alternate version of the hallway I’ve been using for my experiments, then used Vitre to align it with the physical space and create the illusion of a portal opening into this alternate reality. #GaussianSplatting #3DGS #WorldModels0406.7K
- Hugues BruyèreRT @smallfly: I’ve been exploring what can be done with the developer data exported from @XGRIDS_OFFICIAL LCC Studio: combining the LiDAR data from a PortalCam capture with images captured separately, in this case using a @DJIGlobal Osmo 360. Left: 3DGS generated directly in LCC Studio from the PortalCam capture. Right: 3DGS trained in Spirula Studio (which is amazing) using the PortalCam LiDAR data for the geometry and the Osmo 360 images for the pixels. Still a WIP... The hybrid version (trained with Spirula) has better colour and a different overall feel, while the LCC version is still noticeably sharper and preserves more fine detail. Next test: adding some of the PortalCam’s own high-resolution images back into the Spirula dataset (the forward-facing rectilinear frames to start), while keeping the Osmo images as the majority of the dataset. #GaussianSplatting #3DGS #PortalCam #SpirulaStudio #Osmo3600403K
- Hugues BruyèreRT @smallfly: Vitre Experiment #4. Winter is coming, so I brought a bit of spring into the hallway by adding synthetic 3DGS assets to the scene. Still running smoothly. #3DGS #GaussianSplatting #ComputerVision01104.6K
- Hugues BruyèreRT @smallfly: Finally got around to trying Spirula Studio, and I'm really impressed! It simplifies the process of preparing datasets from 360° footage enormously. Trained this 3D Gaussian Splatting scene from footage I captured back in 2022 with an Insta360 ONE RS 1-Inch 360. #3DGS #GaussianSplatting #SpirulaStudio01204.1K
- Hugues BruyèreRT @smallfly: I added a ToF sensor to the equation... an Femto Bolt. Both the captured Gaussian Splat and the Femto Bolt depth data are registered to the physical space and rendered with the same off-axis projection… so now people passing through the hallway are captured and inserted into the 3DGS scene. Still rough around the edges… but I like it. This time I needed some help with the demo. Someone had to be the tracked POV while I strangely walked around the hallway :) #3DGS #GaussianSplatting #ComputerVision040040.8K
- Hugues BruyèreRT @kcimc: anyone who was around for the early 2010s kinect scene out was dreaming about this sorta thing. we just didn't have the fidelity and framerate yet. sooo satisfying to see this for real finally.Hugues Bruyère @smallflyI added a ToF sensor to the equation... an Femto Bolt. Both the captured Gaussian Splat and the Femto Bolt depth data are registered to the physical space and rendered with the same off-axis projection… so now people passing through the hallway are captured and inserted into the 3DGS scene. Still rough around the edges… but I like it. This time I needed some help with the demo. Someone had to be the tracked POV while I strangely walked around the hallway :) #3DGS #GaussianSplatting #ComputerVision013035
- Hugues BruyèreRT @smallfly: This time the tracking runs with a 120 fps global-shutter camera. Getting twice as many position updates, with much fewer gaps when tracking faster movements… It feels much better. The window is still rendering from a position roughly 20–24 ms in the past, but I compensate for some of that by predicting the position slightly forward. It still works best when tracking the phone, though. #3DGS #GaussianSplatting #ComputerVision01660179K
- When I need a break from my screen, I make it disappear. Vitre, experiment #2. This time I’m using one of the monitors on my desk. It happens to sit right in front of a glass wall looking into the hallway and the rest of the office, which makes for a much more interesting view. Press the button and Vitre takes over the monitor, rendering a 3D Gaussian Splat I previously captured of the space behind it. The 3DGS is registered to the physical space, and the view is rendered in real time from the tracked viewer’s position... making the monitor vanish. I’ve been iterating on the tracking and motion-to-photon latency since the first experiment. Still a few things I want to explore, but the illusion is getting better. #3DGS #GaussianSplatting #ComputerVision34543.3K
- Posts @smallflyNext run…tracking the phone with a global-shutter 120 fps camera.
001151 - Show more posts
