Hybrid CAD + Generative AI Hero Experiment
The goal was to create a polished, cinematic website hero that could show off a subject while preserving it exactly. For 5585, an app project, we wanted to see whether we could combine the two approaches we’ve tried and get the strengths of both:
CAD / Deterministic Rendering
- Exact phone geometry
- Exact screen dimensions and aspect ratios
- Exact phone positions, rotations, and perspective
- Literal app screenshots mapped correctly onto the displays
- Highly predictable and repeatable
But the phones, materials, lighting, reflections, and environment don’t look as polished or cinematic as we want.
Generative AI
- Much more convincing product photography
- Better glass, materials, lighting, reflections, shadows, glow, and atmosphere
- Can create the polished studio aesthetic we want
But isn’t sufficiently geometrically predictable.
What We Tried
We created the hero scene in VTK/CAD and supplied three real app screenshots.
The idea was:
Real screenshots → exact CAD phones → generative enhancement → polished cinematic hero
The CAD render would effectively tell the image model:
These are the three phones. This is their exact size, orientation, position, perspective, and screen geometry. Make the scene look like a high-end product photoshoot.
The generated result was visually impressive. The phones and environment looked substantially more cinematic than the CAD version.
However, the model regenerated the phones and screens rather than treating them as immutable geometry. The resulting UI was similar to the original screenshots, but it wasn’t the actual UI. Text, cards, icons, images, spacing, etc. had been reconstructed with obvious degradation.
We Then Tried Restoring the Real Screenshots
We tested whether we could let AI create the more cinematic phones and environment and then simply put the original screenshots back onto the generated phones afterward:
CAD reference → AI scene → remove generated UI → perspective-map real screenshots back in
That failed for a more fundamental reason.
The AI hadn’t merely changed the contents of the screens. It had changed the physical geometry of the phones themselves:
- Screen dimensions
- Display aspect/proportions
- Bezel widths
- Corner geometry
- Phone-body proportions
- Perspective/shape of the display opening
Therefore, the real screenshots no longer fit the generated phones correctly.
Perspective warping can map four corners, but it cannot make a screenshot designed for one physical display geometry correctly fit a different display geometry without distortion.
Ironically, this is exactly the problem that originally caused us to move to CAD.
What the Experiment Demonstrated
The problem isn’t really generative AI’s visual quality anymore.
It already produced the kind of materials, lighting, glass, reflections, and cinematic studio atmosphere we’re looking for.
The missing capability is hard constraint enforcement.
For this workflow, we need to be able to define certain properties as ground truth:
- Phone geometry
- Display geometry
- Phone transforms
- Camera/perspective
- And most importantly, the subject itself: the app screenshot pixels
While giving the model creative freedom over:
- Materials
- Lighting
- Reflections
- Shadows
- Background
- Surface
- Props
- Atmosphere
Current image generation generally treats the CAD image as guidance or a reference, not as a geometric specification it is physically prohibited from changing.
Even if a generated phone differs by only a small percentage, that’s enough to make a real screenshot no longer fit perfectly.
What Would Actually Solve It
The ideal future workflow would essentially be a generative renderer rather than a pure image generator.
We would provide the existing CAD scene containing the exact phones, camera, screen planes, and screenshots. Then tell the system:
Photorealistically render this scene. You may determine materials, lighting and environment, but you may not alter the geometry, transforms, camera projection, screen dimensions, or screen textures.
The CAD scene wouldn’t need to look good. Its purpose would be to establish truth.
Generative AI would then provide the photographic realism without being allowed to reinterpret that truth.
Why We’re Not There Yet
Current tools have masks, reference images, depth/edge conditioning, inpainting, structure guidance, and similar controls. Those can make a model more likely to follow supplied structure.
What we need is stronger:
A guarantee that specified geometry and pixels cannot change.
That’s the difference between:
“Make something that looks like this.”
and:
“This subject is fixed. Preserve it exactly while making everything around it look cinematic.”
The first is something generative AI does extremely well today. The second isn’t yet reliably available as a straightforward generative-image workflow.
An Alternative Workaround
There was another way to approach the problem: stop asking generative AI to render the phones at all.
Instead of:
Exact CAD phones → generative AI modifies the entire scene
We could separate the two parts:
- Generative AI creates the environment
- CAD creates the phones
- The two are composited afterward
This turns the process into something closer to staging a virtual product photoshoot.
Generative AI is responsible for the set:
- Background
- Surface
- Lighting
- Props
- Atmosphere
CAD remains responsible for the product:
- Phone geometry
- Phone positions and perspective
- Display dimensions
- Actual app screenshots
The generated scene is intentionally created without phones and with an open area where the phones will eventually be placed.
That changes how the generated scene has to be designed.
For the app website experiment, we used a flat dark polished-stone surface so the CAD phones would have somewhere believable to stand. Props were kept around the edges so they wouldn’t interfere with the phone placement.
The lighting was also designed around an object that wasn’t there yet.
A broad light source was placed behind the empty phone area. When the CAD phones are later composited into that space, they cover the center of the light source while the glow remains visible around and between them.
The result is effectively:
AI-generated studio → CAD phones → final hero
The phones never pass through the generative model, so their geometry and screenshots never need to be reconstructed.
What This Workaround Does and Doesn’t Solve
This doesn’t solve the original limitation.
Generative AI still cannot reliably take the CAD phones and photorealistically reinterpret them while guaranteeing that their geometry and screen pixels remain unchanged.
Instead, the workaround avoids asking it to do that.
It works particularly well for this type of hero image because the phones don’t need complicated physical interactions with the environment. They can stand on a relatively simple surface while the generated scene provides the lighting, props, and atmosphere around them.
There are still limits.
If a generated object needed to wrap around a phone, a hand needed to hold it, or the environment needed physically exact interactions with the phone geometry, we would run into many of the same problems again.
Reflections also have to be treated carefully. A CAD-generated reflection may not perfectly match the material and lighting of a generated surface. A simpler contact shadow may be more convincing than trying to reproduce every physical interaction.
For a staged product shot, though, those restrictions are manageable.
The Layers Can Remain Separate on the Website
There is another advantage to the workaround: the final result doesn’t necessarily have to become one flattened image.
The website can preserve the layers:
- Generated studio scene
- Transparent CAD phones
- Website content
That means the studio background and phones can respond independently to different screen sizes.
On a large display, more of the virtual set can be visible. On mobile, the environment can be cropped more aggressively while keeping the phones large enough for the actual app screenshots to remain readable.
It may also help solve one of the problems with the existing hero: the render can feel like an image sitting inside the webpage.
If the generated studio becomes the background of the hero section itself, with the CAD phones layered on it, the scene can become part of the page rather than a rectangular image placed on the page.
Where This Leaves the Experiment
The original experiment identified the current limitation clearly:
Generative AI can make a more cinematic-looking scene, but it cannot yet reliably reconstruct a subject while preserving it exactly.
For 5585, where the actual app screenshots and correct display geometry matter, that’s the distinction that prevents generative AI from directly replacing the CAD approach.
The workaround changes the division of responsibility instead.
Generative AI doesn’t need to preserve the phones and screenshots.
It can make the studio.
CAD doesn’t need to make the studio look cinematic.
It can preserve the phones and screenshots.
For this particular use case, combining the two after rendering rather than during generation may be the better approach.