Virtual try-on (VTO)

Virtual try-on (VTO) shows a shopper how a product would look on them, or around them, before they buy it. The input is usually a phone camera feed or a single photo. The output is the same person, or the same room, with the product added.

Where it shows up

Most deployments fall into three groups:

• Face: eyewear, makeup, hair color and earrings. These were the first categories to work well, because faces are easy to track and most of these products sit on fixed points.
• Body and limbs: apparel, footwear, watches, rings and bracelets. Hands and feet move fast and occlude themselves, and clothes have to drape, so this group is harder.
• Space: furniture and decor placed in a room. The problem here is scale and surfaces rather than people, but the same camera and AR tools are used.

Two families of technique

Under the “virtual try-on” label there are really two different technologies. They solve different problems and have very different costs.

Live AR overlays

The first family runs in the camera view at video frame rate. A small model finds landmarks on the face, body or hand in every frame, and a renderer attaches a 3D product model to those points.

Google’s MediaPipe gives a sense of the scale involved. Its Face Landmarker returns 478 3D face landmarks and 52 expression blendshape scores per face, and supports a live stream mode. Its Pose Landmarker returns 33 body landmarks, with an optional person segmentation mask. Its Hand Landmarker returns 21 keypoints per hand. That is enough to seat a pair of glasses on the nose bridge, keep lipstick inside the lip contour, or wrap a watch around a wrist as the hand turns.

The strength of this approach is speed. Landmark models are small and run comfortably on a phone. The weakness is that the product is a rendered 3D asset laid over the image. It works for rigid items like frames, jewelry and shoes, and for cosmetics, which are drawn onto the tracked face surface. It does not handle a shirt that has to fold, stretch and tuck behind an arm.

Image-based garment try-on

The second family takes a photo of a person and a photo of a garment, and generates a new photo of that person wearing it. No 3D asset is needed, which matters for clothing retailers with thousands of products that only exist as flat catalog shots.

The early methods were convolutional networks that split the task into stages. VITON (Han et al., CVPR 2018) used a coarse-to-fine strategy: generate a rough image of the person in the target clothes, warp the flat garment to the predicted clothing shape with a thin-plate spline fitted by classical shape-context matching, then refine the clothing region by compositing the warped garment into the rough image. CP-VTON, short for “Characteristic-Preserving Virtual Try-On Network” (Wang et al., ECCV 2018), replaced that hand-crafted matching with a learned warp. Its Geometric Matching Module learns a thin-plate spline transform that bends the flat garment to fit the person’s body. Its Try-On Module then learns a mask that blends the warped garment with a rendered image of the person. That warp-then-blend pattern shaped much of the work that followed. Later models in this line, such as VITON-HD (CVPR 2021), added GAN (generative adversarial network) training and raised the output resolution to 1024×768.

From around 2023 the field moved to diffusion models, the same family behind modern image generators. TryOnDiffusion (CVPR 2023, from Google) used two UNets in parallel and let cross-attention handle the warping implicitly, rather than computing an explicit warp. IDM-VTON (ECCV 2024) builds on a pretrained diffusion inpainting model and injects garment features at two levels: high-level semantics from an image encoder and fine detail from a parallel UNet. The aim of all this is to keep logos, prints and textures intact while the garment changes shape.

Diffusion results look far more convincing than those of the earlier warp-then-blend methods, especially with complex poses and patterned fabric. The cost is compute. A diffusion model runs many denoising steps, each through a large network, so these methods typically run in the cloud on a GPU and return a still image rather than live video.

What is still hard

Fit versus looks. A generated image shows how a garment might look. It does not tell the shopper whether a medium will be tight across the shoulders. Image models have no notion of fabric stretch or garment measurements, so a convincing picture can still be the wrong size. Sizing needs body measurements and garment data, which is a separate problem from rendering.

Fabric and occlusion. Sheer, shiny and heavily patterned materials are where image models break first. Arms crossed over a jacket, hair over a collar, or a hand in a pocket all force the model to decide what is in front of what. Live AR has the same issue at smaller scale, for example a hand passing in front of glasses.

Latency. Live overlays have to track every frame. If tracking lags, glasses slide and makeup drifts off the lips, and users notice straight away. Generated try-on can tolerate seconds of delay, but not much more before people leave.

Lighting. A rendered product lit differently from the room looks pasted on. Matching the scene’s light direction and color, and keeping skin tone accurate under a warm bulb or a cold window, is as important to realism as geometry.

Skin tone and body diversity. Makeup shades must render correctly across the full range of skin tones, and garment models must work across body shapes, ages and poses. Models trained on narrow fashion datasets tend to do worst on the people least represented in those datasets. This needs testing on purpose, not assuming.

Privacy. Try-on inputs are face and body images, often taken at home. Sending them to a server means storing, securing and eventually deleting some of the most sensitive data a shopper can hand over, and in some jurisdictions face geometry is regulated as biometric data.

Why run it on the device

Running try-on on the phone, or on an in-store mirror or kiosk, helps in three ways.

• Latency: no network round trip, so tracking keeps up with the camera.
• Privacy: the face and body images never leave the device, which removes most of the data handling burden.
• Cost: the shopper’s hardware does the work, so a busy sale day does not translate into a cloud GPU bill.

Getting there means accepting trade-offs. Live AR already fits on a phone, but a production system still needs its landmark, segmentation and lighting models to share a frame budget with the renderer and the rest of the app. That usually means small architectures, int8 quantization, and running each model on whichever accelerator the chip provides: the NPU, GPU or DSP.

Garment try-on is the harder case. Full diffusion pipelines are too heavy to run live on a phone today. The practical routes are a hybrid, where on-device pose and segmentation handle the live preview and a heavier model renders a final still image, or distilled and quantized generators that take fewer steps at lower resolution and accept some loss in detail. Which trade-off is right depends on whether the product needs live video, how much visual fidelity shoppers expect, and how much data can leave the device.

The underlying building blocks are the same ones used across on-device vision: detection, tracking, pose and landmark estimation, and segmentation, compressed until they hit a frame-rate target on a specific chip. That is the kind of work EdgeOps.AI does: building vision and pose models and making them run in real time on phones and embedded boards.