The power and ethical dilemma of AI image generation models

Early discussion of AI-generated images centered on GANs, the networks that produced convincing fake faces. The tools people use today are built differently, they are far more capable, and the rules around them have started to firm up. This is an updated look at both sides.

How today’s image generators work

Most current image generators are diffusion models or close relatives of them. During training, the model sees hundreds of millions to billions of captioned images with noise added in steps and learns to undo that noise. To generate, it starts from pure random noise and removes it step by step, guided by a text prompt that has been turned into numbers by a separate text encoder. The work usually happens in a compressed “latent” space rather than on full-size pixels, which is what made these models practical on ordinary GPUs.

Newer variants change the details rather than the idea. Flow-based training lets a model take straighter paths from noise to image, so it needs fewer steps. Distillation trains a model to get close to the original’s results in a handful of steps instead of many, and it can be paired with shrinking the network itself. Some recent systems skip the separate image model altogether and generate images inside a multimodal language model, which helps with following long instructions and rendering text inside the picture.

What they are used for

In practice the heavy users are not artists making gallery pieces. They are marketing teams producing product shots and ad variations, game and film studios doing concept art, e-commerce sites generating backgrounds and lifestyle scenes, and designers mocking up interiors or packaging before anything is built. Phone makers ship generative editing in their gallery apps to remove objects, extend a frame or restyle a photo. Engineering teams also use generated images as synthetic training data for vision models, to cover rare cases that are hard to photograph.

Where the ethical problems are

Misinformation and deepfakes

A realistic image of an event that never happened costs almost nothing to make. Beyond elections and news, fake product photos, staged insurance claims and fabricated evidence all become cheaper. The reverse effect may be worse: once anything can be faked, real photos are easier to dismiss.

Consent and likeness

Many tools can put a real person’s face into any scene from a few photos. The worst use is non-consensual intimate imagery. Even commercial uses raise questions: a brand using a lookalike of a real model, or the face of someone who has died, needs consent that most workflows never ask for.

Training data and artists’ rights

The large models were trained on images scraped from the web, mostly without asking the people who made them. Artists object that their work trained a product that now competes with them, sometimes imitating a named artist’s style on request. The law is still catching up. In Getty Images v Stability AI, the UK High Court held on 4 November 2025 that the trained model was not an infringing copy of Getty’s photos, because its weights do not store or reproduce them. It did not decide whether training itself infringed, because Getty dropped that claim during the trial. Getty appealed the infringing-copy finding, and the Court of Appeal heard the appeal in November 2026; its decision was still pending when this post was updated.

Bias

A model reflects its training data. Ask for a “CEO” or a “nurse” and a model can default to narrow stereotypes of gender, age and ethnicity. These defaults matter more when generated images go into ads, stock libraries and training sets for other models, because the skew then gets copied forward.

What is being done about it

Provenance and content credentials

The most practical progress is in provenance: recording where an image came from and what was done to it. C2PA is an open standard for this. It attaches a signed record, called Content Credentials, to a file. Google’s Pixel 10 attaches Content Credentials to every JPEG photo taken with the Pixel Camera app, and Google Photos adds them to images edited with AI tools. Samsung’s Galaxy S25 adds them to photos edited with its generative AI features.

The idea is to prove what is real rather than catch every fake. The limit is that metadata can be stripped, and an image without credentials proves nothing either way.

Watermarking

Invisible watermarks go a step further by hiding a signal in the pixels themselves. Google’s SynthID is one widely deployed example, now also used by OpenAI, NVIDIA and Kakao. Since October 2026, anyone can upload an image, video or audio file to Google’s SynthID Detector to check for the mark. It only finds SynthID marks, so it cannot catch content from tools that do not use it, and a determined person can sometimes remove a watermark.

New rules

Regulation is moving from voluntary to required. Under the EU AI Act, transparency obligations have applied since 2 August 2026. Providers of generative systems must mark outputs in a machine-readable, detectable way, and deployers must disclose deepfakes of real people, places or events. A July 2026 amendment gave providers of generative systems already on the market before 2 August 2026 until 2 December 2026 to add the marking. The European Commission published a voluntary Code of Practice on marking and labeling AI-generated content on 10 June 2026 to show how to comply. Since 2 August 2025, providers placing general-purpose AI models on the EU market have also had to publish a summary of the content used for training and put in place a policy to comply with EU copyright law. Providers of models already on the market before that date have until 2 August 2027. A July 2026 amendment to the AI Act adds a ban, applying from 2 December 2026, on AI systems that generate or alter realistic intimate images or videos of an identifiable person without their consent. It covers systems built for that purpose and systems that can readily produce such content because they lack reasonable safeguards.

In the US, the TAKE IT DOWN Act was signed in May 2025. It covers AI-generated intimate images as well as real ones. Since 19 May 2026, covered platforms must remove such images within 48 hours of a valid request, with the FTC enforcing.

None of this solves the problem, but depending on where a product is offered and what role it plays, image generation now comes with concrete obligations around marking, disclosure and takedown.

Image generation on the device

Image generation no longer needs a data center. Qualcomm showed Stable Diffusion running on a Snapdragon 8 Gen 3 phone in under 0.6 seconds in late 2023, using a pruned UNet, step distillation and 8-bit weights with 16-bit activations. In 2024 the EdgeFusion paper reported a 512×512 image in two sampling steps in under a second on a Samsung Exynos NPU.

Running locally helps in three ways. Privacy improves, because a user’s photos and prompts never leave the device. There is no per-image cloud bill. And the feature keeps working offline.

The limits are real. On-device models are smaller, so they follow complex prompts less well and render fine detail and text worse than the largest cloud models. Generation drains battery and heats the device. Safety filtering and watermarking also have to run locally, and a model sitting on a device is easier for someone to extract and modify than one behind an API. Provenance signing can live on the device too, as the Pixel 10 shows by signing Content Credentials without a network connection.

Getting a large generative model to fit a phone or embedded board without losing too much quality is mostly an optimization problem: distillation, quantization, pruning and careful use of the device’s NPU. That is the kind of model work EdgeOps.AI does across vision, speech and language models on edge hardware.