HuggingFace Diffusers: Building Image Generation Applications with Python

# HuggingFace Diffusers: Membangun Aplikasi Generatif Gambar dengan Python Halo temen-temen, kali ini kita bahas library yang jadi fondasi hampir semua aplikasi generatif gambar berbasis Python, nama...

By Ruby Abdullah · · tutorial
diffusersstable-diffusioncontrolnetloragenerative-ai

HuggingFace Diffusers: Building Image Generation Applications with Python

Hey everyone, today we cover the library that underpins nearly every Python based image generation application: Diffusers from HuggingFace. If you have been using ComfyUI or Automatic1111 and wondered how to build image generation directly into your own application, Diffusers is the answer.

The distinction is this. ComfyUI and similar tools are applications for humans, with a visual interface. Diffusers is a library for programmers, letting you embed image generation inside a backend API, an automated pipeline, or your own product. Pleasantly, it is designed to be modular, so you can use ready-made pipelines for speed or take the components apart when you need full control.

We will cover how diffusion models work conceptually, installation, your first generation, the parameters that actually matter, image-to-image and inpainting, ControlNet, LoRA, memory optimization, and finally wrapping it in a production API.

Introduction

How Diffusion Models Work

Understand the concept first, because it makes every parameter later make sense.

Imagine taking a photo and adding a little random noise. Repeat a thousand times and eventually the photo becomes pure formless noise. A diffusion model is trained to do the reverse: given a noisy image, predict which noise to remove to get a slightly cleaner image.

When you generate, the process starts from pure random noise. The model removes noise step by step, as many steps as you specify, until a coherent image emerges. Your text prompt steers that denoising toward something matching the description.

There are three main components. The text encoder turns your prompt into vectors the model understands. The UNet (or a transformer in newer models) is the brain predicting noise at each step. The VAE translates between the small latent space where all computation happens and the full size image you actually see.

Why a latent space? Because computing at full 1024x1024 resolution is very expensive. Modern models work in a much smaller latent space, say 128x128, and the VAE expands it into a full image at the end. That optimization is what makes all of this practical.

Installation

python -m venv venv

source venv/bin/activate # Linux / Mac

venv\Scripts\activate # Windows

pip install diffusers transformers accelerate

pip install torch --index-url https://download.pytorch.org/whl/cu121

pip install pillow safetensors

For extra features we use later:

pip install peft controlnet-aux

Your First Image

import torch

from diffusers import StableDiffusionXLPipeline

pipe = StableDiffusionXLPipeline.frompretrained(

"stabilityai/stable-diffusion-xl-base-1.0",

torchdtype=torch.float16,

variant="fp16",

usesafetensors=True,

)

pipe = pipe.to("cuda")

prompt = "product photo of arabica coffee in kraft packaging, wooden background, soft studio lighting, sharp, commercial"

negative = "blurry, distorted, text, watermark, low quality"

image = pipe(

prompt=prompt,

negativeprompt=negative,

numinferencesteps=30,

guidancescale=7.0,

width=1024,

height=1024,

).images[0]

image.save("product.png")

Let us go through the parameters that genuinely matter, since these are the most misunderstood.

num
inferencesteps is how many times denoising repeats. More steps means more detail, but there is a saturation point. For SDXL, 25 to 40 steps is usually optimal. Above 50 you are burning time for no meaningful gain. guidancescale (often called CFG) controls how strictly the model follows your prompt. Low values like 3 produce more creative but less faithful images. High values like 15 follow the prompt closely but often look over-contrasted and unnatural. For SDXL, 5 to 8 is the healthy range. negativeprompt lists what you do not want. It is very effective and frequently underused. Fill it with whatever spoiled your previous results. width and height should match the model's training resolution. SDXL was trained around 1024x1024, so asking it for 512x512 actually gives worse results. Conversely, SD 1.5 was trained at 512x512 and falls apart when asked for 1024.

Reproducibility With Seeds

This matters for serious work. Without a seed, your results cannot be repeated.

import torch

generator = torch.Generator(device="cuda").manualseed(42)

image = pipe(

prompt=prompt,

generator=generator,

numinferencesteps=30,

).images[0]

Store the seed alongside the prompt in your database. Later when a client says "yesterday's version was better", you can regenerate it exactly.

Changing the Scheduler

The scheduler determines how denoising steps are scheduled. Swapping it can change quality and speed significantly without changing the model.

from diffusers import DPMSolverMultistepScheduler, EulerAncestralDiscreteScheduler

Fast and good, a common favorite

pipe.scheduler = DPMSolverMultistepScheduler.fromconfig(

pipe.scheduler.config, usekarrassigmas=True

)

An alternative with different character, more variation

pipe.scheduler = EulerAncestralDiscreteScheduler.fromconfig(pipe.scheduler.config)

print("compatible schedulers:", pipe.scheduler.compatibles)

DPMSolver with Karras sigmas produces good results at fewer steps, around 20 to 25. That saves real inference time in production.

Image-to-Image

Instead of starting from pure noise, you can start from an existing image.

import torch

from diffusers import StableDiffusionXLImg2ImgPipeline

from PIL import Image

pipe = StableDiffusionXLImg2ImgPipeline.frompretrained(

"stabilityai/stable-diffusion-xl-base-1.0",

torchdtype=torch.float16,

).to("cuda")

init = Image.open("sketch.png").convert("RGB").resize((1024, 1024))

image = pipe(

prompt="colored digital illustration, modern poster style, vivid colors",

image=init,

strength=0.6,

guidancescale=7.0,

numinferencesteps=30,

).images[0]

image.save("i2iresult.png")

The key parameter is strength, between 0 and 1. Low values like 0.2 change very little, suitable for polishing or recoloring. High values like 0.8 nearly ignore the source. To turn a sketch into a finished illustration, 0.5 to 0.7 is usually the sweet spot.

Inpainting

Inpainting replaces a specific region of an image according to a mask you supply.

import torch

from diffusers import StableDiffusionXLInpaintPipeline

from PIL import Image

pipe = StableDiffusionXLInpaintPipeline.frompretrained(

"diffusers/stable-diffusion-xl-1.0-inpainting-0.1",

torchdtype=torch.float16,

).to("cuda")

original = Image.open("room.png").convert("RGB").resize((1024, 1024))

mask = Image.open("mask.png").convert("L").resize((1024, 1024))

result = pipe(

prompt="modern brown leather sofa",

image=original,

maskimage=mask,

strength=0.99,

numinferencesteps=30,

).images[0]

result.save("inpaintresult.png")

The mask is a grayscale image where white regions get replaced and black regions are preserved. Important tip: make the mask slightly larger than the object being replaced and blur its edges so the transition is smooth. Hard edged masks often leave a visible seam.

This is a very useful pattern for business applications, such as swapping product photo backgrounds or removing unwanted objects from real estate photos.

ControlNet: Controlling Composition

ControlNet lets you control image structure precisely, using edge maps, depth maps, or pose skeletons as guidance.

import torch

from diffusers import StableDiffusionXLControlNetPipeline, ControlNetModel

from controlnetaux import CannyDetector

from PIL import Image

controlnet = ControlNetModel.frompretrained(

"diffusers/controlnet-canny-sdxl-1.0",

torchdtype=torch.float16,

)

pipe = StableDiffusionXLControlNetPipeline.frompretrained(

"stabilityai/stable-diffusion-xl-base-1.0",

controlnet=controlnet,

torchdtype=torch.float16,

).to("cuda")

Extract edges from a reference image

canny = CannyDetector()

reference = Image.open("roomlayout.png").convert("RGB").resize((1024, 1024))

edgemap = canny(reference, lowthreshold=100, highthreshold=200)

image = pipe(

prompt="minimalist scandinavian living room interior, natural light, architectural photo",

image=edgemap,

controlnetconditioningscale=0.7,

numinferencesteps=30,

).images[0]

image.save("interior.png")

controlnetconditioningscale controls how strongly the structural guidance is applied. A value of 1.0 follows the structure very tightly, sometimes rigidly. Values of 0.5 to 0.8 usually balance structure adherence with natural appearance.

This is the technique that makes generative AI genuinely usable for professional work, because you stop hoping for luck and start controlling composition.

LoRA: Custom Styles and Characters

LoRA is a small set of additional weights that changes style or adds new concepts without retraining the whole model.

import torch

from diffusers import StableDiffusionXLPipeline

pipe = StableDiffusionXLPipeline.frompretrained(

"stabilityai/stable-diffusion-xl-base-1.0",

torchdtype=torch.float16,

).to("cuda")

Load LoRA from HuggingFace or a local file

pipe.loadloraweights("path/to/lora", weightname="companystyle.safetensors",

adaptername="style")

Set its strength

pipe.setadapters(["style"], adapterweights=[0.8])

image = pipe(prompt="company mascot illustration waving").images[0]

image.save("mascot.png")

Unload when done

pipe.unloadloraweights()

You can even combine several LoRAs:

pipe.loadloraweights("stylelora", adaptername="style")

pipe.loadloraweights("characterlora", adaptername="character")

pipe.setadapters(["style", "character"], adapterweights=[0.7, 0.5])

For companies, this is the most practical way to keep all generated images consistent with brand identity. Train one LoRA on 20 to 50 on-brand images and everything afterward follows that style.

Memory Optimization

When VRAM is limited, several techniques help a great deal.

# 1. Move components to CPU when idle (big savings, somewhat slower)

pipe.enablemodelcpuoffload()

2. More aggressive per-layer offload (max savings, slowest)

pipe.enablesequentialcpuoffload()

3. Attention slicing, reduces peak memory

pipe.enableattentionslicing()

4. VAE tiling, important at large resolutions

pipe.enablevaetiling()

5. Compile the UNet for speed (PyTorch 2.0 and newer)

pipe.unet = torch.compile(pipe.unet, mode="reduce-overhead", fullgraph=True)

In practice, enablemodelcpuoffload() lets you run SDXL on an 8 GB GPU. enablesequentialcpuoffload() gets you down to 6 GB, though slowly. Do not combine torch.compile with aggressive offload, since they interfere with each other.

One warning about torch.compile: the first compilation takes one to two minutes. That is normal. What you must avoid is constantly changing image resolution, since each new input shape triggers recompilation.

Wrapping It in an API

from fastapi import FastAPI

from pydantic import BaseModel

from diffusers import StableDiffusionXLPipeline

import torch, io, base64, uuid

app = FastAPI()

Load ONCE at startup

pipe = StableDiffusionXLPipeline.frompretrained(

"stabilityai/stable-diffusion-xl-base-1.0",

torchdtype=torch.float16,

variant="fp16",

).to("cuda")

pipe.enableattentionslicing()

class Request(BaseModel):

prompt: str

negativeprompt: str = "blurry, distorted, watermark, low quality"

steps: int = 30

guidance: float = 7.0

seed: int | None = None

@app.post("/generate")

def generate(req: Request):

seed = req.seed if req.seed is not None else int(uuid.uuid4().int % (2**31))

generator = torch.Generator(device="cuda").manualseed(seed)

image = pipe(

prompt=req.prompt,

negativeprompt=req.negativeprompt,

numinferencesteps=req.steps,

guidancescale=req.guidance,

generator=generator,

).images[0]

buf = io.BytesIO()

image.save(buf, format="PNG")

return {

"seed": seed,

"image_base64": base64.b64encode(buf.getvalue()).decode(),

}

Two production notes. First, do not let requests hit a single GPU in parallel without a queue, or you will hit out of memory. Use Celery or a simple semaphore. Second, always return the seed to the client so results can be reproduced.

Tips and Best Practices

Always record the prompt, negative prompt, seed, scheduler, and every parameter in your database alongside the image. Without that you cannot improve anything systematically.

Respect the model's training resolution. SDXL around 1024, SD 1.5 around 512. Straying far produces strange anatomy and repeated composition.

For brand consistency, LoRA is far more effective than writing extremely long prompts. Long prompts often cause the model to lose focus instead.

Use ControlNet when composition matters. Fighting to control layout with prompts alone is an unnecessary struggle.

For production, consider few-step optimized models such as SDXL Turbo or LCM variants, which produce usable images in 1 to 4 steps. Quality is slightly lower but cost is dramatically better, and it is often enough for previews.

Mind the legal and ethical side. Do not train LoRAs on other people's work without permission, and do not generate images of real people without their consent. This is not only a legal matter, it is about your product's reputation.

Conclusion

Diffusers is the right entry point if you want image generation inside your own product. The summary:

Diffusion models work by gradually denoising random noise under prompt guidance, and every important parameter makes sense once you understand that process.

The parameters that truly matter are step count, guidance scale, negative prompt, model-appropriate resolution, and scheduler choice.

Image-to-image and inpainting unlock real business use cases such as replacing product backgrounds and repairing photos.

ControlNet gives precise structural control, and that is what turns generative AI from a toy into a work tool.

LoRA is the cheapest and fastest way to keep brand style consistent across all output.

Memory optimizations like CPU offload let SDXL run even on an 8 GB GPU.

Start from one real need at work, perhaps generating background variations for catalog product photos. Build a small pipeline with inpainting and ControlNet, then measure how many design hours it saves. That number is usually what gets your generative AI project approved. Happy generating.

Related Articles

Stable Diffusion Tutorial: Generative AI for Image Generation

Stable Diffusion: Tutorial Komprehensif Daftar Isi Pendahuluan Prasyarat Memahami Arsitektur Stable Diffusion 4....

Torchtune Tutorial: Fine-Tuning LLMs with PyTorch Native

Tutorial Torchtune: Fine-Tuning LLM dengan PyTorch Native Pendahuluan Torchtune adalah library resmi dari PyTorch untuk ...

Complete LlamaFactory Tutorial: All-in-One LLM Fine-Tuning Toolkit

Tutorial Lengkap LlamaFactory: Toolkit Fine-Tuning LLM All-in-One Fine-tuning Large Language Model (LLM) sering kali men...

Unsloth Tutorial: Fast and Memory-Efficient LLM Fine-Tuning

Fine-Tuning LLM Secara Efisien dengan Unsloth Dahulu, melakukan fine-tuning model bahasa besar membutuhkan server multi-...