ComfyUI Model Guide - Checkpoint, LoRA, VAE, and How to Choose the Right Model

A user-focused guide to the model files in ComfyUI: what Checkpoint, LoRA, VAE, text encoder, ControlNet and upscale models are, how base, fine-tuned, merged and distilled models differ, and which model family fits which job.

October 30, 2026

ai comfyui stable-diffusion

ComfyUI Model Guide - Checkpoint, LoRA, VAE, and How to Choose the Right Model


Reqruies

ComfyUI: Based on the latest stable version
Model info: As of mid-2026 (models change rapidly)

Introduction

When you launch ComfyUI for the first time, connecting nodes is usually manageable. The real confusion starts right after. Inside the models directory, you find a long list of subfolders like checkpoint, LoRA, VAE, clip, and controlnet. On Civitai or Hugging Face, thousands of models are listed, and their names alone rarely make their differences clear. Terms like "Base," "Merge," "Turbo," "Pony," "Illustrious," and "Flux" appear everywhere, yet few places explain what they actually mean.

This article does not cover how models are trained. Instead, it explains from a user's practical perspective what role each file serves, how to choose the right one for your scenario, where to place it, and how to use it.

Question	What This Guide Covers
What is each file in the models folder?	Folder roles and node connections
What is the difference between Checkpoint, Base, and Merge?	Distinguishing a model's "origin"
Which should I use: SD1.5, SDXL, Pony, or Flux?	Family characteristics and practical use cases
How do I use LoRAs?	Strength, trigger words, and compatibility
What if errors occur or results look distorted?	Compatibility rules and causes by symptom

The Big Picture: What Goes Into One Image

A single generated image is the result of multiple model files working in sequence. Understanding which stage of the pipeline each file handles clarifies almost every folder name.

flowchart TB
    A["Text Encoder<br/>(CLIP / T5)"] --> B["Diffusion Model<br/>(UNet / DiT)"]
    L["LoRA"] -.-> B
    L -.-> A
    C["ControlNet"] -.-> B
    B --> D["VAE Decode"]
    D --> E["Upscale Model"]

Stage	What It Does	File Type
Prompt Interpretation	Translates text into numbers the model understands	Text Encoder (CLIP, T5)
Image Generation	Progressively shapes an image out of noise	Diffusion Model (UNet, DiT)
Adjustment	Modifies art style, character, or composition	LoRA, ControlNet, Embedding
Pixel Conversion	Decodes the internal latent representation into real pixels	VAE
Post-processing	Upscales resolution and refines details	Upscale Model, Detailer

The core rule is that the Diffusion Model is the center, and everything else must match that model. This rule will come up repeatedly throughout this guide.

The models Folder Map

In a default installation, models reside under ComfyUI/models/. After placing your files there, refresh the browser (R key) or restart ComfyUI, and they will appear in the node dropdown menus.

Folder	What Goes In	Nodes Used
`checkpoints`	All-in-one models (Diffusion + CLIP + VAE)	Load Checkpoint
`diffusion_models` (formerly `unet`)	Standalone Diffusion Models, GGUF files	Load Diffusion Model, UNet Loader
`text_encoders` (formerly `clip`)	Text parsing models such as CLIP and T5	Load CLIP, DualCLIP Loader
`vae`	Standalone VAE files	Load VAE
`loras`	LoRA files	Load LoRA
`controlnet`	ControlNet, T2I-Adapter	Load ControlNet Model
`embeddings`	Textual Inversion	`embedding:name` in prompts
`upscale_models`	ESRGAN-family upscalers	Load Upscale Model
`clip_vision`	Image-understanding CLIP models (for IP-Adapter, Redux)	Load CLIP Vision
`ipadapter`	IP-Adapter models (requires custom node)	IPAdapter nodes
`ultralytics/bbox`, `sams`	Face detection, segmentation (for Impact Pack)	Detector, SAM Loader

If you already use Automatic1111 or another UI, you do not need to duplicate your model files. Rename extra_model_paths.yaml.example in your ComfyUI root folder to extra_model_paths.yaml, specify your paths inside, and both tools can share the same directories. You don't have to keep duplicate copies of multi-gigabyte models.

Checkpoint: The All-in-One File

A Checkpoint is the most common file type you will download. Because it bundles the Diffusion Model, Text Encoder, and VAE into a single file, a single Load Checkpoint node outputs all three: MODEL, CLIP, and VAE. Initially, connecting just this node to a KSampler and VAE Decode is enough to produce an image.

Load Checkpoint Output	Where to Connect
MODEL	KSampler (or through a LoRA if using one)
CLIP	CLIP Text Encode (prompt input node)
VAE	VAE Decode

Checkpoint vs split models

SD1.5 and SDXL are commonly distributed as all-in-one Checkpoints. In contrast, modern models such as Flux, SD3.5, Qwen-Image, and Wan are frequently distributed in modular components. Because these models are so large, the Text Encoder alone can take up several gigabytes.

Category	All-in-One (Checkpoint)	Modular (Split)
Files	1 file	Diffusion Model + Text Encoder + VAE
Folder	`checkpoints`	`diffusion_models`, `text_encoders`, `vae`
Nodes	Load Checkpoint	Load Diffusion Model + Load CLIP + Load VAE
Advantages	Simple to set up	Allows swapping Text Encoders to different precisions or sharing across models
Examples	SD1.5, SDXL families	Flux, SD3.5, Z-Image, Qwen-Image, Wan

For modular models, loading an official workflow (example JSON) first is the easiest route. The node wiring and expected filenames are already configured for you. Model-specific templates can be found under Workflow → Browse Templates inside ComfyUI.

File Formats and Precision

Even for the exact same model, suffixes appended to the filename indicate significant differences in size and security.

Extension

Extension	Description	Recommendation
`.safetensors`	Safe format containing raw numbers (tensors) only	Always download this
`.ckpt`	Based on Python pickle; can execute arbitrary code when loaded	Avoid whenever possible
`.gguf`	Format for quantized large models (requires `ComfyUI-GGUF` custom node)	When VRAM is limited
`.pt`, `.pth`, `.bin`	Legacy formats; still found in some upscalers	Only from trusted sources

Precision (fp32, fp16, bf16, fp8)

Precision refers to how many bits are used to store each numeric value. Fewer bits result in smaller file sizes and lower VRAM usage, though a subtle loss in quality may occur.

Notation	Meaning	Characteristics
fp32	32-bit	Largest file size; rarely used
fp16, bf16	16-bit	Standard default; practically zero quality loss
fp8 (`e4m3fn`, `e5m2`)	8-bit	Half the size; differences are often imperceptible to the eye
Q8, Q5, Q4 (GGUF)	Quantized	Smaller numbers use less memory with slight, progressive quality degradation

You will also frequently encounter labels like pruned and ema in checkpoint names. A pruned file strips out training-only weights; users generating images should download this version. As a general rule of thumb, an SD1.5 fp16 model is around 2GB, while an SDXL fp16 model is around 6–7GB.

Base, Fine-tune, Merge: Where a Model Comes From

Checkpoints on Civitai fall broadly into three categories based on "how they were created." Knowing this distinction makes reading model descriptions much easier.

Type	How It Was Made	Characteristics	Examples
Base (Original)	Trained from scratch by an organization on large datasets	General-purpose; lacks specific style biases, acting as the foundation for other models	SD1.5, SDXL 1.0, Flux.1 dev
Fine-tune	Base model trained on additional curated data	Specialized toward a specific domain (anime, realism, specific subjects)	Pony Diffusion, Illustrious XL
Merge	Weighted combination of multiple models	Blended without training to achieve appealing, customized aesthetics	Most popular models on Civitai

Are Merges Bad?

Not at all. You simply need to understand their trade-offs.

Pros: They combine the strengths of multiple models, producing appealing results right out of the box.
Cons: It is often unclear which models were blended in what ratios, and prompt adherence varies between checkpoints. Licenses also become mixed.
Tip: For beginners, highly rated merges are often the easiest to use. Advanced users often prefer taking a Base or Fine-tune model and layering LoRAs for precise control.

Distilled: Turbo, Lightning, Hyper, schnell

These are models distilled to generate images in significantly fewer steps. They reduce generations that originally required 20–30 steps down to 1–8 steps.

Name	Typical Steps	CFG	Notes
SDXL Turbo, SD Turbo	1–4	Around 1	Focused on speed; lower base resolution
SDXL-Lightning, Hyper-SD	2–8	1–2	Also distributed as LoRAs that can be applied to any SDXL model
LCM	4–8	1–2	LoRA-based acceleration; more sensitive to settings
Flux.1 schnell	1–4	1	Permissive license (Apache 2.0)

While excellent for rapid composition sketching, standard models usually offer better fine detail and prompt fidelity. A common workflow is sketching compositions with a Turbo model, then switching to a standard model for the final render.

Other labels you will see

Label	Meaning
Inpainting	Models specialized for repainting selected image regions. Many standard models handle inpainting well today, making dedicated models optional
Refiner	Auxiliary models designed to polish late-stage SDXL steps. Rarely used today
Edit / Kontext	Instruction-based editing models where you provide an image and instruct "modify it like this"
NSFW / SFW	The bias of the training data. Check the model page guidelines and licensing

Model Families: Which One Should I Use

Regardless of how many model names exist, they all trace back to "which Base they originated from." Only models within the exact same lineage can share LoRAs, ControlNets, and VAEs.

Family	Base Resolution	Prompt Style	VRAM Baseline	Strengths
SD 1.5	512	Short tags + sentences	4–6GB	Lightweight with the largest ecosystem; vast library of legacy LoRAs and ControlNets
SDXL	1024	Tags + sentences	8GB+	Balanced, general-purpose; has the widest variety of choices
Pony / Illustrious / NoobAI	1024	Tag-focused	8GB+	Anime, illustration, character understanding (SDXL-based)
Flux.1 (dev, schnell)	1024+	Natural language	12GB+ (reducible via fp8, GGUF)	Realistic rendering, hands, text rendering, long prompt comprehension
SD 3.5 (Large, Medium)	1024	Natural language	8GB+	High prompt fidelity; check terms for licensing
Z-Image (Turbo)	1024+	Natural language	12–16GB	Fast realistic photos with a compact model (~8 steps); renders Korean, Chinese, and English text
Qwen-Image	1024+	Natural language	High	Exceptional text rendering inside images (including Korean)
Flux.1 Kontext, Qwen-Image-Edit	Based on input image	Instructions	High	Edits existing images based on instructions
Wan, HunyuanVideo, LTX-Video	Video	Natural language	High	Text/Image-to-Video generation

Choosing by Use Case

What You Want to Do	Recommended Choice
First time; just want to generate something working	A popular SDXL-based merge
Anime, illustration, specific characters	Illustrious / NoobAI family (or Pony)
Photorealistic people or products	Flux.1 dev family or an SDXL realism fine-tune
Rendering text inside images	Flux, Qwen-Image, or Z-Image
Fast photo-style images without an ultra-high-end GPU	Z-Image Turbo (~8 steps)
VRAM is 8GB or less	SD1.5, SDXL (fp16), or Flux GGUF Q4–Q5
Editing existing photos with instructions	Flux.1 Kontext, Qwen-Image-Edit
Short video clips	Wan family
Commercial use	Check the model card license first (Flux dev has non-commercial terms)

Practical Tips by Family

SD1.5: Being an older architecture, keep resolution around 512px and upscale afterward for high-res outputs. Anime checkpoints frequently require Clip skip 2.
SDXL: Generate around 1024x1024 (or equivalent surface areas like 1216x832, 832x1216). Straying too far from native resolution causes composition issues.
Pony: Requires prepending quality tags (such as score_9, score_8_up, score_7_up).
Illustrious / NoobAI: Primarily driven by Danbooru tags. Follow the quality tags recommended on the model card.
Z-Image Turbo: Being modular, you download the Diffusion Model, Text Encoder (Qwen-series LLM), and VAE separately. Set steps to ~8, CFG around 1, and prompt using descriptive sentences. Using the official template workflow is easiest.
Flux: Prompt using natural language sentences. Keep CFG at 1 and adjust guidance scale (~3.5) with the FluxGuidance node. Note that negative prompts are practically ignored by Flux dev.

Because recommended parameters vary by model, following the Sampler, Steps, CFG, and Clip skip recommendations on the model details page is always the fastest path to good results.

Family	Steps	CFG	Example Samplers
Z-Image Turbo	~8	1	Follow model card
SD1.5 / SDXL	20–30	5–8	`dpmpp_2m` + `karras`, `euler_ancestral`
Illustrious / Pony	25–30	4–7	`euler_ancestral`, `dpmpp_2m`
Flux dev	20–30	1 (+ FluxGuidance 3.5)	`euler` + `simple`
Flux schnell	1–4	1	`euler` + `simple`
Turbo / Lightning	2–8	1–2	Follow model card

LoRA: Small Add-on Files

A LoRA is a compact modifier file applied on top of a Checkpoint. Weighing only tens to hundreds of megabytes, it lets you introduce specific characters, artistic styles, clothing, or poses without having to download an entirely new base model.

LoRA Type	Purpose	Example
Character	Specific individuals or characters	Preserving facial features and outfits
Style	Art styles and palettes	Watercolor, specific artist styles
Concept	Specific objects, props, poses	Specialized clothing, backgrounds
Detail / Slider	Single-axis adjustments like detail, age, or expression	Slider LoRAs
Speed	Enables generation with fewer steps	Lightning, Hyper, LCM

How to use it in ComfyUI

Place the file in models/loras and refresh.
Wire the Load LoRA node downstream of the Load Checkpoint node. It accepts MODEL and CLIP inputs and outputs modified MODEL and CLIP signals.
Pass the resulting MODEL to KSampler, and CLIP to CLIP Text Encode.
To apply multiple LoRAs, chain Load LoRA nodes sequentially.

Node Input	Meaning	Starting Value
strength_model	Influence on image generation model	0.6 – 1.0
strength_clip	Influence on text interpretation	Same as strength_model, or lower if issues occur

For architectures that do not modify the Text Encoder, such as Flux or video models, use the LoraLoaderModelOnly node instead.

Essential Rules When Using LoRAs

Lineages must match. An SD1.5 LoRA cannot be used on SDXL. A Flux LoRA only works on Flux.
Check trigger words. Many LoRAs require specific trigger keywords in the prompt to activate. These are listed on the model page.
Start with lower strength. If faces look distorted or colors appear oversaturated/burned at 1.0, lower it to 0.6–0.8.
Stacking too many causes interference. If total cumulative strength exceeds 1.5–2.0, the image is prone to breaking down.
Even within SDXL, using a Pony LoRA on Illustrious (or vice versa) often results in poor compatibility. Always check which base architecture the LoRA was trained on.

VAE, Text Encoder, Embedding

VAE

The VAE converts compressed latent representations from the model into actual pixel images. Most Checkpoints bundle a built-in VAE, so downloading a separate one is usually unnecessary. Connect an external VAE only when troubleshooting specific visual flaws.

Symptom	Cause & Solution
Colors look dull and washed out	The embedded VAE is weak. Connect the model's recommended VAE using Load VAE
Pure black images on SDXL	fp16 NaN overflow issue. Use a patched VAE such as `sdxl-vae-fp16-fix`
Strange colors on Flux/SD3	Flux requires its dedicated `ae.safetensors`. It is incompatible with SD1.5 or SDXL VAEs

Text Encoder (CLIP, T5)

Translates prompt text into numerical vectors that the model understands. Different families use different combinations.

Family	Text Encoder
SD1.5	CLIP-L
SDXL	CLIP-L + CLIP-G
Flux, SD3.5	CLIP-L + T5-XXL (for long sentence comprehension)
Z-Image, Qwen-Image	Qwen-family LLMs (for multilingual comprehension, including Korean)

T5-XXL is large—around 9–10GB in fp16—so fp8 variants are commonly used when memory is tight. For SD1.5 anime models, setting a CLIP Set Last Layer node to -2 is equivalent to Clip skip 2.

Embedding (Textual Inversion)

Very small files (tens of KB) that package a specific concept into a single keyword. Place them in models/embeddings and invoke them in your prompt using embedding:filename. Negative embeddings like EasyNegative were once common, but their usage has declined with modern models and LoRAs.

ControlNet, IP-Adapter and Upscalers

ControlNet: Control the Composition

ControlNet extracts structural guides—such as outlines, depth maps, or poses—from a reference image to lock down the composition. Preprocessor nodes are available via the comfyui_controlnet_aux custom node pack.

Type	Input Feature Used	Best Used For
Canny	Edge outlines	Preserving precise silhouettes while changing art style
Depth	Depth / distance data	Maintaining spatial depth and perspective
OpenPose	Human skeletal keypoints	Explicitly directing human poses
Lineart / Scribble	Line drawings, rough sketches	Turning concept sketches into finished art
Tile	Original image textures	Preserving textures while upscaling and adding detail

ControlNet models must also match the base family. SD1.5 ControlNets cannot be used with SDXL, and Flux uses dedicated ControlNet or Union models.

IP-Adapter and Redux: Image as a Prompt

Transfers styles or subject likeness using reference images instead of text prompts. IP-Adapter requires a custom node (ComfyUI_IPAdapter_plus) and a clip_vision model; Redux serves an equivalent role in the Flux ecosystem. It is an excellent alternative when training a custom LoRA is impractical.

Upscale Models

These are ESRGAN-based models used via Load Upscale Model → Upscale Image (using Model). They scale resolution 2x to 4x instantly without additional training. Pick a model tailored to your style: anime upscalers for anime art, and photo upscalers for photography.

Method	Description	Characteristics
Upscale Model	Directly scales the pixel image	Fast and highly consistent
Latent Upscale + Resampling	Enlarges the latent and regenerates	Introduces new fine details, but high denoise values can alter shapes
Hybrid Approach	Scale with model first, then refine with low denoise (0.2–0.4)	The most widely used approach

Detailer (Impact Pack)

When faces render small in the frame and lose detail, this technique detects the face region and repaints just that segment at higher resolution. Place detection models in ultralytics/bbox and segmentation models in sams. Installing Impact Pack via ComfyUI Manager downloads the required components.

Compatibility Rules: Mix Only What Matches

Whenever an error occurs or the image looks broken, mismatched model families are almost always the cause.

Combination	Works?	Notes
SDXL Checkpoint + SDXL LoRA	Yes	Ideal pairing
SDXL Checkpoint + SD1.5 LoRA	No	Produces errors or gibberish output
Pony Checkpoint + Illustrious LoRA	Mostly awkward	Both originate from SDXL, but different tuning approaches cause poor compatibility
SD1.5 Checkpoint + SDXL ControlNet	No	Architectures must match
Flux + SDXL VAE	No	Flux requires its dedicated VAE
SDXL Checkpoint + Different SDXL VAE	Yes	Useful to fix color or NaN issues

Error logs often tell the story. Messages like mat1 and mat2 shapes cannot be multiplied or size mismatch almost always indicate that models from different families were wired together.

VRAM Guide

Because ComfyUI aggressively conserves memory by loading and unloading models on demand, workflows can often complete even when memory is tight, albeit more slowly. The following can serve as a baseline when planning your setup.

VRAM	Comfortable Usage	Fallbacks When Memory Is Tight
6–8GB	SD1.5, SDXL (fp16)	Lower resolution; use tiled VAE
12GB	General SDXL, Flux fp8, Z-Image Turbo	Flux GGUF Q5
16GB	Flux dev fp8, SD3.5 Large	Shorter frame counts for video models
24GB+	Flux fp16, Qwen-Image, Video models

When memory runs short, try: ① fp8 or GGUF variants of the model, ② an fp8 version of heavy Text Encoders like T5, and ③ the --lowvram launch flag. If an Out Of Memory (OOM) error occurs, lowering the generation resolution is the fastest fix.

Where to Get Models and How to Read a Model Page

Source	Characteristics
Hugging Face	Official releases. Base models, split components, and official workflow templates reside here
Civitai	Community models. Largest collection of Checkpoints and LoRAs, with image galleries to preview styles
ComfyUI Manager	Search and install models and custom nodes directly from within ComfyUI

Key fields to check on a Civitai model page:

Field	What to Check
Base Model	Architecture family (SD 1.5, SDXL 1.0, Pony, Illustrious, Flux.1 D, etc.). Determines compatibility with your LoRAs
Type	Checkpoint, LoRA, VAE, Embedding, ControlNet
Version Notes	Recommended Sampler, Steps, CFG, Clip skip, and VAE settings
Trigger Words	Specific keywords required to activate the LoRA
License	Commercial use permissions and restrictions
Preview Images	Click the info button on sample images to view exact prompts and generation workflows

Pre-download checklist:

Confirm the file extension is .safetensors.
Verify that the Base Model matches your target workflow family.
Select the precision (fp16, fp8, pruned) that matches your VRAM capacity.
Check the license and source. Avoid unfamiliar .ckpt files or archives bundled with .exe files.

Troubleshooting

Symptom	Possible Cause	Solution
Model does not appear in dropdown	Wrong folder location or interface not refreshed	Move to correct folder and press `R`
`mat1 and mat2 shapes...` error	Incompatible model families connected	Match the Base Model across Checkpoint, LoRA, and ControlNet
Solid black image	fp16 VAE NaN overflow, incorrect VAE	Connect a patched VAE or an fp32 VAE
Washed out / dull colors	VAE issue	Use the recommended VAE
LoRA produces no noticeable effect	Missing trigger word, strength too low, mismatched family	Add trigger word, increase strength, verify family match
Distorted, messy generation	Resolution deviates significantly from native base	Keep SDXL around 1024px
Malformed human anatomy	CFG set too high, LoRA over-application	Lower CFG and reduce LoRA strength
Negative prompt has no effect on Flux	Flux dev is designed without negative prompting	Describe desired attributes affirmatively in the main prompt
Out Of Memory	Insufficient VRAM	Reduce resolution, use fp8/GGUF, pass `--lowvram`
Image finishes too quickly and looks corrupted	Standard settings (CFG 7, 30 steps) applied to Turbo/Lightning	Adjust to 4–8 steps and CFG 1–2

Summary

Here is a recap of what we covered in this post:

Topic	Key Takeaway
models Folder	Each file type has a dedicated folder and node; refresh the UI to update dropdowns
Checkpoint	All-in-one bundle containing Diffusion + CLIP + VAE; newer architectures lean modular
File Formats	Always choose `.safetensors`, and select fp16 / fp8 / GGUF to fit your available VRAM
Base / Fine-tune / Merge	Original, specialized, or blended weights. Popular merges offer the smoothest start for beginners
Distilled	Turbo, Lightning, and schnell achieve fast results with few steps, but require lower CFG settings
Model Families	Choose among SD1.5, SDXL, Pony / Illustrious, Flux, Z-Image, Qwen-Image, and video models based on use case and VRAM
LoRA	Match the base family, check trigger words, and start with conservative strength values
Auxiliary Models	VAE, Text Encoder, ControlNet, IP-Adapter, Upscalers, and Detailers
Compatibility Rules	Components only blend within the same family; errors almost always stem from architecture mismatches

NextAI Agents for Beginners - 15. Computer Use Agent (CUA)