IOPaint Beta release #420
Replies: 19 comments 31 replies
-
|
M1 16G, Py 3.9
|
Beta Was this translation helpful? Give feedback.
-
Can you share what errors you encountered? I found that the dependency package "peft" was missing, so there was an error. It has been fixed in version
Maybe it's because I removed |
Beta Was this translation helpful? Give feedback.
-
|
I had similar errors when trying Beta 3 the other day. I asked GPT-4 about them, here's a copy/paste of the console I submitted to it on the 5th (My co Details
|
Beta Was this translation helpful? Give feedback.
-
|
Thank you. After updating to beta5, the problem has been solved. |
Beta Was this translation helpful? Give feedback.
-
|
Hi, Thank you for your work. In few months ago I bought the Windows Installer, and I would like to know if you will able to download the last version through the link I got? Thank you, |
Beta Was this translation helpful? Give feedback.
-
|
Yes, I will update it in a few days. |
Beta Was this translation helpful? Give feedback.
-
|
hi, I have already uploaded the installer for IOPaint. You can download it from the original order page through the email from lemon squeeze. If you can't find the original email, you can log in to lemon squeeze using your mailbox to find the order. 
|
Beta Was this translation helpful? Give feedback.
-
|
Any tasted steps? |
Beta Was this translation helpful? Give feedback.
-
|
I have been testing this today. Looks and works great! Perhaps a random question - but do you know of any outpainting models that can run on a system with only 16gb memory? I tried |
Beta Was this translation helpful? Give feedback.
-
|
Added a new parameter However, there seems to be an issue with running the |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for this. I updated to beta6, and the That's okay though, I do not "need" outpainting. I mainly use IOPaint to remove small things here and there, and the lama model works amazingly well for that. Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
Beta5 is definitely better - no errors on launch. However, I do get random errors with outpainting. "Match Histogram" & controlnet both give errors for example. Histogram yields a too many indicies for array error, and controlnet gives a piperclass error. Also, the results I get for the outpainting (expander) are not very good. They're more like filling with the border colors than generating like Adobe's generative fill (which I assume it should be attempting to replicate). The Controlnet developers have a fantastic thread about how to use CN to mimic the Adobe firefly generative fill here: Mikubill/sd-webui-controlnet#1464 <---- Is this the functionality that should be expected? Console w/ the histogram/controlnet errors. Details``` (venv) D:\IOPaint>iopaint start --model-dir "d:\iopaint" --enable-interactive-seg --disable-nsfw-checker --device cuda --no-half - Platform: Windows-10-10.0.19045-SP0 - Python version: 3.10.6 - torch: 2.1.2+cu118 - torchvision: 0.16.2 - Pillow: 10.2.0 - diffusers: 0.25.0 - transformers: 4.36.0 - opencv-python: 4.9.0.80 - accelerate: 0.25.0 - iopaint: 1.0.0b5 - rembg: 2.0.53 - realesrgan: 0.3.0 - gfpgan: 1.3.82024-01-08 17:43:36.894 | INFO | iopaint.plugins:build_plugins:31 - Initialize InteractiveSeg plugin During handling of the above exception, another exception occurred: Traceback (most recent call last): During handling of the above exception, another exception occurred: Traceback (most recent call last): During handling of the above exception, another exception occurred: Traceback (most recent call last): During handling of the above exception, another exception occurred: Traceback (most recent call last): During handling of the above exception, another exception occurred: Traceback (most recent call last): During handling of the above exception, another exception occurred: Traceback (most recent call last): During handling of the above exception, another exception occurred: Traceback (most recent call last): During handling of the above exception, another exception occurred: Traceback (most recent call last): During handling of the above exception, another exception occurred: Traceback (most recent call last): During handling of the above exception, another exception occurred: Traceback (most recent call last): During handling of the above exception, another exception occurred: Traceback (most recent call last): During handling of the above exception, another exception occurred: Traceback (most recent call last): During handling of the above exception, another exception occurred: Traceback (most recent call last): During handling of the above exception, another exception occurred: Traceback (most recent call last): |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
Edit: Now I have it working - must be the guidance/strength scales. I notice the strength slider isn't there all the time so perhaps an incorrect value got stuck? I'll keep playing with it and see. Love this feature!! -- So odd. I tried it again this morning, and the first time I opened the image, clicked "Dream" it gave me a generated result like the one you shared above. But then every time after that all I can get it the colors expanded to the new borders. I even just tried a completely new install on a different drive and got the same results.... I guess I'll wait and see if anyone else encounters this or for the official release. Definitely excited to try that (also "AnyText" if/when you have time to add that feature). Appreciate all your work!! |
Beta Was this translation helpful? Give feedback.
-
|
Ok, yea, so my problem was that the strength slider was set to "0" which gave the results of just blurred colors in the expanded areas. The bug though, is that the strength slider goes away if a model from the "Other Diffusers" is selected. It'll keep the value of whatever it was set on if used w/ an inpaint/outpaint/etc model previously. Perhaps it could get added on models in the "Other Diffusers" tab? The model I'm using from the "Other Diffusers" tab is the PowerPaint model. Screenshots:
|
Beta Was this translation helpful? Give feedback.
-
|
Thank you for reporting this bug. Whether to support sd_strength is determined by each model. PowerPaint should support sd_strength (the strength slider should not disappear). Fixed in beta7. |
Beta Was this translation helpful? Give feedback.
-
|
I try a model mentioned in description. Got error I get it from develop branch, not this one? |
Beta Was this translation helpful? Give feedback.
-
|
This is probably due to a network issue, beta5 did not display the correct error message, beta6 will display the correct error message (if it is indeed a network issue, you may need to set up a proxy in the command line). |
Beta Was this translation helpful? Give feedback.
-
|
On Beta 6 can't seem to load an SDXL model. Keep getting, " TypeError: StableDiffusionInpaintPipeline.init() got an unexpected keyword argument 'text_encoder_2". Red banner error on screen says axios error. Console: Details |
Beta Was this translation helpful? Give feedback.
-
|
The error in the log is due to loading sd_xl_base_1.0.safetensors as sd1.5. Currently I am using directories to differentiate between single file(ckpt/safetensors) sd/sdxl models. If the file name contains the word 'inpaint', it will be recognized as a sd-inpainting model. The relevant code can be found here.
So, all you need to do is to place |
Beta Was this translation helpful? Give feedback.
-
|
On Beta 6: Disabling the Stable Diffusion NSFW checker (command line argument --disable-nsfw-checker) doesn't work. Black images are still received on questionable content w/ an error in the console. There is mention in the console that the checker has been disabled (so the parameter was understood), but results are still being blocked w/ a msg saying, "Potential NSFW content was detected in one or more images. A black image will be returned instead. Try again with a different prompt and/or seed." Console: Details
Disabling INFO: Started server process [12768] Edit:Hmm...well seems like after letting the app download SDXL Inpaint .1 itself that model will play correclty w/ the disable NSFW blocker. Realistic Vision, on the other hand, will still put black marks and a message. And as mentioned above I can't load normal SDXL models for some reason (Error 500 - Axios) |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
Beta 7 - Bug: Stable Diffusion-xl-1.0-inpainting-0.1 Running beta 7 and if I use the SDXL .1 inpainting model (downloaded by the app itself) it either locks up at 100% with the cpu/gpu pinned, or I think on the initial run it'll finish but then if you run it again it'll lock up. There is no way to cancel/interrupt the process (as far as I can tell) so have to restart the console. The only error I've seen is a warning that pops up when the model is loaded about its config.json: My config.json is located at the path listed below.. The folder naming doesn't match the others, but not sure if that's an issue". I determined there was an error w/ this model and then deleted/let the app regrab it (using cmd hyperparameter) so it shouldn't be a corrupt download. Warning on model load is: INFO | iopaint.model_manager:init_model:39 - Loading model: diffusers/stable-diffusion-xl-1.0-inpainting-0.1
Loading pipeline components...: 71%|█▍| 5/7 [00:00<00:00, 8.60it/s]The config attributes {'decay': 0.9999, 'inv_gamma': 1.0, 'min_decay': 0.0, 'optimization_step': 37000, 'power': 0.6666666666666666, 'update_after_step': 0, 'use_ema_warmup': False} were passed to UNet2DConditionModel, but are not expected and will be ignored. Please verify your config.json configuration file.Full console info: Details
2024-01-11 07:26:24.181 | INFO | iopaint.plugins:build_plugins:31 - Initialize InteractiveSeg plugin Contents of .json (after redownload): Details
{
"_class_name": "UNet2DConditionModel",
"_diffusers_version": "0.21.0.dev0",
"_name_or_path": "valhalla/sdxl-inpaint-ema",
"act_fn": "silu",
"addition_embed_type": "text_time",
"addition_embed_type_num_heads": 64,
"addition_time_embed_dim": 256,
"attention_head_dim": [
5,
10,
20
],
"attention_type": "default",
"block_out_channels": [
320,
640,
1280
],
"center_input_sample": false,
"class_embed_type": null,
"class_embeddings_concat": false,
"conv_in_kernel": 3,
"conv_out_kernel": 3,
"cross_attention_dim": 2048,
"cross_attention_norm": null,
"decay": 0.9999,
"down_block_types": [
"DownBlock2D",
"CrossAttnDownBlock2D",
"CrossAttnDownBlock2D"
],
"downsample_padding": 1,
"dual_cross_attention": false,
"encoder_hid_dim": null,
"encoder_hid_dim_type": null,
"flip_sin_to_cos": true,
"freq_shift": 0,
"in_channels": 9,
"inv_gamma": 1.0,
"layers_per_block": 2,
"mid_block_only_cross_attention": null,
"mid_block_scale_factor": 1,
"mid_block_type": "UNetMidBlock2DCrossAttn",
"min_decay": 0.0,
"norm_eps": 1e-05,
"norm_num_groups": 32,
"num_attention_heads": null,
"num_class_embeds": null,
"only_cross_attention": false,
"optimization_step": 37000,
"out_channels": 4,
"power": 0.6666666666666666,
"projection_class_embeddings_input_dim": 2816,
"resnet_out_scale_factor": 1.0,
"resnet_skip_time_act": false,
"resnet_time_scale_shift": "default",
"sample_size": 128,
"time_cond_proj_dim": null,
"time_embedding_act_fn": null,
"time_embedding_dim": null,
"time_embedding_type": "positional",
"timestep_post_act": null,
"transformer_layers_per_block": [
1,
2,
10
],
"up_block_types": [
"CrossAttnUpBlock2D",
"CrossAttnUpBlock2D",
"UpBlock2D"
],
"upcast_attention": null,
"update_after_step": 0,
"use_ema_warmup": false,
"use_linear_projection": true
}Lastly, the in app progress bar (that sits below the positive TXT prompt enter bar) disappeared. The only time I noticed it back is when I deleted the SDXL .1 Inpaint model to redownload it to confirm it wasn't corrupt. I dunno if it's related at all, but regardless that popup has been missing often - I'll try to keep an eye on whether or not it comes back (w/o any reinstalling) and if so when it's there vs. not. The progress always shows in the console so I've been watching that. |
Beta Was this translation helpful? Give feedback.
-
I'm not sure what impact this warning has (shouldn't be a problem), here is the log when I load the 
Is "lock up" referring to being unable to perform inpainting on the webpage and constantly staying in switching mode?
fixed in |
Beta Was this translation helpful? Give feedback.
-
|
What does this "padding count" warning mean: 2024-01-08 17:48:03.487 | WARNING | iopaint.model.base:_do_outpainting:330 - padding count(2) not 0 or 3, may result in bad edge outpainting 100%| I'm getting this using the expander. Is there a setting or resolution I'm using incorrectly? Thanks for any insight!! (Full console w/ those errors are in my post earlier in this thread here: #420 (comment)) |
Beta Was this translation helpful? Give feedback.
-
|
The |
Beta Was this translation helpful? Give feedback.
-
|
Ok, cool, so long as I'm not introducing worse results (at the sections where the outpainting meets the original image) a warning doesn't matter to me. If due to resolution/AR I'm causing more noticeable "seams" then I'd love to figure out how to perfect my method or update my libraries/models. Thanks again for your work on this! Excellent tool. |
Beta Was this translation helpful? Give feedback.
-
Beta 9:By locking up I just mean it gets stuck at 100% (or just now it got stuck at 97%) with the GPU pegged at full w/o it ever going to complete. I'm not exactly sure how to reproduce but I've mostly been trialing the expander out on the X axis. Screenshot:
Testing out the "LCM Lora" feature. Get this warning each time (minor): Also: *Bug - Adjusting the steps slider while a render is in progress will cause the progress bar to change/move also. *Minor: "LoRA" (acronym) vs. "Lora" - just mentioning in case "Lora" is an oversight vs. aesthetic choice. |
Beta Was this translation helpful? Give feedback.
-
|
This issue persists to this day for me. I hadn't used the updated IOPaint in a while due to this (reverted to last Llama cleaner version). I wanted to try Brushnet, though, so I pulled the dev branch of that yesterday. I still often have this issue where the console will hit 23/23 or 20/20 (full steps), but the UI will never reach 100% and finish the generation. I'm very surprised no one else has mentioned this, so maybe it's just my setup? I dunno, but this is a clean install again and the issue persists. |
Beta Was this translation helpful? Give feedback.
-
|
Testing IOPaint 1.0.0b8, on Windows 10, I'm not sure how to make it use the GPU when working (RTX 3090). Am I doing something wrong? isn't it already fully compatible on WindowS? |
Beta Was this translation helpful? Give feedback.
-
|
Yes, install the GPU version of pytorch first. pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cu118Then specify the device when starting the service. iopaint start --device cuda |
Beta Was this translation helpful? Give feedback.
-
|
I'm Testing IOPaint 1.0.0b9 (i think - just pulled the latest master) on Fedora 39 with an AMD GPU / ROCm. I had to update 'transformers' to 4.36.2 to avoid a rust build failure when building wheels for 'tokenizers' with the rust compiler version included in the distro. No idea yet if this might break something, just thought I would mention, on Fedora 39 the included Rust compiler won't build too old a version of this dependency. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for your feedback, I have never used Fedora before...
|
Beta Was this translation helpful? Give feedback.
-
|
iopaint start --model Sanster/AnyTexttest image
AnyText.mp4 |
Beta Was this translation helpful? Give feedback.
-
|
What happened to the two folders for models? I guess having tested the beta really screwed me up for v1+ It deleted all the models I had, and no idea how to download/put them anywhere. Don't need to grab them from HF. Can't get it working now.... (Edit: Didn't delete the models - but won't recognize them regardless of folder it seems......anyway got it mostly sorted now) |
Beta Was this translation helpful? Give feedback.
-
|
I did use the model parameter before, but for some reason it wouldn't pick the models I had up regardless of where I put them (including the .cache folder default). Finally just removed the --model-dir argument (which set my model dir to d:\iopaint) and redownloaded them one by one to the .cache default. No matter what I've tried I can't get any models I have locally to get picked up. Putting them in model_dir/stable_diffusion (or _xl) no longer works for me. Tried using the config.json method w/o luck. Finally have it back to where I was but I wouldn't be able to take a local model I have (say a Dreambooth trained model) and pull it up in IOPaint. It could be something I botched on setup (I did create a venv w/ python) - but I dunno......it doesn't find anything it seems unless it downloads them. I'll try again in the morning. Love the app and install would have likely been super easy had I not installed the beta version and got comfortable w/ that first. |
Beta Was this translation helpful? Give feedback.
-
|
I get this error sometimes when attempting to download HF models using the "download --model" command. This is one I would like to try - it's the custom inpaint model Fooocus uses: https://huggingface.co/diffusionbee/fooocus_inpainting/tree/main <---diffusers version hosted (Assume this is functional) https://huggingface.co/lllyasviel/fooocus_inpaint/tree/main <---developer but don't think these files can be used https://huggingface.co/WaterKnight/fooocus-inpaint/tree/main <---Alternate conversion/hosting(?) Is it possible to install any of those? Details
|
Beta Was this translation helpful? Give feedback.
-
|
Fixed in |
Beta Was this translation helpful? Give feedback.
-
|
Is it possible to re-add the recognition of the local diffusers format? I downloaded the safetensors format from civitai, and then use converted_original_stable_diffusion_to_diffusers.py convert it to diffusers format and put it in stable_diffusion folder, cannot be recognized. |
Beta Was this translation helpful? Give feedback.
-
|
I am a bit confused about stable_diffusion_xl
├── converted_sdxl
│ ├── model_index.json
│ ├── scheduler
│ ├── text_encoder
│ ├── text_encoder_2
│ ├── tokenizer
│ ├── tokenizer_2
│ ├── unet
│ └── vae
├── sd_xl_base_1.0.safetensors
└── sd_xl_base_1.0_inpainting_0.1.safetensorsstable_diffusion
├── converted_sd15
│ ├── feature_extractor
│ ├── model_index.json
│ ├── safety_checker
│ ├── scheduler
│ ├── text_encoder
│ ├── tokenizer
│ ├── unet
│ └── vae
├── sd-v1-5-inpainting.ckpt
├── sd-v1-5-inpainting.safetensors
└── v1-5-pruned-emaonly.safetensors |
Beta Was this translation helpful? Give feedback.
-
This is strange. Can you try the |
Beta Was this translation helpful? Give feedback.
-
|
I found where to put the Stable_diffusion folder (I had been putting it in .cache hugginface/hub/etc where I'd found all the others. It seems to get picked up if I just put that folder (and XL) in the .cache root), but I'm running into this error now if I try to start iopaint or do iopaint list. As an example, one of the models I'd like to add that is not on HF is an XL Inpaint version of Juggernaut: https://civitai.com/models/245423?modelVersionId=288402 List of what I have in the Stable_Diffusion_XL folder:
With that as is I get the error (full below in details) about missing the open ai clip model. I had that before The clip model was never a problem in beta (nor do I think I did anything specifically to download it). It also seems to be in my HF/hub folder already. My command line does not have the local only flag - I'm using "iopaint start --enable-remove-bg --enable-interactive-seg --disable-nsfw-checker --device cuda --no-half "
If I move the Stable_Diffusion_XL folder out of .cache (so it's not found), list shows the following models: Details
|
Beta Was this translation helpful? Give feedback.
-
|
I spent some time doing a search, and I think this is due to the upgrade to diffusers==0.26.1. The behavior of
I made some simple modifications in |
Beta Was this translation helpful? Give feedback.
-
|
Works great now! Picks up the Stable_Diffusion folders in .\cache and I'm able to use them (IE: Juggernaut). Thanks so much for researching and implementing a fix! |
Beta Was this translation helpful? Give feedback.
-
|
Good afternoon. Can you add an option to change the rendering resolution size like in lama cleaner? |
Beta Was this translation helpful? Give feedback.
-
|
@Sanster Didn't know where else to put this, but wanted to mention that youtuber "NerdyRodent", who has a pretty good following for all things SD, posted a nice tutorial/heads-up about IOPaint yesterday: https://www.youtube.com/watch?v=G_67tX2uTWM |
Beta Was this translation helpful? Give feedback.
-
|
We have recently developed a new version of PowerPaint, drawing inspiration from Brushnet. The Online Demo has been updated accordingly. In the coming days, we intend to make the model weights available as open source. Feel free to try it out and give it a star! The PowerPaint model possesses the ability to carry out diverse inpainting tasks, such as object insertion, object removal, shape-guided object insertion, and outpainting. Notably, we have retained the cross-attention layer that BrushNet had removed, which is essential for task prompt input.
|
Beta Was this translation helpful? Give feedback.
-
|
can PowerPaint be installed in Pinokio on a mac? maybe You do not know: all Graphic designers , all Photographers and all Advertising agencies can use Powerpaint really well, but "all of them use MAC" , and never Windows. |
Beta Was this translation helpful? Give feedback.
-
|
i wish powerpaint were faster, on A100 it seems powerpaint V2 takes 20 seconds, whereas fooocus_inpainting only takes 7 seconds (less than half) was your experience the same? @zhuang2002 |
Beta Was this translation helpful? Give feedback.
-
IOPaint Beta Release
It's been a while since the last release of lama-cleaner(now renamed to IOPaint.), partly due to the fact that during this time I have released my own first macOS application OptiClean and started playing Baldur's Gate 3, which has taken up a lot of my free time. Apart from time constraints, another reason is because the code for the project has become increasingly complex, making it difficult to add new features. I was hesitant to make changes to the code, but in the end, I made the decision to completely refactor the front-end and back-end code of the project, hoping that this change will help the project moving forward.
The refactoring includes: switch to Vite, new css framework tailwindcss, new state management library zustand, new UI library shadcn/ui, using more modern python libraries such as fastapi and typer. These refactors were painful and involved significant changes, but I think they were worth it. They made the project structure clearer, easier to add new features, and more accessible for others to contribute code.
Although I am not an AIGC artist or creator, I enjoy developing tools and I wanted to make this project more practical for inpainting and outpainting needs. Below are the new features/models that have been added with this beta release:
New cli command: Batch processing
After
pip install iopaint, you can use theiopaintcommand in the command line.iopaint startis the command for starting the model service and web interface, whileiopaint runis used for batch processing. You can useiopaint start --helporiopaint run --helpto view the supported arguments.Better API doc
Thanks to fastapi, now all backend interfaces have clear API documentation. After starting the service with
iopaint start, you can accesshttp://localhost:8080/docsto view the API documentation.Support all sd1.5/sd2/sdxl normal or inpainting model from HuggingFace
Previously, lama-cleaner only supported the built-in
anything4andrealisticVision1.4SD inpaint models.iopaint start --modelnow supports automatically downloading models from HuggingFace, such asLykon/dreamshaper-8-inpainting. You can search available sd inpaint models on HuggingFace. Not only inpaint models, but you can also specify regular sd models, such asLykon/dreamshaper-8. The downloaded models can be seen and switched on the front-end page.Single file
ckpt/safetensorsmodels are also supported. IOPaint will search model understable_diffusionfolder in the--model-dir(default value ~/.cache) directory.Outpainting
You can use
Extenderto expand images, with optional directions of x, y, and xy. You can use the built-in expansion ratio or adjust the expansion area yourself.outpainting.mp4
Generate mask from segmentation model
RemoveBGandAnime Segmentationare two segmentation models. In previous versions, they could only be used to remove the background of an image. Now, the results from these two models can be used to generate masks for inpainting.Enpand or shrink mask
It is possible to extend or shrink interactive segmentation mask or removebg/anime segmentation mask.
More samplers
I have added more samplers based on this Issue A1111 <> Diffusers Scheduler mapping #4167.
LCM Lora
https://huggingface.co/docs/diffusers/main/en/using-diffusers/inference_with_lcm_lora
Latent Consistency Models (LCM) enable quality image generation in typically 2-4 steps making it possible to use diffusion models in almost real-time settings. The model will be downloaded on first use.
FreeU
https://huggingface.co/docs/diffusers/main/en/using-diffusers/freeu
FreeU is a technique for improving image quality. Different models may require different FreeU-specific hyperparameters, you can find how to adjust these parameters in the original project: https://github.com/ChenyangSi/FreeU
New Models
--model Sanster/PowerPaint-V1-stable-diffusion-inpaintingPowerPaint is a stable diffusion model optimized for inpainting/outpainting/remove_object tasks. We can control the model's results by specifying the task.--model kandinsky-community/kandinsky-2-2-decoder-inpaint--model diffusers/stable-diffusion-xl-1.0-inpainting-0.1--interactive-seg-model mobile_samLightweight SAM model, faster and requires fewer resources. Currently, there are many other variations of the SAM model. Feel free to submit a PR!Other improvement
leftorrightto switch ImageWindow installer user
For 1-click window installer users, first of all, I would like to thank you for your support. The current version of IOPaint is a beta version, and I will update the installation package after the official release of IOPaint.
Maybe a cup of coffee
During the development process, lots of coffee was consumed. If you find my project useful, please consider buying me a cup of coffee. Thank you! ❤️. https://ko-fi.com/Z8Z1CZJGY
This discussion was created from the release IOPaint Beta release.
Beta Was this translation helpful? Give feedback.
All reactions