Noticed cropped results when generating with FLUX.2 klein 4B with reference images.

[matemato]

Hi, when using FLUX.2 Klein 4B with reference images. I noticed generated images were cropped/stretched. I looked at the already existing issues and found this:

@nitinmukesh There is a private marked parameter that you can pass to disable the automatic resizing:

diffusers/src/diffusers/pipelines/flux/pipeline_flux_kontext.py

Line 759 in 0454fbb

_auto_resize: bool = True,

We do the resizing by default to keep the implementation aligned with original implementation, and because there are only certain specific resolutions that Kontext was trained with. Other resolutions may work but how it affects the quality is unknown.

Originally posted by @a-r-r-o-w in #11886

Does this also affect FLUX.2 Klein pipeline and how to disable this?

EDIT: I do not see such cropping/stretching when using ComfyUI

Thanks!

[matemato]

Fixed by editing the pipeline_flux2_klein.py.

I commented out the unnecessary scaling for FLUX.2 klein and use it as (it works for only 1 reference image with this fix and input height and width has to be divisible by 16 beforehand but script can be further improved if needed):

 # 4. process images
        if image is not None and not isinstance(image, list):
            image = [image]

        # condition_images = None
        # if image is not None:
        #     for img in image:
        #         self.image_processor.check_image_input(img)

        #     condition_images = []
        #     for img in image:
        #         image_width, image_height = img.size
        #         if image_width * image_height > 1024 * 1024:
        #             img = self.image_processor._resize_to_target_area(img, 1024 * 1024)
        #             image_width, image_height = img.size

        #         multiple_of = self.vae_scale_factor * 2
        #         image_width = (image_width // multiple_of) * multiple_of
        #         image_height = (image_height // multiple_of) * multiple_of
        #         img = self.image_processor.preprocess(img, height=image_height, width=image_width, resize_mode="crop")
        #         condition_images.append(img)
        #         height = height or image_height
        #         width = width or image_width

        # height = height or self.default_sample_size * self.vae_scale_factor
        # width = width or self.default_sample_size * self.vae_scale_factor

        img = self.image_processor.preprocess(image[0], height=height, width=width, resize_mode="crop")
        condition_images = [img]

[asomoza]

HI @matemato I'm just answering because of the wording and so people don't get the wrong idea, this is not an issue because we're just following the original implementation, same as kontext, and also yours is not a fix but more of what I call a quick hack.

You just commented code and cropped the image in the case where is one image, not really a fix.

I do agree with you that the original implementation is not ideal for all use cases, probably they just assumed that people will edit an image with the same size or just use it as a real reference, which are the most common use cases, but also just cropping the image is not a real solution and I don't see a real use case for it either, if you're using the image as a reference, cropping it will make it lose the information of the parts you crop.

You're free to hack any of the pipelines as you wish and as it fits for your use case, but we should be careful with the wording since people can and will end here with a real issue and will think you posted a real fix.

Just in case people want to know what this issue is about since you didn't provide any code or images:

prompt = "change the color of the car to orange"

source	original	OP code

To demonstrate why your hack is also not the ideal, if I use the image as a reference:

prompt: "grab the car from image 1 and put it on a dirt road in the country side"

original	OP Code

As you can see, with your code, as it doesn't have the full context of the image because you cropped the front and back of the car, it tries to guess by adding something weird on the back and two licenses plates on the front.

Hope this gives a better explanation of why the original implementation was like this and why sometimes matching other implementations made by other libraries is not the best solution for all use cases.

P.S.: Also in case you didn't notice, for your solution, the best method would be to edit the image with the same aspect ratio as the original image and then crop it post inference, there's not a real need for a quick hack

[matemato]

Hi @asomoza

I apologize for not adding enough context for the issue, I agree it would be very beneficial.
I was in a hurry.

When I said fix I meant for my issue, not globally. Apologize if I was misleading. And I agree it is very much a quick hack :D.

To be clear I am doing the exact opposite. I DON'T want to crop my input because as you said I lose information. What I DO want is to get the output with the same resolution as my input but without ANY cropping/scaling. With my change to the code I am removing the unnecessary cropping. I am only using self.image_processor.preprocess to get the processed image, but no cropping is involved here since my input image already has width and height divisible by 16.

As for your examples, I get the following images with my code when using width and height the same as input:

Input	Original code	My code

For my examples this is what I'm talking about. WIth input image and prompt "Add a cartoon car in the middle of the image" I get:

Input	Original code	My code

See how the output is cropped on ALL sides (text is cropped) when using the original code vs my code.

I used this script to generate the images:

diffusers_cropping.py

Thank you for your help, hope now my issue is more clear.

[asomoza]

@matemato thanks for your reply, yeah, now I have a lot more context of your issue.

As for your examples, I get the following images with my code when using width and height the same as input:

yeah, the original code and your changes works ok when the reference image matches the source and are less than 1mp, if they are more than that, it has to downscale it to be less than 1mp while trying to maintain the aspect ratio, sadly in your very specific use case, the image gets cropped to be able do this because of the vae division by 16.

But again, as a side effect, in your code you're cropping the reference images when they don't match the aspect ratio of the inference, as the example I provided, which is to take something for the reference image and use it in a different aspect ratio inference. With the original code the % of pixel loss is a lot less than with your code, which works only for the very specific image size you're editing.

If the reference image fits within 1mp with the same aspect ratio and it is divisible by 16, with the original code there are no pixels lost.

As a better solution with diffusers for your specific use case, I would recommend to expand the image a couple of pixels or even better, to do inpainting which will give you more precision and details for your edit.

But then again, the quick hack method is also a very good solution and that's why diffusers make the pipelines simple, so people can just adapt them to their specific needs.

P.S.: Also looking at the edits of the car you posted, I think the original code one is better, with your code it changed some things it shouldn't, this could be the seed though, but I think it's because they trained the model with 1mp reference images and that's why they put that restriction (to save VRAM).

[matemato]

Yeah sorry. For the blue/orange car I used different seeds. If I understand correctly, the outputs should be the same with same seeds, since the input image is 1mp.

I will probably just use the quick hack, since the example I provided was used just to explain the issue.

Thank you for your time and suggestions, really glad we figured this one out :).

I will also explore resizing below 1mp, but maintaining the aspect ratio + divisibility by 16 if that maybe improves the output. Thanks!

[yiyixuxu]

Thanks for sharing your findings with us @matemato ! We really encourage this kind of discussion

just want to mention that modular diffusers syteem is designed for this type of customization, so you don't need to go in the pipeline code and comment out things. see our blogpost https://huggingface.co/blog/modular-diffusers and documentation https://huggingface.co/docs/diffusers/main/en/modular_diffusers/quickstart#sub-blocks

for example, to apply the "quick hack" you wanted here, you can just create your own image process step and swap it like this

from diffusers import ModularPipeline
from diffusers.modular_pipelines import ModularPipelineBlocks

class MyProcessImageStep(ModularPipelineBlocks):
    # your custom crop/resize/preprocessing logic here
    ...

# load the official blocks for flux2-klein
blocks = ModularPipeline.from_pretrained("black-forest-labs/FLUX.2-klein-4B").blocks.get_workflow("image_conditioned")

# explore the structure to find the right sub-block
# print(blocks)
# print(blocks.sub_blocks["vae_encoder.preprocess"].doc)

blocks.sub_blocks["vae_encoder.preprocess"] = MyProcessImageStep()
pipe = blocks.init_pipeline("black-forest-labs/FLUX.2-klein-4B")
pipe.load_components(torch_dtype=torch.bfloat16)

you can also publish your custom block/ppeline to the hub with proper documentation on model card so others can reuse it directly

[matemato]

This will come in handy. Thank you so much!

[sayakpaul]

And let us know if you create blocks!