pods get oom-killed when multiple heavy spark applications start at the same time

[maxgruber19]

Affected Stackable version

25.3.0

Current and expected behavior

when multiple spark applications get started at the same time, all of them having 300 executors, a lot of pvcs are submitted which need to be satisfied by the secret-operator. in basic configuration the secret operator container has 128mb of memory which seems to be not enough because in that case all the pods get oom-killed

Possible solution

Increase the memory limit of the secret-operator container from 128mb to 1gb

Additional context

@soenkeliebau like mentioned today, I'll try to get further details. I think this is reproducible by submitting a 3000 exec application and keeping an eye on the secret-operator ds

Environment

No response

Would you like to work on fixing this bug?

None

[lfrancke]

Thanks @maxgruber19 !

We just discussed this in another ticket which we then scoped down.
We always struggle a bit because we get tickets like this and then when we increase the resources we get tickets from other customers that the stack doesn't run anymore in resource constrained environments.

I don't want to say "we're not going to do this" but I also don't want to promise anything. I've recorded your request.

I don't know if @soenkeliebau promised anything either :)

[maxgruber19]

Hi @lfrancke thanks for the quick reply, nothing was promised by @soenkeliebau either 😄

We agreed on opening an issue, we were able to fix that by increasing the limits so thats fine for me for the moment