can't pickle lru_cache function with loky

[basnijholt]

The following fails:

from functools import lru_cache
import adaptive
adaptive.notebook_extension()

@lru_cache
def g(x):
    return x

def f(x):
    return g(x)

learner = adaptive.SequenceLearner(f, range(2))
runner = adaptive.Runner(learner, adaptive.SequenceLearner.done)

runner.live_info()

Related to loky issue: joblib/loky#268

This worked fine with the concurrent.futures.ProcessPoolExecutor.

[basnijholt]

This is because of cloudpipe/cloudpickle#178
and can be reproduced with

import cloudpickle
from functools import lru_cache

@lru_cache
def g(x):
    return x

dump = cloudpickle.dumps(g)
del g
g = cloudpickle.loads(dump)
g(1)

[akhmerov]

What's the use case for lru_cache'd functions?

[basnijholt]

This occurs in some simulation software I use.

For example

@lru_cache
def make_kwant_syst():
    ...
    return syst

def f(x):
    syst = make_kwant_syst()
    return conductance(x, syst)

[akhmerov]

I see. In the meantime you could hack around it by making syst global:

def make_kwant_syst():
    try:
        return syst
    except NameError:
        global syst = ...
        return syst

[basnijholt]

Also, it seems like this is blocked until at least the release of Python 3.9: cloudpipe/cloudpickle#309 (comment).

Other alternatives include using your own memorization decorator:

def memoize(f):
    memo = {}
    def helper(x):
        if x not in memo:            
            memo[x] = f(x)
        return memo[x]
    return helper

or using the concurrent.futures.ProcessPoolExecutor.

[akhmerov]

Should we then close this as an upstream bug? It seems like it will require no action either way.

[basnijholt]

I think closing it will suggest that it's fixed. I rather leave it open with the "Blocked" label attached.

[akhmerov]

But it's just not our bug, that is the reason to close it.

[basnijholt]

Closing because it is an upstream issue

• can't get lru_cache decorated function joblib/loky#268
• Unable to pickle functions that are decorated with functools.lru_cache() cloudpipe/cloudpickle#178