AttributeError: Can't pickle local object 'Subprocess.start.<locals>.bootstrap

[houdinisparks]

Hi,

when i try to initialise the KinesisProducer object, it throws this error.

self.kproducer = KinesisProducer(stream_name=produce_stream_name)

Traceback:

('2018-07-28 23:58:19,660 - botocore.loaders - MainThread - DEBUG - Loading JSON file: C:\Users\662176\Documents\_Projects\_makerfaire\drone_showcase\pose_recognizer\venv\lib\site-packages\botocore\data\_retry.json
('2018-07-28 23:58:19,662 - botocore.client - MainThread - DEBUG - Registering retry handlers for service: kinesis
Traceback (most recent call last):
  File "C:\Users\662176\AppData\Roaming\JetBrains\PyCharm Community Edition 2017.3.3\helpers\pydev\pydevd.py", line 1668, in <module>
    main()
  File "C:\Users\662176\AppData\Roaming\JetBrains\PyCharm Community Edition 2017.3.3\helpers\pydev\pydevd.py", line 1662, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "C:\Users\662176\AppData\Roaming\JetBrains\PyCharm Community Edition 2017.3.3\helpers\pydev\pydevd.py", line 1072, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "C:\Users\662176\AppData\Roaming\JetBrains\PyCharm Community Edition 2017.3.3\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "C:/Users/662176/Documents/_Projects/_makerfaire/drone_showcase/pose_recognizer/src/webapp/app.py", line 27, in <module>
    consumer_stream_name="poserec_results")
  File "C:\Users\662176\Documents\_Projects\_makerfaire\drone_showcase\pose_recognizer\src\poseapp\poseapp_kinesis.py", line 41, in __init__
    self.kproducer = KinesisProducer(stream_name=produce_stream_name)
  File "C:\Users\662176\Documents\_Projects\_makerfaire\drone_showcase\pose_recognizer\venv\lib\site-packages\kinesis\producer.py", line 142, in __init__
    max_size=max_size, boto3_session=boto3_session)
  File "C:\Users\662176\Documents\_Projects\_makerfaire\drone_showcase\pose_recognizer\venv\lib\site-packages\kinesis\producer.py", line 78, in __init__
    self.start()
  File "C:\Users\662176\Documents\_Projects\_makerfaire\drone_showcase\pose_recognizer\venv\lib\site-packages\offspring\process.py", line 55, in start
    self.process.start()
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\popen_spawn_win32.py", line 65, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'Subprocess.start.<locals>.bootstrap'
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "C:\Users\662176\Documents\_Projects\_makerfaire\drone_showcase\pose_recognizer\venv\lib\site-packages\offspring\process.py", line 75, in atexit
    recursively_shutdown(cls)
  File "C:\Users\662176\Documents\_Projects\_makerfaire\drone_showcase\pose_recognizer\venv\lib\site-packages\offspring\process.py", line 71, in recursively_shutdown
('2018-07-28 23:58:19,957 - offspring.process - MainThread - DEBUG - Shutting down <kinesis.producer.AsyncProducer object at 0x000002746D1E68D0>
    recursively_shutdown(klass)
  File "C:\Users\662176\Documents\_Projects\_makerfaire\drone_showcase\pose_recognizer\venv\lib\site-packages\offspring\process.py", line 74, in recursively_shutdown
    obj.shutdown()
  File "C:\Users\662176\Documents\_Projects\_makerfaire\drone_showcase\pose_recognizer\venv\lib\site-packages\offspring\process.py", line 85, in shutdown
    self.wait()
  File "C:\Users\662176\Documents\_Projects\_makerfaire\drone_showcase\pose_recognizer\venv\lib\site-packages\offspring\process.py", line 89, in wait
    self.process.join()
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\process.py", line 123, in join
    assert self._popen is not None, 'can only join a started process'
AssertionError: can only join a started process

Will greatly appreciated any help.

[WisdomWolf]

I'm seeing a similar issue. I thought it may have been due to PyCharm debugger issues or the fact that I'm running Python 3.8, but I get the same issue outside of PyCharm and this issue shows that someone was having a similar problem way before 3.8 was released.

Traceback (most recent call last):
  File "/Users/ryanbeaman/PycharmProjects/inpatient-realtime-processor/local_consumer/kinesis_consumer.py", line 32, in <module>
    for message in consumer:
  File "/Users/ryanbeaman/.pyenv/versions/inpatient-realtime-processor/lib/python3.8/site-packages/kinesis/consumer.py", line 188, in __iter__
    self.setup_shards()
  File "/Users/ryanbeaman/.pyenv/versions/inpatient-realtime-processor/lib/python3.8/site-packages/kinesis/consumer.py", line 150, in setup_shards
    self.shards[shard_data['ShardId']] = ShardReader(
  File "/Users/ryanbeaman/.pyenv/versions/inpatient-realtime-processor/lib/python3.8/site-packages/kinesis/consumer.py", line 31, in __init__
    self.start()
  File "/Users/ryanbeaman/.pyenv/versions/inpatient-realtime-processor/lib/python3.8/site-packages/offspring/process.py", line 55, in start
    self.process.start()
  File "/Users/ryanbeaman/.pyenv/versions/3.8.1/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Users/ryanbeaman/.pyenv/versions/3.8.1/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/Users/ryanbeaman/.pyenv/versions/3.8.1/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
    return Popen(process_obj)
  File "/Users/ryanbeaman/.pyenv/versions/3.8.1/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Users/ryanbeaman/.pyenv/versions/3.8.1/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/ryanbeaman/.pyenv/versions/3.8.1/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/Users/ryanbeaman/.pyenv/versions/3.8.1/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'Subprocess.start.<locals>.bootstrap'
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/Users/ryanbeaman/.pyenv/versions/inpatient-realtime-processor/lib/python3.8/site-packages/offspring/process.py", line 75, in atexit
    recursively_shutdown(cls)
  File "/Users/ryanbeaman/.pyenv/versions/inpatient-realtime-processor/lib/python3.8/site-packages/offspring/process.py", line 71, in recursively_shutdown
    recursively_shutdown(klass)
  File "/Users/ryanbeaman/.pyenv/versions/inpatient-realtime-processor/lib/python3.8/site-packages/offspring/process.py", line 74, in recursively_shutdown
    obj.shutdown()
  File "/Users/ryanbeaman/.pyenv/versions/inpatient-realtime-processor/lib/python3.8/site-packages/offspring/process.py", line 84, in shutdown
    self.process.terminate()
  File "/Users/ryanbeaman/.pyenv/versions/3.8.1/lib/python3.8/multiprocessing/process.py", line 133, in terminate
    self._popen.terminate()
AttributeError: 'NoneType' object has no attribute 'terminate'

Process finished with exit code 1

[namioto]

I'm also saw this issue. My python version is 3.8.1.

[INFO] 2020-04-01 16:59:10,173 (kinesis.consumer) Shard reader for shardId-000000000000 does not exist, creating...
[INFO] 2020-04-01 16:59:10,173 (kinesis.consumer) shardId-000000000000 iterator arguments: {'ShardIteratorType': 'LATEST'}
[ERROR] 2020-04-01 16:59:10,207 (main) Can't pickle local object 'Subprocess.start.<locals>.bootstrap'
Traceback (most recent call last):
  File "stream_engine_v2.py", line 105, in <module>
    for message in consumer:
  File "/opt/miniconda3/envs/control-tower/lib/python3.8/site-packages/kinesis/consumer.py", line 188, in __iter__
    self.setup_shards()
  File "/opt/miniconda3/envs/control-tower/lib/python3.8/site-packages/kinesis/consumer.py", line 150, in setup_shards
    self.shards[shard_data['ShardId']] = ShardReader(
  File "/opt/miniconda3/envs/control-tower/lib/python3.8/site-packages/kinesis/consumer.py", line 31, in __init__
    self.start()
  File "/opt/miniconda3/envs/control-tower/lib/python3.8/site-packages/offspring/process.py", line 55, in start
    self.process.start()
  File "/opt/miniconda3/envs/control-tower/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/opt/miniconda3/envs/control-tower/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/opt/miniconda3/envs/control-tower/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
    return Popen(process_obj)
  File "/opt/miniconda3/envs/control-tower/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/opt/miniconda3/envs/control-tower/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/opt/miniconda3/envs/control-tower/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/opt/miniconda3/envs/control-tower/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'Subprocess.start.<locals>.bootstrap'
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/opt/miniconda3/envs/control-tower/lib/python3.8/site-packages/offspring/process.py", line 75, in atexit
    recursively_shutdown(cls)
  File "/opt/miniconda3/envs/control-tower/lib/python3.8/site-packages/offspring/process.py", line 71, in recursively_shutdown
    recursively_shutdown(klass)
  File "/opt/miniconda3/envs/control-tower/lib/python3.8/site-packages/offspring/process.py", line 74, in recursively_shutdown
    obj.shutdown()
  File "/opt/miniconda3/envs/control-tower/lib/python3.8/site-packages/offspring/process.py", line 84, in shutdown
    self.process.terminate()
  File "/opt/miniconda3/envs/control-tower/lib/python3.8/multiprocessing/process.py", line 133, in terminate
    self._popen.terminate()
AttributeError: 'NoneType' object has no attribute 'terminate'

However, it worked normally in Python 3.7.7.

[corbanvilla]

For me it was an issue on macOS but not on linux...

It works fine on Python 3.7.7 on macOS but not 3.8.5, though it does work on ubuntu on 3.8.5

[mikenac]

Confirmed failure on Mac Python 3.8. It sounds like this might describe the issue: https://chrissardegna.com/blog/multiprocessing-changes-python-3-8/

Cross posted here: borgstrom/offspring#4

[mikenac]

Left solution over at offspring project. If merged, then it's just a matter of re-pinning the offspring lib at the new version.

[Bhavikpatel576]

Any updates on this error?

[abiudrodas]

a workaround is provided in borgstrom/offspring#4, basically you need to add the following line of code to your script

multiprocessing.set_start_method("fork")