[MaskRCNN] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

# Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR ```sh 2020-08-13 12:35:55.764579: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally 2020-08-13 12:35:56.120940: E tensorflow/stream_executor/cuda/cuda_dnn.cc:334] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2020-08-13 12:35:56.122617: E tensorflow/stream_executor/cuda/cuda_dnn.cc:334] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR Exception in thread maskrcnn: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 754, in run self.__target(*self.__args, **self.__kwargs) File "/root/catkin_ws/src/aislam/MaskRCNN_ROS/script/action_server.py", line 188, in worker masked_image, result = maskrcnn.segment(color_image) File "/root/catkin_ws/src/aislam/MaskRCNN_ROS/script/action_server.py", line 141, in segment results = self.model.detect([image], verbose=1) File "build/bdist.linux-x86_64/egg/mrcnn/model.py", line 2526, in detect self.keras_model.predict([molded_images, image_metas, anchors], verbose=0) File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 1169, in predict steps=steps) File "/usr/local/lib/python2.7/dist-packages/keras/engine/training_arrays.py", line 294, in predict_loop batch_outs = f(ins_batch) File "/usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py", line 2715, in __call__ return self._call(inputs) File "/usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py", line 2675, in _call fetched = self._callable_fn(*array_vals) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1439, in __call__ run_metadata_ptr) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/errors_impl.py", line 528, in __exit__ c_api.TF_GetCode(self.status.status)) UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node conv1/convolution}}]] [[{{node mrcnn_detection/map/TensorArrayUnstack/range}}]] ``` # Solution ```python import tensorflow as tf config = tf.ConfigProto() # config.gpu_options.per_process_gpu_memory_fraction=0.8 config.gpu_options.allow_growth = True sess = tf.Session(config=config) ``` # References - [Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR](https://github.com/matterport/Mask_RCNN/issues/1652) - [“Failed to get convolution algorithm” problem](https://forums.developer.nvidia.com/t/failed-to-get-convolution-algorithm-problem/71462) - [使用mask rcnn心得](https://blog.csdn.net/Innovat1on/article/details/95174557)