Cannot fit model: DST tensor is not initialized

Question

Dragula_2022 OP

Created Aug ’22

Replies 2

Boosts 0

Views 3.1k

Participants 1

I'm a student at WGU who is using a curriculum from DataCamp, which does not share publication dates for its course materials. I have succeeded in creating and fitting models in other DataCamp courses. I'm running Python 3.10.4 with tensorflow 2.9.2 installed using the Apple Developer instructions on an M1 Mac mini running OS 12.5.1.

I am trying to fit an encoder-decoder model using Enron mail data from: https://raw.githubusercontent.com/ParakweetLabs/EmailIntentDataSet/master/src/resources/Ask0729-fixed.txt ...as a file was not provided by the instructor of the course.

I am getting this error:

---------------------------------------------------------------------------
InternalError                             Traceback (most recent call last)
Input In [54], in <cell line: 2>()
      1 # Fit the model
----> 2 model.fit(x = [input_data_prefix, input_data_suffix], y = target_data, batch_size = 1, epochs = 1, validation_split = 0.2)

File /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/keras/utils/traceback_utils.py:67, in filter_traceback.<locals>.error_handler(*args, **kwargs)
     65 except Exception as e:  # pylint: disable=broad-except
     66   filtered_tb = _process_traceback_frames(e.__traceback__)
---> 67   raise e.with_traceback(filtered_tb) from None
     68 finally:
     69   del filtered_tb

File /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/tensorflow/python/framework/constant_op.py:102, in convert_to_eager_tensor(value, ctx, dtype)
    100     dtype = dtypes.as_dtype(dtype).as_datatype_enum
    101 ctx.ensure_initialized()
--> 102 return ops.EagerTensor(value, ctx.device_name, dtype)

InternalError: Failed copying input tensor from /job:localhost/replica:0/task:0/device:CPU:0 to /job:localhost/replica:0/task:0/device:GPU:0 in order to run _EagerConst: Dst tensor is not initialized.

When tensorflow initializes, I get this output:

Metal device set to: Apple M1
2022-08-21 17:12:03.632710: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support.
2022-08-21 17:12:03.633559: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>)

...which leads me to believe I am missing some kind of argument to tensorflow around the GPU. As you can see, I've already tried reducing the batch size.

If I'm wasting everyone's time due the the outdated nature of this code, please accept my apologies. DataCamp doesn't publish system version support in their courses as they expect you only to run in their local shell environment.

Boost

Answer 1

Dragula_2022 OP

Aug ’22

I found an unanswered post on Stack Overflow with a solution that seems to work: https://stackoverflow.com/questions/72441453/tensorflow-on-mac-m1-pro-dst-tensor-is-not-initialized

The author was able to get around the error by adding this line:

tf.config.set_visible_devices([], 'GPU')

0

Answer 2

Dragula_2022 OP

Aug ’22

Ch4exercises.py.txt

I've added the python script in the attached text file. I thought the model was fitting, but when I checked it this morning, the kernel had crashed. It's currently estimating completion in just under 14 hours. That seems excessive for a 348kb file.

0