I am using a 2022 Intel based iMac 27" with an AMD Radeon Pro GPU running OS X Monterey 12.3 Beta. I am working with Francois Chollet's GAN example: https://github.com/fchollet/deep-learning-with-python-notebooks/blob/master/chapter12_part05_gans.ipynb Here is my tensorflow-metal environment: Package Version Editable project location ---------------------------- ---------- ---------------------------- ... Send2Trash 1.8.0 sentence-transformers 2.1.0 sentencepiece 0.1.96 setuptools 47.1.0 shap 0.40.0 six 1.15.0 slicer 0.0.7 smart-open 5.2.1 sniffio 1.2.0 soupsieve 2.3.1 sympy 1.9 tables 3.6.1 tabulate 0.8.9 tbb 2021.5.0 tbb-devel 2021.5.0 tenacity 8.0.1 tensorboard 2.6.0 tensorboard-data-server 0.6.1 tensorboard-plugin-profile 2.5.0 tensorboard-plugin-wit 1.8.0 tensorflow 2.6.0 tensorflow-addons 0.14.0 tensorflow-consciousness 0.1 tensorflow-datasets 4.4.0 tensorflow-estimator 2.7.0 tensorflow-gan 2.1.0 tensorflow-hub 0.12.0 tensorflow-io-gcs-filesystem 0.22.0 tensorflow-macos 2.7.0 tensorflow-metadata 1.2.0 tensorflow-metal 0.3.0 tensorflow-probability 0.14.1 tensorflow-similarity 0.13.45 tensorflow-text 2.7.3 termcolor 1.1.0 terminado 0.12.1 testpath 0.5.0 threadpoolctl 3.0.0 tokenizers 0.10.3 toml 0.10.2 top2vec 1.0.26 torch 1.10.1 torchvision 0.11.2 tornado 6.1 tqdm 4.62.3 traitlets 5.1.0 transformers 4.11.3 typeguard 2.13.0 typing-extensions 3.7.4.3 umap-learn 0.5.1 urllib3 1.26.7 wcwidth 0.2.5 webencodings 0.5.1 websocket-client 1.2.1 Werkzeug 2.0.2 wheel 0.37.0 widgetsnbextension 4.0.0b0 wordcloud 1.8.1 wrapt 1.12.1 xgboost 1.5.1 zipp 3.6.0 (tensorflow-metal) (base) davidlaxer@x86_64-apple-darwin13 ~ % The Keras Adam optimizer crashes tensorflow. The SGD optimzer runs but the GAN generates a single color block image. The Adagrad optimizer runs but also generates a one color block image. Here is output: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2022-02-04 07:36:35.810367: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support. 2022-02-04 07:36:35.810606: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>) Metal device set to: AMD Radeon Pro 5700 XT 2022-02-04 07:36:38.833846: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled. Example output image: The code ran for 4 days and completed with no errors, however, the images were always single color blocks. I tried running with the Adam optimizer on the CPU but it was too slow.

I am running on an iMac 27" 2020 with an AMD Radeon Pro 5700 XT GPU. % pip show tensorflow-macos WARNING: Ignoring invalid distribution -umpy (/Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages) Name: tensorflow-macos Version: 2.8.0 Summary: TensorFlow is an open source machine learning framework for everyone. Home-page: https://www.tensorflow.org/ Author: Google Inc. Author-email: packages@tensorflow.org License: Apache 2.0 Location: /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages Requires: absl-py, astunparse, flatbuffers, gast, google-pasta, grpcio, h5py, keras, keras-preprocessing, libclang, numpy, opt-einsum, protobuf, setuptools, six, tensorboard, termcolor, tf-estimator-nightly, typing-extensions, wrapt Required-by: (tensorflow-metal) (base) davidlaxer@x86_64-apple-darwin13 deep-learning-with-python-notebooks % pip show tensorflow-metal WARNING: Ignoring invalid distribution -umpy (/Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages) Name: tensorflow-metal Version: 0.4.0 Summary: TensorFlow acceleration for Mac GPUs. Home-page: https://developer.apple.com/metal/tensorflow-plugin/ Author: Author-email: License: MIT License. Copyright © 2020-2021 Apple Inc. All rights reserved. Location: /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages Requires: six, wheel Required-by: Here's the code in a Jupyter Notebook: https://github.com/fchollet/deep-learning-with-python-notebooks/blob/master/chapter11_part02_sequence-models.ipynb Monterey crashed in Epoch 1 of training: callbacks = [ keras.callbacks.ModelCheckpoint("one_hot_bidir_lstm.keras", save_best_only=True) ] model.fit(int_train_ds, validation_data=int_val_ds, epochs=10, callbacks=callbacks) model = keras.models.load_model("one_hot_bidir_lstm.keras") print(f"Test acc: {model.evaluate(int_test_ds)[1]:.3f}") Epoch 1/10 2022-04-01 08:55:17.903921: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled. 2022-04-01 08:55:18.427409: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled. 2022-04-01 08:55:18.463345: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled. 2022-04-01 08:55:32.254473: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled. 2022-04-01 08:55:32.276648: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled. 3/625 [..............................] - ETA: 2:03:10 - loss: nan - accuracy: 0.5000 Kernel Panic

I am trying to build AI-Feynman https://github.com/SJ001/AI-Feynman OS X 12.4 XCode 13.4 % xcode-select -p /Library/Developer/CommandLineTools % which gfortran /Users/davidlaxer/anaconda3/envs/AI-Feynman/bin/gfortran (AI-Feynman) davidlaxer@x86_64-apple-darwin13 AI-Feynman % gfortran --version GNU Fortran (GCC) 4.8.5 Copyright (C) 2015 Free Software Foundation, Inc. GNU Fortran comes with NO WARRANTY, to the extent permitted by law. You may redistribute copies of GNU Fortran under the terms of the GNU General Public License. For more information about these matters, see the file named COPYING I am getting this error trying in link AI-Feynman: /Users/davidlaxer/anaconda3/envs/AI-Feynman/bin/x86_64-apple-darwin13.4.0-gfortran -Wall -g -arch x86_64 -Wall -g -undefined dynamic_lookup -bundle build/temp.macosx-10.9-x86_64-3.9/build/src.macosx-10.9-x86_64-3.9/aifeynman/_symbolic_regress1module.o build/temp.macosx-10.9-x86_64-3.9/build/src.macosx-10.9-x86_64-3.9/build/src.macosx-10.9-x86_64-3.9/aifeynman/fortranobject.o build/temp.macosx-10.9-x86_64-3.9/aifeynman/symbolic_regress1.o build/temp.macosx-10.9-x86_64-3.9/build/src.macosx-10.9-x86_64-3.9/aifeynman/_symbolic_regress1-f2pywrappers.o -L/Users/davidlaxer/anaconda3/envs/AI-Feynman/bin/../lib/gcc/x86_64-apple-darwin11.4.2/4.8.5 -L/Users/davidlaxer/anaconda3/envs/AI-Feynman/bin/../lib/gcc/x86_64-apple-darwin11.4.2/4.8.5/../../.. -L/Users/davidlaxer/anaconda3/envs/AI-Feynman/bin/../lib/gcc/x86_64-apple-darwin11.4.2/4.8.5/../../.. -lgfortran -o aifeynman/_symbolic_regress1.cpython-39-darwin.so ld: unsupported tapi file type '!tapi-tbd' in YAML file '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib/libSystem.tbd' for architecture x86_64 collect2: error: ld returned 1 exit status error: Command "/Users/davidlaxer/anaconda3/envs/AI-Feynman/bin/x86_64-apple-darwin13.4.0-gfortran -Wall -g -arch x86_64 -Wall -g -undefined dynamic_lookup -bundle build/temp.macosx-10.9-x86_64-3.9/build/src.macosx-10.9-x86_64-3.9/aifeynman/_symbolic_regress1module.o build/temp.macosx-10.9-x86_64-3.9/build/src.macosx-10.9-x86_64-3.9/build/src.macosx-10.9-x86_64-3.9/aifeynman/fortranobject.o build/temp.macosx-10.9-x86_64-3.9/aifeynman/symbolic_regress1.o build/temp.macosx-10.9-x86_64-3.9/build/src.macosx-10.9-x86_64-3.9/aifeynman/_symbolic_regress1-f2pywrappers.o -L/Users/davidlaxer/anaconda3/envs/AI-Feynman/bin/../lib/gcc/x86_64-apple-darwin11.4.2/4.8.5 -L/Users/davidlaxer/anaconda3/envs/AI-Feynman/bin/../lib/gcc/x86_64-apple-darwin11.4.2/4.8.5/../../.. -L/Users/davidlaxer/anaconda3/envs/AI-Feynman/bin/../lib/gcc/x86_64-apple-darwin11.4.2/4.8.5/../../.. -lgfortran -o aifeynman/_symbolic_regress1.cpython-39-darwin.so" failed with exit status 1 % find /Applications/Xcode.app -name libSystem.tbd -ls 227714500 0 lrwxr-xr-x 1 root wheel 15 Jul 29 2021 /Applications/Xcode.app/Contents/Developer/Platforms/AppleTVOS.platform/Developer/SDKs/AppleTVOS.sdk/usr/lib/libSystem.tbd -> libSystem.B.tbd 227847317 64 -rw-r--r-- 1 root wheel 170869 Feb 21 20:37 /Applications/Xcode.app/Contents/Developer/Platforms/DriverKit.platform/Developer/SDKs/DriverKit.sdk/System/DriverKit/usr/lib/libSystem.tbd 228117342 0 lrwxr-xr-x 1 root wheel 15 Jul 29 2021 /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk/usr/lib/libSystem.tbd -> libSystem.B.tbd find: /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS.simruntime/Contents/Resources/RuntimeRoot/private/var/mobile: Permission denied 227900857 0 lrwxr-xr-x 1 root wheel 15 Jul 29 2021 /Applications/Xcode.app/Contents/Developer/Platforms/WatchOS.platform/Developer/SDKs/WatchOS.sdk/usr/lib/libSystem.tbd -> libSystem.B.tbd 227872436 0 lrwxr-xr-x 1 root wheel 15 Jul 29 2021 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/lib/libSystem.tbd -> libSystem.B.tbd 228110285 0 lrwxr-xr-x 1 root wheel 15 Jul 29 2021 /Applications/Xcode.app/Contents/Developer/Platforms/WatchSimulator.platform/Developer/SDKs/WatchSimulator.sdk/usr/lib/libSystem.tbd -> libSystem.B.tbd 227845798 0 lrwxr-xr-x 1 root wheel 15 Jul 29 2021 /Applications/Xcode.app/Contents/Developer/Platforms/AppleTVSimulator.platform/Developer/SDKs/AppleTVSimulator.sdk/usr/lib/libSystem.tbd -> libSystem.B.tbd 228409847 0 lrwxr-xr-x 1 root wheel 15 Jul 29 2021 /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator.sdk/usr/lib/libSystem.tbd -> libSystem.B.tbd (AI-Feynman) davidlaxer@x86_64-apple-darwin13 AI-Feynman % diff /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib/libSystem.tbd /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/lib/libSystem.tbd

I measured a significant performance difference running the 'keras-io' example 'text_extraction_with_bert.ipynb' on Google Colab and my tensorflow_metal GPU (AMD Radeon Pro 5700 XT). Google Colab Pro w/TPU finished 3 epochs in 11 minutes, while tensorflow_metal ran for many hours for 1 epoch. So, I tried to profile the model in both environments. I was able to profile text_extraction_with_bert.ipynb on Google Colab Pro, but not on tensorflow_metal. My Mac has 128gb ... the OOM exception happened when the Python 3.8 process got to ~85GB. ResourceExhaustedError: 2 root error(s) found. (0) Resource exhausted: OOM when allocating tensor with shape[256,384,3072] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator Simple allocator [[node model/tf_bert_model/bert/encoder/layer_._6/intermediate/Gelu/add (defined at Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/transformers/models/bert/modeling_tf_bert.py:354) ]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode. [[model/tf_bert_model/bert/encoder/layer_._7/attention/output/dense/Tensordot/Prod/_632]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode. (1) Resource exhausted: OOM when allocating tensor with shape[256,384,3072] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator Simple allocator [[node model/tf_bert_model/bert/encoder/layer_._6/intermediate/Gelu/add (defined at Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/transformers/models/bert/modeling_tf_bert.py:354) ]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode. 0 successful operations. 0 derived errors ignored. [Op:__inference_train_function_24748] Function call stack: train_function -> train_function Here's the model: https://github.com/keras-team/keras-io/blob/master/examples/nlp/ipynb/text_extraction_with_bert.ipynb

I am getting a segmentation violation in Tensorflow running the Top2vec model. https://github.com/ddangelov/Top2Vec Here's my code snippet: import sys import faulthandler import numpy as np import pandas as pd import json import os import ipywidgets as widgets from IPython.display import clear_output, display from top2vec import Top2Vec faulthandler.enable(file=sys.stderr, all_threads=True) def filter_short(papers_df): papers_df["token_counts"] = papers_df["text"].str.split().map(len) papers_df = papers_df[papers_df.token_counts > 50].reset_index(drop=True) papers_df.drop('token_counts', axis=1, inplace=True) return papers_df papers_prepared_df = pd.read_feather("/Users/davidlaxer/Downloads/archive/covid19_papers_processed.feather") papers_feathered_filtered_df = filter_short(papers_prepared_df) top2vec_trained = Top2Vec(documents=papers_feathered_filtered_df.text.tolist(), embedding_model="universal-sentence-encoder", use_embedding_model_tokenizer=True, embedding_model_path="/Users/davidlaxer/Downloads/universal-sentence-encoder_4/", workers=4) Here's the stack trace: /Users/davidlaxer/tensorflow-metal/bin/python /Users/davidlaxer/Top2Vec/notebooks/test1.py 2021-12-24 08:50:04,782 - top2vec - INFO - Pre-processing documents for training /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function get_feature_names is deprecated; get_feature_names is deprecated in 1.0 and will be removed in 1.2. Please use get_feature_names_out instead. warnings.warn(msg, category=FutureWarning) 2021-12-24 08:51:37,701 - top2vec - INFO - Loading universal-sentence-encoder model at /Users/davidlaxer/Downloads/universal-sentence-encoder_4/ 2021-12-24 08:51:37.837087: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2021-12-24 08:51:37.838170: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support. 2021-12-24 08:51:37.838380: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>) Metal device set to: AMD Radeon Pro 5700 XT systemMemory: 128.00 GB maxCacheSize: 7.99 GB 2021-12-24 08:51:39.764518: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled. 2021-12-24 08:51:41,157 - top2vec - INFO - Creating joint document/word embedding 2021-12-24 08:51:41.241496: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled. Fatal Python error: Segmentation fault Thread 0x000000010c32a600 (most recent call first): File "/Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 58 in quick_execute File "/Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 598 in call File "/Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 1959 in _call_flat File "/Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 3130 in __call__ File "/Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 949 in _call File "/Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 910 in __call__ File "/Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/util/traceback_utils.py", line 150 in error_handler File "/Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/saved_model/load.py", line 701 in _call_attribute File "/Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/top2vec/Top2Vec.py", line 538 in _embed_documents File "/Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/top2vec/Top2Vec.py", line 344 in __init__ File "/Users/davidlaxer/Top2Vec/notebooks/test1.py", line 24 in <module> Process finished with exit code 139 (interrupted by signal 11: SIGSEGV) This is likely related to: https://github.com/ddangelov/Top2Vec/issues/232 However, SIGSEGV indicates some other problem(s). % python --version Python 3.8.5 % pip show tensorflow_macos Name: tensorflow-macos Version: 2.7.0 Summary: TensorFlow is an open source machine learning framework for everyone. Home-page: https://www.tensorflow.org/ Author: Google Inc. Author-email: packages@tensorflow.org License: Apache 2.0 Location: /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages Requires: absl-py, astunparse, flatbuffers, gast, google-pasta, grpcio, h5py, keras, keras-preprocessing, libclang, numpy, opt-einsum, protobuf, six, tensorboard, tensorflow-estimator, tensorflow-io-gcs-filesystem, termcolor, typing-extensions, wheel, wrapt Required-by: % pip show tensorflow_metal Name: tensorflow-metal Version: 0.3.0 Summary: TensorFlow acceleration for Mac GPUs. Home-page: https://developer.apple.com/metal/tensorflow-plugin/ Author: Author-email: License: MIT License. Copyright © 2020-2021 Apple Inc. All rights reserved. Location: /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages Requires: six, wheel Required-by:

I got this error building PyTorch 1.12 on OS X Monterey 12.4, XCode 13.4 in the linker: ld: in lib/libnnpack.a(conv1x1.py.o), section __TEXT/__const address out of range for architecture x86_64 clang: error: linker command failed with exit code 1 (use -v to see invocation) % clang --version Apple clang version 13.1.6 (clang-1316.0.21.2.5) Target: x86_64-apple-darwin21.5.0 Thread model: posix InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin % lipo -info ./build/lib/*.a Non-fat file: ./build/lib/libXNNPACK.a is architecture: x86_64 Non-fat file: ./build/lib/libasmjit.a is architecture: x86_64 Non-fat file: ./build/lib/libbenchmark.a is architecture: x86_64 Non-fat file: ./build/lib/libbenchmark_main.a is architecture: x86_64 Non-fat file: ./build/lib/libcaffe2_protos.a is architecture: x86_64 Non-fat file: ./build/lib/libclog.a is architecture: x86_64 Non-fat file: ./build/lib/libcpuinfo.a is architecture: x86_64 Non-fat file: ./build/lib/libcpuinfo_internals.a is architecture: x86_64 Non-fat file: ./build/lib/libdnnl.a is architecture: x86_64 Non-fat file: ./build/lib/libfbgemm.a is architecture: x86_64 Non-fat file: ./build/lib/libfmt.a is architecture: x86_64 Non-fat file: ./build/lib/libfoxi_loader.a is architecture: x86_64 Non-fat file: ./build/lib/libgmock.a is architecture: x86_64 Non-fat file: ./build/lib/libgmock_main.a is architecture: x86_64 Non-fat file: ./build/lib/libgtest.a is architecture: x86_64 Non-fat file: ./build/lib/libgtest_main.a is architecture: x86_64 Non-fat file: ./build/lib/libkineto.a is architecture: x86_64 Non-fat file: ./build/lib/libnnpack.a is architecture: x86_64 Non-fat file: ./build/lib/libnnpack_reference_layers.a is architecture: x86_64 Non-fat file: ./build/lib/libonnx.a is architecture: x86_64 Non-fat file: ./build/lib/libonnx_proto.a is architecture: x86_64 Non-fat file: ./build/lib/libprotobuf-lite.a is architecture: x86_64 Non-fat file: ./build/lib/libprotobuf.a is architecture: x86_64 Non-fat file: ./build/lib/libprotoc.a is architecture: x86_64 Non-fat file: ./build/lib/libpthreadpool.a is architecture: x86_64 Non-fat file: ./build/lib/libpytorch_qnnpack.a is architecture: x86_64 Non-fat file: ./build/lib/libqnnpack.a is architecture: x86_64 (base) davidlaxer@x86_64-apple-darwin13 pytorch linkText linkText Any suggestions?