25265

How can I use tensorflow metric function within keras models?

Question:

using python 3.5.2 tensorflow rc 1.1

I'm trying to use a tensorflow metric function in keras. the required function interface seems to be the same, but calling:

import pandas import numpy import tensorflow.contrib.keras as keras import tensorflow def target_function(row): return float(row[0] - row[1] < 0.5) df = pandas.DataFrame(numpy.random.rand(10000,2)) label = df.apply(target_function, axis=1) input_layer = keras.layers.Input(shape=(2,)) net = keras.layers.Dense(1)(input_layer) model = keras.models.Model(inputs=[input_layer], outputs=[net]) model.compile(loss='binary_crossentropy', optimizer='adam', metrics=[tensorflow.metrics.auc]) model.fit(df.as_matrix(), label.as_matrix(), epochs=10, validation_split=0.2, batch_size=100)

results with the error:

Using TensorFlow backend. Traceback (most recent call last): File "/Users/ophir/dev/ophir/tf_keras_metrics.py", line 49, in <module> metrics=[precision, recall, tensorflow.metrics.auc] File "/Users/ophir/anaconda3/envs/p3/lib/python3.5/site-packages/keras/engine/training.py", line 956, in compile metric_result = masked_metric_fn(y_true, y_pred, mask=masks[i]) File "/Users/ophir/anaconda3/envs/p3/lib/python3.5/site-packages/keras/engine/training.py", line 489, in masked return K.mean(score_array) File "/Users/ophir/anaconda3/envs/p3/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 1120, in mean axis = _normalize_axis(axis, ndim(x)) File "/Users/ophir/anaconda3/envs/p3/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 437, in ndim dims = x.get_shape()._dims AttributeError: 'tuple' object has no attribute 'get_shape' Process finished with exit code 1

<strong>EDIT</strong>

After incoporating the suggestion from Marcin Możejko

The code is:

import pandas import numpy import tensorflow.contrib.keras as keras import tensorflow def metric_function(y_true, y_pred): return tensorflow.metrics.precision(y_true,y_pred)[0] def target_function(row): return float(row[0] - row[1] < 0.5) df = pandas.DataFrame(numpy.random.rand(10000,2)) label = df.apply(target_function, axis=1) input_layer = keras.layers.Input(shape=(2,)) net = keras.layers.Dense(1)(input_layer) model = keras.models.Model(inputs=[input_layer], outputs=[net]) model.compile(loss='binary_crossentropy', optimizer='adam', metrics=[metric_function]) model.fit(df.as_matrix(), label.as_matrix(), epochs=10, validation_split=0.2, batch_size=100)

and the error is:

/Users/ophir/anaconda3/envs/p3/bin/python /Users/ophir/dev/ophir/tf_keras_metrics.py Train on 8000 samples, validate on 2000 samples Epoch 1/10 2017-04-04 16:05:30.959006: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations. 2017-04-04 16:05:30.959022: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations. 2017-04-04 16:05:30.959026: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations. 2017-04-04 16:05:30.959031: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations. 2017-04-04 16:05:31.124262: W tensorflow/core/framework/op_kernel.cc:1152] Failed precondition: Attempting to use uninitialized value precision/true_positives/count [[Node: precision/true_positives/count/read = Identity[T=DT_FLOAT, _class=["loc:@precision/true_positives/count"], _device="/job:localhost/replica:0/task:0/cpu:0"](precision/true_positives/count)]] Traceback (most recent call last): File "/Users/ophir/anaconda3/envs/p3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1051, in _do_call return fn(*args) File "/Users/ophir/anaconda3/envs/p3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1033, in _run_fn status, run_metadata) File "/Users/ophir/anaconda3/envs/p3/lib/python3.5/contextlib.py", line 66, in __exit__ next(self.gen) File "/Users/ophir/anaconda3/envs/p3/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status pywrap_tensorflow.TF_GetCode(status)) tensorflow.python.framework.errors_impl.FailedPreconditionError: Attempting to use uninitialized value precision/true_positives/count [[Node: precision/true_positives/count/read = Identity[T=DT_FLOAT, _class=["loc:@precision/true_positives/count"], _device="/job:localhost/replica:0/task:0/cpu:0"](precision/true_positives/count)]] During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/Users/ophir/dev/ophir/tf_keras_metrics.py", line 23, in <module> model.fit(df.as_matrix(), label.as_matrix(), epochs=10, validation_split=0.2, batch_size=100) File "/Users/ophir/anaconda3/envs/p3/lib/python3.5/site-packages/tensorflow/contrib/keras/python/keras/engine/training.py", line 1494, in fit initial_epoch=initial_epoch) File "/Users/ophir/anaconda3/envs/p3/lib/python3.5/site-packages/tensorflow/contrib/keras/python/keras/engine/training.py", line 1138, in _fit_loop outs = f(ins_batch) File "/Users/ophir/anaconda3/envs/p3/lib/python3.5/site-packages/tensorflow/contrib/keras/python/keras/backend.py", line 2245, in __call__ updated = session.run(self.outputs + [self.updates_op], feed_dict=feed_dict) File "/Users/ophir/anaconda3/envs/p3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 786, in run run_metadata_ptr) File "/Users/ophir/anaconda3/envs/p3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 994, in _run feed_dict_string, options, run_metadata) File "/Users/ophir/anaconda3/envs/p3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1044, in _do_run target_list, options, run_metadata) File "/Users/ophir/anaconda3/envs/p3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1064, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.FailedPreconditionError: Attempting to use uninitialized value precision/true_positives/count [[Node: precision/true_positives/count/read = Identity[T=DT_FLOAT, _class=["loc:@precision/true_positives/count"], _device="/job:localhost/replica:0/task:0/cpu:0"](precision/true_positives/count)]] Caused by op 'precision/true_positives/count/read', defined at: File "/Users/ophir/dev/ophir/tf_keras_metrics.py", line 21, in <module> model.compile(loss='binary_crossentropy', optimizer='adam', metrics=[metric_function]) File "/Users/ophir/anaconda3/envs/p3/lib/python3.5/site-packages/tensorflow/contrib/keras/python/keras/engine/training.py", line 958, in compile metric_result = masked_metric_fn(y_true, y_pred, mask=masks[i]) File "/Users/ophir/anaconda3/envs/p3/lib/python3.5/site-packages/tensorflow/contrib/keras/python/keras/engine/training.py", line 487, in masked score_array = fn(y_true, y_pred) File "/Users/ophir/dev/ophir/tf_keras_metrics.py", line 8, in metric_function return tensorflow.metrics.precision(y_true,y_pred)[0] File "/Users/ophir/anaconda3/envs/p3/lib/python3.5/site-packages/tensorflow/python/ops/metrics_impl.py", line 1377, in precision updates_collections=None, name=None) File "/Users/ophir/anaconda3/envs/p3/lib/python3.5/site-packages/tensorflow/python/ops/metrics_impl.py", line 1274, in true_positives updates_collections) File "/Users/ophir/anaconda3/envs/p3/lib/python3.5/site-packages/tensorflow/python/ops/metrics_impl.py", line 1211, in _count_condition count = _create_local('count', shape=[]) File "/Users/ophir/anaconda3/envs/p3/lib/python3.5/site-packages/tensorflow/python/ops/metrics_impl.py", line 197, in _create_local validate_shape=validate_shape) File "/Users/ophir/anaconda3/envs/p3/lib/python3.5/site-packages/tensorflow/python/ops/variables.py", line 197, in __init__ expected_shape=expected_shape) File "/Users/ophir/anaconda3/envs/p3/lib/python3.5/site-packages/tensorflow/python/ops/variables.py", line 316, in _init_from_args self._snapshot = array_ops.identity(self._variable, name="read") File "/Users/ophir/anaconda3/envs/p3/lib/python3.5/site-packages/tensorflow/python/ops/gen_array_ops.py", line 1343, in identity result = _op_def_lib.apply_op("Identity", input=input, name=name) File "/Users/ophir/anaconda3/envs/p3/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 768, in apply_op op_def=op_def) File "/Users/ophir/anaconda3/envs/p3/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2336, in create_op original_op=self._default_original_op, op_def=op_def) File "/Users/ophir/anaconda3/envs/p3/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1228, in __init__ self._traceback = _extract_stack() FailedPreconditionError (see above for traceback): Attempting to use uninitialized value precision/true_positives/count [[Node: precision/true_positives/count/read = Identity[T=DT_FLOAT, _class=["loc:@precision/true_positives/count"], _device="/job:localhost/replica:0/task:0/cpu:0"](precision/true_positives/count)]] Exception ignored in: <bound method BaseSession.__del__ of <tensorflow.python.client.session.Session object at 0x1140626d8>> Traceback (most recent call last): File "/Users/ophir/anaconda3/envs/p3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 595, in __del__ AttributeError: 'NoneType' object has no attribute 'TF_NewStatus' Process finished with exit code 1

Answer1:

Ok - I think I found an error. This funtion returns a tuple with (auc, other stuff). You need to overcome this by defying your own auc by:

def auc_tf_metric(y_true, y_pred): return tensorflow.metrics.auc(y_true, y_pred)[0]

Answer2:

write by K.switch import keras.backend as K

def k_mean_iou(NUM_CLASSES): """ assume 0 is background for labels and prediction labels,prediction with shape of [batch,height,width,class_number] """ def switch_mean_iou(labels, predictions): mean_iou = K.variable(0.0) seen_classes = K.variable(0.0) for c in range(1,NUM_CLASSES): labels_c = K.cast(K.equal(labels, c), K.floatx()) pred_c = K.cast(K.equal(predictions, c), K.floatx()) labels_c_sum = K.sum(labels_c) pred_c_sum = K.sum(pred_c) intersect = K.sum(labels_c*pred_c) union = labels_c_sum + pred_c_sum - intersect iou = intersect / union condition = K.equal(union, 0) mean_iou = K.switch(condition, mean_iou, mean_iou+iou) seen_classes = K.switch(condition, seen_classes, seen_classes+1) mean_iou = K.switch(K.equal(seen_classes, 0), mean_iou, mean_iou/seen_classes) return mean_iou return switch_mean_iou model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=k_mean_iou(class_number))

Answer3:

Regarding the new Error: Generally, FailedPreconditionsErrors occur when tf.global_variables_initializer() is ran in the session. Be sure you haven't ommited it.

Answer4:

You can wrap tf.metrics (e.g. tf.metrics.auc) into keras.metrics.

See my answer to <a href="https://stackoverflow.com/a/50527423/1099817" rel="nofollow">How to use TensorFlow metrics in Keras</a>:

Recommend

  • Tensorflow: open a PIL.Image?
  • Why does this recursive copy function copy all files into every directory above the correct one?
  • No provider for Http StaticInjectorError
  • php printf adds number to output
  • Get String from asynctask android
  • Nested foreach loops to add DirectoryInfo.GetDirectories and GetFiles to an array?
  • listView filter mistake
  • How to create global constant variables in C# for the entire application
  • Keras Multi GPU example gives ResourceExhaustedError
  • How can I access injected Grails beans in an abstract class method?
  • compare two arrays based on length: skip empty values
  • c# - how to improve this “file explorer” startup speed?
  • Informatica - Concatenate all rows of 1 column to a single value
  • undirected graphs algorithms
  • python numpy on solaris, blas slow or not linked?
  • dot 2 pixels together
  • So we've got MEF working with MVC4, how about the convention based model?
  • Msql: Counting growth over time
  • PowerShell IComparable with subclasses
  • best layer for placing dependency injection related code in layered proeject
  • Why is my database not being updated when someone visits my website?
  • Go template remove the last comma in range loop
  • How to get attributes from parent?
  • integrating GWT with CDI (Seam/Weld)
  • Assembly Language Absolute addresses and segment registers
  • Using CUDA 8.0 with GCC 6.x - bad function overloading complaint
  • Dependency Injection - Choose DLL and class implementation at runtime through configuration file
  • Trying to delete files older than X number of days on insert in .NET MVC
  • Einsum optimize fails for basic operation
  • Invoking a controller's action by button in View without redirecting to any view
  • OpenCV Python: Draw minAreaRect ( RotatedRect not implemented)
  • Problems with matplotlib.pyplot.xticks()
  • unable to get jsonEncode in magento2
  • Django Haystack Rebuild Index
  • To display the title for the current loaction in map in iphone
  • Akka Routing: Reply's send to router ends up as dead letters
  • AT Commands to Send SMS not working in Windows 8.1
  • Rails 2: use form_for to build a form covering multiple objects of the same class
  • How do I configure my settings file to work with unit tests?
  • Is it possible to post an object from jquery to bottle.py?