How to create a custom metric in TfLearn or TensorFlow?

Question

TensorFlow offers many common evaluation metrics, but I don't know how to create my own metric. I'm building a CNN model based on AlexNet for grasp detection and I want to use the rectangle metric when evaluating the data (like in this paper: https://arxiv.org/pdf/1412.3128.pdf). Rectangle metric means both of the following conditions are satisfied:

- The grasp angle is within 30 degree of the ground truth grasp.
- The Jaccard index of the predicted grasp and the ground truth is greater than 25 percent.

So my first try is to use the model of AlexNet which is available on TFLearn (https://github.com/tflearn/tflearn/blob/master/examples/images/alexnet.py) and create a file for calculating the metric with numpy. Below is the metric file which includes incomplete code (because I'm not allowed to share), but it has the main parts as following:

def grasp_error(grasps,targets,max_angle = 30,min_overlap=0.25):
        return np.mean([np.max([grasp_classification(grasps[i],targets,max_angle,min_overlap) for i in range(grasps.shape[0])])]) #for target in targets[i]])

#compute the error of the test set
def grasp_classification(grasp,target,max_angle = 30,min_overlap = 0.25):
    ...
    if abs(np.arctan2(grasp[sinpos],grasp[cospos]) - np.arctan2(target[sinpos],target[cospos]))< (max_angle * 2./180.)*np.pi:
        if jaccard_index(grasp,target) > min_overlap:
            return 1
    return 0

# computes Jaccard index of two grasping rectangeles
def jaccard_index(grasp,target):
    ...
    return intersect/overall

I have tried adding this to the metrics.py file in Tflearn folder:

class Rectangle(Metric):
def __init__(self, name="Rechtangle"):
    super(Rectangle, self).__init__(name)
    self.tensor = None

def build(self, predictions, targets, inputs=None):
    with tf.name_scope('Rechtangle'): # <--------- name scope
         with tf.Session() as sess:
              #tf.InteractiveSession()
              prediction = predictions.eval()
              target = targets.eval()
              self.tensor = tf.convert_to_tensor(grasp_error(prediction, target, max_angle = 30,min_overlap=0.25))
    self.built = True
    self.tensor.m_name = self.name
    return self.tensor

And then using this in the end of the AlexNet:

rect_metric = tflearn.metrics.Rectangle()
network = regression(network, metric=rect_metric, optimizer='momentum',
                     loss='mean_square',
                     learning_rate=0.0005)

And I got this error:

 File "gnet.py", line 57, in <module>
    learning_rate=0.0005)   
  File "/usr/local/lib/python2.7/dist-packages/tflearn/layers/estimator.py", line 159, in regression
    metric.build(incoming, placeholder, inputs)
  File "/usr/local/lib/python2.7/dist-packages/tflearn/metrics.py", line 119, in build
    prediction = predictions.eval()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 569, in eval
    return _eval_using_default_session(self, feed_dict, self.graph, session)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3741, in _eval_using_default_session
    return session.run(tensors, feed_dict)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 778, in run
    run_metadata_ptr)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 982, in _run
    feed_dict_string, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1032, in _do_run
    target_list, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1052, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.FailedPreconditionError: Attempting to use uninitialized value is_training
     [[Node: is_training/read = Identity[T=DT_BOOL, _class=["loc:@is_training"], _device="/job:localhost/replica:0/task:0/cpu:0"](is_training)]]

There's is a problem with implementing the sess.eval() in the metrics file but it's the only way to turn a tensor to numpy array, isn't it? Please let me know if you have any ideas to fix this. Thank you very much!

EDIT: I tried another way, as recommended here: https://github.com/tflearn/tflearn/issues/207 and implementing this in the code:

def rect_metric(prediction, target, inputs):
    x = []
    sess = tf.InteractiveSession()
    with sess as default:
        pred = prediction.eval(session=sess)
        tar =  target.eval(session=sess)
        x = tf.reduce_sum(grasp_error(pred,tar))
    return x

Now the error does not show up but the training stops with this exception:

Reminder: Custom metric function arguments must be defined as: custom_metric(y_pred, y_true, x).

How to create a custom metric in TfLearn or TensorFlow?

0 Answers0