7

I have written a custom layer in keras. in part of this custom layer lets say I have a matrix like this:

c = tf.cast(tf.nn.top_k(tf.nn.top_k(n, tf.shape(n)[1])[1][:, ::-1], tf.shape(n)[1])[1][:, ::-1], dtype=tf.float32)

My question is that How can I track the values of result of this per epoch?

for example, if I have 20 epoch, I need to have 20 of this matrix saved in a csv file.

(I know how to save the weights of the model but this one is the result of a middle layer operation and I need to keep track of this matrix).

what I have done:

This is the structure of my layer:

class my_layer(Layer):
    def __init__(self, topk, ctype, **kwargs):
    self.x_prev = None
    self.topk_mat = None

   def call(self, x):
     'blah blah'

   def get_config(self):
      'blah blah'

   def k_comp_tanh(self,x, f=6):
     'blah blah'
      if self.topk_mat is None:
            self.topk_mat = self.add_weight(shape=(20, 25),
                                          initializer='zeros',
                                          trainable=False,
                                          # dtype=tf.float32,
                                          name='topk_mat')

     c = tf.cast(tf.nn.top_k(tf.nn.top_k(n, tf.shape(n)[1])[1][:, ::-1], tf.shape(n)[1])[1][:, ::-1], dtype=tf.float32)
     self.topk_mat.assign(c)

Code for building the model and fitting on the data:

class AutoEncoder(object):
def __init__(self, input_size, dim, comp_topk=None, ctype=None, save_model='best_model'):
    self.input_size = input_size
    self.dim = dim
    self.comp_topk = comp_topk
    self.ctype = ctype
    self.save_model = save_model
    self.build()

def build(self):
    input_layer = Input(shape=(self.input_size,))
    encoded_layer = Dense(self.dim, activation=act, kernel_initializer="glorot_normal", name="Encoded_Layer")
    encoded = encoded_layer(input_layer)
    encoder_model = Model(outputs=encoded, inputs=input_layer)
    encoder_model.save('pathto/encoder_model')

    self.encoded_instant = my_layer(self.comp_topk, self.ctype)
    encoded = self.encoded_instant(encoded)
    decoded = Dense_tied(self.input_size, activation='sigmoid',tied_to=encoded_layer, name='Decoded_Layer')(encoded)

    # this model maps an input to its reconstruction
    self.autoencoder = Model(outputs=decoded, inputs=input_layer)

    # this model maps an input to its encoded representation
    self.encoder = Model(outputs=encoded, inputs=input_layer)

    # create a placeholder for an encoded input
    encoded_input = Input(shape=(self.dim,))
    # retrieve the last layer of the autoencoder model
    decoder_layer = self.autoencoder.layers[-1]
    # create the decoder model
    self.decoder = Model(outputs=decoder_layer(encoded_input), inputs=encoded_input)

def fit(self, train_X, val_X, nb_epoch=50, batch_size=100, contractive=None):
    import tensorflow as tf
    optimizer = Adam(lr=0.0005)

    self.autoencoder.compile(optimizer=optimizer, loss='binary_crossentropy') # kld, binary_crossentropy, mse

    cbk = tf.keras.callbacks.LambdaCallback(
        on_epoch_begin=lambda epoch, logs: np.savetxt("foo.csv", tf.keras.backend.eval(self.encoded_instant.topk_mat), delimiter=","))
    self.autoencoder.fit(train_X[0], train_X[1],
                    epochs=nb_epoch,
                    batch_size=batch_size,
                    shuffle=True,
                    validation_data=(val_X[0], val_X[1]),
                    callbacks=[
                                ReduceLROnPlateau(monitor='val_loss', factor=0.2, patience=3, min_lr=0.01),
                                EarlyStopping(monitor='val_loss', min_delta=1e-5, patience=5, verbose=1, mode='auto'),
                                cbk,
                  save_best_only=True, mode='auto')
                                CustomModelCheckpoint(custom_model=self.encoder, filepath="pathtocheckpoint/{epoch}.hdf5",save_best_only=True,  monitor='val_loss', mode='auto')
                    ]
                    )

    return self


cbk = tf.keras.callbacks.LambdaCallback(
    on_epoch_begin=lambda epoch, logs: np.savetxt("mycsvtopk.csv", tf.keras.backend.eval(my_layer.topk_mat, delimiter=",")))
                                       )
self.autoencoder.fit(train_X[0], train_X[1],
                epochs=nb_epoch,
                batch_size=batch_size,
                shuffle=True,
                validation_data=(val_X[0], val_X[1]),
                callbacks=[cbk,CustomModelCheckpoint(custom_model=self.encoder, filepath="path_to_file/{epoch}.hdf5",save_best_only=True,  monitor='val_loss', mode='auto')
                    ]
                    )

and this is where I call the Autoencoder class

ae = AutoEncoder(n_vocab, args.n_dim, comp_topk=args.comp_topk, ctype=args.ctype, save_model=args.save_model)
ae.fit([X_train_noisy, X_train], [X_val_noisy, X_val], nb_epoch=args.n_epoch, \
        batch_size=args.batch_size, contractive=args.contractive)

It raises error:

tensorflow.python.framework.errors_impl.FailedPreconditionError: Attempting to use uninitialized value mylayer_1/topk_mat
     [[{{node _retval_mylayer_1/topk_mat_0_0}} = _Retval[T=DT_FLOAT, index=0, _device="/job:localhost/replica:0/task:0/device:CPU:0"](mylayer_1/topk_mat)]]
Exception TypeError: TypeError("'NoneType' object is not callable",) in <bound method Session.__del__ of <tensorflow.python.client.session.Session object at 0x7f56ae01bc50>> ignored

The examples I see with CustomCallback all are related to metric already model is aware of like loss, accuracy, ... What I have done above based on @Jhadi idea is to save the result of this in one variable initially initialized with None, and then in the fitting part pass this variable to save it in a csv format. This seems has to work though I am getting this error and have tried many ways to fix it but no success. It seems to me like a Keras library issue.

sariii
  • 1,725
  • 3
  • 20
  • 44
  • 1
    I'm not sure I understand what you are trying in the first line. It seems to me both TopK operations are equivalent to [`tf.argsort`](https://www.tensorflow.org/api_docs/python/tf/argsort) with `direction='DESCENDING'`, that is, you could just have `tf.dtypes.cast(tf.argsort(tf.argsort(n, order='DESCENDING'), order='DESCENDING'), dtype=tf.float32)`, is that correct? – jdehesa Jun 09 '20 at 10:33
  • 1
    In any case, about the question, you could have a new layer variable to store that information (again, `add_weight` with `trainable=False`), you'd just have to assign the value to it on each iteration. As before, though, this variable would need to have a fixed size. Other than that, I don't see an issue with your current approach (although `my_layer` should be the name of the layer instance, not the layer class). – jdehesa Jun 09 '20 at 10:35
  • Yes, that is correct that I could have done that with `tf.argsort`, I remember I had some version issues that I could not use `tf.argsort`. – sariii Jun 09 '20 at 20:57
  • So you mean I need to declare a hypothetical variable and save the result of this operation on it, then I can have the same thing to get it saved. Though, is it doable to save it in a CSV format? I mean I need to access that later when the training was done – sariii Jun 09 '20 at 21:02
  • Actually, I tried what you said. there is three issues: one is that I am using tf1 which does not support `tf.print`. Also, I need to have it saved in a file instead of printing as I can not query it. The third thing, is there any way I can save it per epoch rather than batch (the batch size is fixed but I need the final value per epoch or any way I can keep track which value is the final batch of each epoch). Do you know any approach such that I can meet these three thing please? – sariii Jun 10 '20 at 03:21
  • 1
    About using epochs instead of batches, just use one of the `on_epoch_*` callbacks in [`Callback`](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/Callback) or [`LambdaCallback`](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/LambdaCallback). I think I would store the `c` matrix in a non-trainable variable, then in the callback [`eval`](https://www.tensorflow.org/api_docs/python/tf/keras/backend/eval) that variable and store its value in a CSV file. – jdehesa Jun 10 '20 at 09:06
  • @jdehesa Thank you so much for the links. I went through them and learned some stuff. Accordingly, I have updated my question. I feel this is more trickier than what I thought. could you please have a look and comment on that – sariii Jun 10 '20 at 16:28
  • 1
    The value returned by `eval` is a NumPy array, see [Dump a NumPy array into a csv file](https://stackoverflow.com/q/6081008/1782792). – jdehesa Jun 10 '20 at 16:37
  • Followed what you said, still it does not recognize 'topk_mat`. I have updated my question with entire update. – sariii Jun 10 '20 at 16:55
  • Its like the way I am accessing that variable is not correct. – sariii Jun 10 '20 at 17:24
  • @jdehesa I spent the time from yesterday trying different things that may help in removing error, still no success. Could you please have a quick look at my update. I have included how I am doing it. – sariii Jun 11 '20 at 23:19
  • I see two issues, first I think you keep mixing the _class_ and the _instances_. Your class is `my_layer`, although it should probably be called `MyLayer`. You should make an instance of that layer class, not try to access attributes of `my_layer` directly. Then, your error says `type object 'my_layer' has no attribute 'c'`, but I don't see where you are trying to do `my_layer.c`. You should be accessing `my_layer_instance.topk_mat`. Also, in your `np.savetxt(...)`, your parentheses are misplaced (`delimiter` is being passed to `eval`, not `savetxt`). – jdehesa Jun 12 '20 at 08:47
  • @jdehesa, Thanks again. I spent quite some time on it still not able to get it done. I understand your point about making a new instance, though, not sure if it is valid to create an instance of a layer while I want to fit on the data? Is the instance create here would be the same as the one created automatically?(Most probably dumb question). The part that you mentioned in the comment "but I don't see where you are trying to do my_layer.c." is right and typos, as I changed the variable names but forgot to update it here. Thanks for pointing out of misplacing the delimiter :) – sariii Jun 14 '20 at 21:34
  • I tried different things in the last two days though still could not get it to work. By any chance do you know of any sample like this please? – sariii Jun 19 '20 at 04:15
  • 1
    Ah I lost track of this. As I said you need to instance the class, so do something like `my_layer_inst = my_layer(...)`, then add `my_layer_inst` to your model, if it is a sequential model just with `model.add(my_layer_inst)` or if it is a functional model do something like `y = my_layer_inst(y)`. Then in the callback you do `tf.keras.backend.eval(my_layer_inst.topk_mat)` (`delimiter=","` is still misplaced in the posted code...). – jdehesa Jun 19 '20 at 09:12
  • I really followed what you say. please have a look at my updated question. I think what I may have missed here is that I have these things in different `classes` thats why I have included the piece of the code in the three classes here. Could you please have a quick look again? – sariii Jun 19 '20 at 21:42
  • 1
    I'm confused by this. You seem to create an instance of `my_layer` at `encoded = my_layer(self.comp_topk, self.ctype)(encoded)`, but you are not storing the layer object in a variable, only its output. Again, you should have `my_layer_inst = my_layer(self.comp_topk, self.ctype)` and then `encoded = my_layer_inst(encoded)`, and again, you should _not_ do `my_layer.topk_mat`, but instead `my_layer_inst.topk_mat` (and again, `delimiter` parameter is misplaced). I'm not sure if you're understanding what I mean, but I don't think I can explain it clearer. – jdehesa Jun 22 '20 at 10:02
  • @jdehesa Thank you so much for being patient with me. And thanks for giving detailed instruction(after reading this sentence `but you are not storing the layer object in a variable, only its output` I got what you meant previously. sorry i dont know why I wasnt getting it, My Bad). And sorry for mentioning `delimeter several times I had corrected it in my script but forgot to get it updated here. Now it does not complain about the logic. Though error raised complaining the `topk_mat` is uninitialized. As you see I have followed exactly what we did in the previous question. – sariii Jun 23 '20 at 05:35
  • Do u have any idea what could be the reason? It logically should not complain as this part is exactly what we have done before. – sariii Jun 23 '20 at 05:35
  • 1
    Glad you're making progress. I don't know about your current error, though, I don't really use Keras myself so I'm not so familiarized with its caveats and such. – jdehesa Jun 23 '20 at 09:24
  • 1
    Top K of what? Can't you write a custom callback? – BlackBear Jun 25 '20 at 12:04
  • 1
    You can use train_on_batch. It gives you more freedom to do whatever you want in between epochs and hence is a likely option. – Arka Mitra Jun 25 '20 at 18:52
  • @BlackBear In the above example I need to save result of `c` in each epoch. All the available example are covering some predefined metrics or weight like `loss, accuracy, weight of the model....`I have not seen any example trying to do this. – sariii Jul 04 '20 at 22:27
  • 1
    Why don't you that callback at the end of an epoch? – BlackBear Jul 06 '20 at 06:39
  • @BlackBear Thanks for following with my question. I have `callback` in the `fit`. There I have used `lambdacallback` and `on_epoch_begin`, though changing it to `on_epoch_end` does not change anything just I received the error after the first epoch. Do you have any idea what could be the reason and why I get `attempting to use uninitialized value for topk_mat`? It is obvious that it does not like it I have initialized `topk_mat`=None in the layer. but do not know what other way could be the alternative way to do this. – sariii Jul 07 '20 at 03:49
  • Are you sure the method is called and the matrix is created? Put a print after that. Also, usually you create weights in the `build` method, so try to move the call to `add_weight` there – BlackBear Jul 07 '20 at 06:53
  • Yes, the rest of the things and the function of the layer works just fine before adding this callback (to save the matrix ...). Could you please elaborate more? what do you mean by create weight in build? in the build that I created the architecture? there it can not recognize `add_weight`. Could you refer me to a link @BlackBear please? – sariii Jul 08 '20 at 14:43
  • See [here](https://keras.io/guides/making_new_layers_and_models_via_subclassing/#best-practice-deferring-weight-creation-until-the-shape-of-the-inputs-is-known), quoting: "In the Keras API, we recommend creating layer weights in the build(self, inputs_shape) method of your layer" – BlackBear Jul 08 '20 at 15:28

1 Answers1

1

I think you could save the variable using a list-tracking Checkpoint.

you need to add code in the training so you need to code your training loop and save the variable at the end of each epoch.

def fit_and_save_log(self, train_X, val_X, nb_epoch=50, batch_size=100, contractive=None):
    import tensorflow as tf
    optimizer = Adam(lr=0.0005)

    self.autoencoder.compile(optimizer=optimizer, loss='binary_crossentropy') # kld, binary_crossentropy, mse   
    
    save = tf.train.Checkpoint()
    save.listed = []
    
    # Prepare dataset
    X, y = train_X
    train_ds = tf.data.Dataset.from_tensor_slices((x, y))
    train_ds = train_ds.shuffle(10000)
    train_ds = train_ds.batch(batch_size)
    iterator = train_ds.make_initializable_iterator()
    next_batch = iterator.get_next()

    for epoch in range(nb_epoch):
        sess.run(iterator.initializer)           
        
        while True:
            try:
                self.autoencoder.train_on_batch(next_batch[0], next_batch[1])
            except tf.errors.OutOfRangeError:
                break
        
        save.listed.append(self.encoded_instant.topk_mat)

        # you can compute validation results here 

    save_path = save.save('./topk_mat_log', session=tf.keras.backend.get_session())
    return self

Or you can use the model.fit function if you prefer it. Doing it this way can be easier, as we do not need to care about creating the batches. However, repeatedly calling model.fit may result in memory leak. You can give it a try and check how it behaves. [1]

def fit_and_save_log(self, train_X, val_X, nb_epoch=50, batch_size=100, contractive=None):
    import tensorflow as tf
    optimizer = Adam(lr=0.0005)

    self.autoencoder.compile(optimizer=optimizer, loss='binary_crossentropy') # kld, binary_crossentropy, mse   
    
    save = tf.train.Checkpoint()
    save.listed = []
    
    for epoch in range(nb_epoch):
        self.autoencoder.fit(train_X[0], train_X[1],
                epochs=1,
                batch_size=batch_size,
                shuffle=True,
                validation_data=(val_X[0], val_X[1]))
        
        save.listed.append(self.encoded_instant.topk_mat)

        # you can compute validation results here 

    save_path = save.save('./topk_mat_log', session=tf.keras.backend.get_session())
    return self

Then you can restore the saved variable like this

restore = tf.train.Checkpoint()
restore.restore(save_path)
restore.listed = []
v1 = tf.Variable(0.)
restore.listed.append(v1) # Now v1 corresponds with topk_mat in the first epoch
Pedrolarben
  • 1,053
  • 8
  • 18
  • Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackoverflow.com/rooms/217636/discussion-on-answer-by-pedrolarben-how-to-store-result-of-an-operation-like-to). – Samuel Liew Jul 11 '20 at 08:16
  • @Pedrolarben Any update on this? I am asking cause once I sent you the script you did not reply to any of my emails. I really appreciate it if you could just update me on whether you could solve it. Thanks again. – sariii Jul 20 '20 at 18:10
  • @Pedrolarben I dont know why you did not reply me back right after I sent you the script. I hope everything is going okay with you. Two main functions has been removed on that script and I replace them with predefined True, so it does not function as it should. it was just an example to showcase my problem ;). I hope everything is going okay with you. – sariii Jul 22 '20 at 22:26