3

I'm running a reinforcement learning program in a gym environment(BipedalWalker-v2) implemented in tensorflow. I've set the random seed of the environment, tensorflow and numpy manually as follows

os.environ['PYTHONHASHSEED']=str(42)
random.seed(42)
np.random.seed(42)
tf.set_random_seed(42)

env = gym.make('BipedalWalker-v2')
env.seed(0)

config = tf.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)
config.gpu_options.allow_growth = True
sess = tf.Session(config=config)
# run the graph with sess

However, I get different results every time I run my program (without changing any code). Why are the results not consistent and what should I do if I want to obtain the same result?

Update:

The only places that I can think of may introduce randomness (other than the neural networks) are

  1. I use tf.truncated_normal to generate random noise epsilon so as to implement noisy layer
  2. I use np.random.uniform to randomly select samples from replay buffer

I also spot that the scores I get are pretty consistent at the first 10 episodes, but then begin to differ. Other things such as losses also show a similar trend but are not the same in numeric.

Update 2

I've also set "PYTHONHASHSEED" and use single-thread CPU as @jaypops96 described, but still cannot reproduce the result. Code has been updated in the above code block

Community
  • 1
  • 1
Maybe
  • 1,667
  • 2
  • 15
  • 32
  • Are you using `env.action_space.sample` or something similar anywhere? That could do it. Also if your agent uses any randomness in its graph (not uncommon, to make the agent more robust), you'll wind up with varying results. – aedificatori Jan 04 '19 at 23:58
  • @aedificatori Thanks for your suggestion, I've double checked my code and updated the question. I begin to conjecture whether it is about the loss of precision or something similar? – Maybe Jan 05 '19 at 01:17
  • While reinforcement learning can be somewhat ridiculous when it comes to navigating its error manifold, loss of precision is likely still too small to be an issue (by perhaps a few orders of magnitude, unless you've got rounding or floating point errors, which you'd probably recognize out-of-hand). I might recommend testing Tensorflow and Numpy individually to make sure your seeds are being used properly. So for Numpy, in a few places in your program, just print a random number. For Tensorflow, same thing, but you'll have to extract it from your graph if you're not using EagerExecution. – aedificatori Jan 05 '19 at 01:32
  • You need to add constant seed to all operations that accept seed as an argument, like `tf.nn.dropout`, `tf.contrib.layers.xavier_initializer`, and so on. It seems `tf.set_random_seed` is not doing its job properly. – Ali Salehi Apr 18 '19 at 00:45
  • 2
    I suggest checking whether your TensorFlow graph contains nondeterministic operations. Notably, `reduce_sum` is one such operation. These operations are nondeterministic because floating-point addition and multiplication are nonassociative (the order in which floating-point numbers are added or multiplied affects the result) and because such operations don't guarantee their inputs are added or multiplied in the same order every time. I'm not aware of a complete list of nondeterministic TensorFlow operations. – Peter O. Sep 25 '19 at 12:35
  • @PeterO, thank you for sharing such valuable information! – Maybe Sep 26 '19 at 00:13
  • [This post](https://stackoverflow.com/questions/51249811/reproducible-results-in-tensorflow-with-tf-set-random-seed/) is related and several answers are worth reading. – GZ0 Sep 26 '19 at 00:44

3 Answers3

5

I suggest checking whether your TensorFlow graph contains nondeterministic operations. For example, reduce_sum before TensorFlow 1.2 was one such operation. These operations are nondeterministic because floating-point addition and multiplication are nonassociative (the order in which floating-point numbers are added or multiplied affects the result) and because such operations don't guarantee their inputs are added or multiplied in the same order every time. See also this question.

EDIT (Sep. 20, 2020): The GitHub repository framework-determinism has more information about sources of nondeterminism in machine learning frameworks, particularly TensorFlow.

Peter O.
  • 28,965
  • 14
  • 72
  • 87
2

It seems that tensorflow neural networks introduce randomness during training that isn't controlled by a numpy random seed. The randomness appears to possibly come from python hash operations and parallelized operations executing in non-controlled ordering, at the very least.

I had success getting 100% reproducibility using a keras-tensorflow NN, by following the setup steps in this response: How to get reproducible results in keras

specifically, I used the formulation proposed by @Poete Maudit in that link.

They key was to set random seed values UP FRONT, for numpy, python, and tensorflow, then also to make tensorflow run on single-thread CPU in a specially-configured session.

Here's the code i used, updated very slightly from the link i posted. 

print('Running in 1-thread CPU mode for fully reproducible results training a CNN and generating numpy randomness.  This mode may be slow...')
# Seed value
# Apparently you may use different seed values at each stage
seed_value= 1

# 1. Set `PYTHONHASHSEED` environment variable at a fixed value
import os
os.environ['PYTHONHASHSEED']=str(seed_value)
seed_value += 1

# 2. Set `python` built-in pseudo-random generator at a fixed value
import random
random.seed(seed_value)
seed_value += 1

# 3. Set `numpy` pseudo-random generator at a fixed value
import numpy as np
np.random.seed(seed_value)
seed_value += 1

# 4. Set `tensorflow` pseudo-random generator at a fixed value
import tensorflow as tf
tf.set_random_seed(seed_value)

# 5. Configure a new global `tensorflow` session
session_conf = tf.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)
sess = tf.Session(graph=tf.get_default_graph(), config=session_conf)
tf.keras.backend.set_session(sess)

#rest of code...
jaypops96
  • 131
  • 5
  • Thanks for your answer :-). I've done exact setup as you described, except that I do not use keras. However, I still cannot reproduce the same result. – Maybe Jan 08 '19 at 22:29
  • Well it may be a little different since you're using just TF and not Keras, but a few notes: (1) try setting the random seeds before doing ANYTHING else in the code, and make sure you set them in the exact order I specified. @Poete Maudit emphasized it's important to do them first and in that exact order. (2) it looks like your code has no analogue to my line "tf.keras.backend.set_session(sess)" - you configure the session but never set the tensorflow backend to use that session. try to find something comparable to my line in the tensorflow documentation? – jaypops96 Jan 09 '19 at 21:59
  • Thanks. I've reordered my code to be consistent with yours, but still spot different results. By the way, I write my code in tensorflow so I use the session to run the tensorflow graph directly instead of using it as a backup – Maybe Jan 09 '19 at 23:01
  • Hmm then I may be out of ideas. You're running the code I posted as the very first thing in your script? like, before any other imports or anything? Other folks posting have emphasized that these seeds need to be set before ANYTHING else happens in your code. maybe should even happen right when you start the interpreter session if you're running in the interpreter...? Just a final thought. – jaypops96 Jan 10 '19 at 00:10
0

Maybe you can try to set the number of parallelism threads to 1. I have the same problem: the loss became different to the seventh decimal place start from the second episode. It fixed when I set 

tf.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)
AnsFourtyTwo
  • 2,150
  • 2
  • 9
  • 27
Monica
  • 1