14

I am trying to create an experiment using Firebase remote config. The criteria is as follows: It should target only new users who have not used the app ( opening the app for the first time)

Now on further research I found that there is a user property as below:

Automatically collected user Property by Firebase

However, this is not available in the experiment window or a similar property that fulfils the above criteria in the Firebase console as seen below:

enter image description here

I can only see the user properties set by my code. One way I can think of is to use one of my custom user property which is not yet set (but set to a value like null), but I don't know how I can do this.

References

https://support.google.com/firebase/answer/6317486?hl=en https://firebase.googleblog.com/2016/10/better-user-targeting-with-firebase.html

sjmach
  • 418
  • 6
  • 13
  • Alternative that works -> Cloud Functions that gets triggered when first open event is fired and then send a Notification to such users. – sjmach Jan 16 '18 at 10:26
  • Yeah we have this problem too, seems that there is no such function out of the box for Remote Configs experiments. For the experiments only for new users we usually setup extra User Property specifically for the targeting of this experiment. Something like 'example_experiment_enrolled' and you setup enrolled/not_enrolled on the client side (and keep it in preferences for example) and create targeting with matching these criteria. – Peregreen Mar 01 '18 at 07:12

2 Answers2

16

I will share our experience with experiments for new users and how the different combinations of targeting work because we worked with it a lot already. You can skip the investigation part if you're not interested and just check the solution that we use now.

Our Investigations Part: Firstly, when we integrated Firebase, we were afraid that Audiences won't work properly for experiments targeting cuz all current users will be treated as new ones after integration, so we checked several approaches and went with an approach of creating specific User Properties that we specified on the client side differently for old/new users. For instance, we created a User Property called adv_experiment_enrolled and specified on client side values 'enrolled'/'not_enrolled', so all new users after installation of this version became 'enrolled' and old ones after updating the version just became 'not_enrolled'. And we just used that User Property as targeting in an experiment. That worked well, but it wasn't a general approach that we could use easily for all experiments and we needed to create User Properties for each new experiment.

So we've tried the Audiences approach after few months of integration that was what @jackes described here with First Open Time user property: https://stackoverflow.com/a/50075684/2723437 And we've got several problems, first of all, seems they had some troubles with populating of that kind of audience and just ~3-5% of new users were getting there. We also created an Audience depending on First Open event itself and used it too, it was populating better and was close to the real number of installs that we had. But we've noticed problems with this approach as well and the biggest one was that an experiment had only 20-30% of users from that Audience. We tested it and noticed by some of our metrics that seems users are not enrolled in this experiment in their first session cuz 1) Firebase takes some time to enroll a user in the Audience and 2) Remote Config has 12h cache by default, so it wasn't really a data for most of the new installations.

A solution that seems to work well for now:
We were surprised with that Firebase does have User Property for First Open Time, but doesn't allow to use this as the targeting for experiments (It would be very helpful to solve this problem tbh), so we just decided to try our good experience with User Property targeting and apply the general approach First Open Time user property, so we've created our own custom_first_open_time especially to target installations after some specific time (we just used current timestamps for platforms in seconds).
Important notes:
- You have to set up user properties before loading a remote config.
- You have to keep this first open time on the client side persistently once you generated it (usually you use NSUserDefaults/SharedPreferences for iOS/Android for that)

Sample of experiment configuration:

enter image description here

Peregreen
  • 1,090
  • 8
  • 14
  • 1
    How quickly does the user properties update at the Firebase end. We need to apply A/B policy at the very launch of out app and will already delay the loading of the app to get the remote config value according to the assigned test group. I'm afraid it won't work, if we first set a user property and immediately fetch the remote config. Will the user property update affect the A/B assignment? – Matti Jokipii Dec 07 '18 at 16:44
  • 3
    @MattiJokipii our tests showed that it will. Seems they either force sync or even send user properties for the remote config fetching, but you have to setup them before calling fetch remote configs. It worked well on our side for both iOS / Android SDKs. Sometimes we even use extra User Properties for targeting instead of audiences for this reason when we want to provide that for 1st session cuz audiences take more time to be assigned. – Peregreen Dec 12 '18 at 00:14
  • 1
    Yes, based on our experience, i can confirm that this is the way to accomplish targeting new users. – Matti Jokipii Mar 04 '19 at 16:32
  • it doesn't work for ios, its just ignores custom property – orium Sep 02 '19 at 07:14
  • @orium what do you exactly mean by `it just ignores custom property`? You have to make sure that you setup the values for the custom properties before actual fetching of the Remote Config. – Peregreen Sep 04 '19 at 05:47
  • @Peregreen I am not sure, at least it doesn't work for test devices, it is hard to test on real ones when test already started – orium Sep 09 '19 at 16:22
  • @orium as far as I remember, test devices always use config that you set up for them, so yes, it's a bit convoluted. I usually tested it through a dev a/b test, just created one in advance with a condition for minVersion of the app higher than your production one, verified the setup, and only after that rolled it out for all users. – Peregreen Oct 16 '19 at 05:23
  • 2
    We faced the same problem with Firebase Audience based on its own `first_time_open` user property not being accurate. We switched to using custom User Property for Remote Config Condition. Worked. – AzaFromKaza Jan 14 '20 at 12:26
  • I tried to use this method, but smth strange happens when I start the experiment. 1) I created a new experiment, set custom_first_open_time > 1581944923450 in the targeting section 2) I used 1 of 10 remote config params for the experiment. Once I started, all users stopped obtaining any remote config param and inside the app was used params set as default in code (inside the app). – alectogeek Feb 17 '20 at 14:03
  • 1
    @alectogeek it seems that you used timestamp in milliseconds, while it should be in seconds if you track it in the similar manner. – Peregreen Feb 19 '20 at 21:34
  • @Peregreen Yeah, it was because of milliseconds and looks like firebase can't compare so large numbers. – alectogeek Feb 20 '20 at 09:07
0

Didn't tried, but it should work. Create an Audience in Firebase Console with following condition: First Open Time is greater than or equal to some date. Where the date should be the next day after you starting an experiment.

jackes
  • 69
  • 3