136

Our development team has been using the GitFlow branching strategy and it has been great !

Recently we recruited a couple testers to improve our software quality. The idea is that every feature should be tested/QA by a tester.

In the past, developers work on features on separate feature branches and merge them back to the develop branch when done. The developer will test his work himself on that feature branch. Now with testers, we start asking this Question

On which branch should the tester test new features ?

Obviously, there are two options:

  • on the individual feature branch
  • on the develop branch

Testing On Develop Branch

Initially, we believed this is the sure way to go because:

  • The feature is tested with all other features merged to the develop branch since it's development started.
  • Any conflicts can be detected earlier than later
  • It makes the tester's job easy, he is only dealing with one branch (develop) at all time. He doesn't need to ask the developer about which branch is for which feature ( feature branches are personal branches managed exclusively and freely by relevant developers )

The biggest problems with this is:

  • The develop branch is polluted with bugs.

    When the tester finds bugs or conflicts, he reports them back to the developer, who fixes the issue on the develop branch (the feature branch were abandoned once merged ), and there could be more fixes required afterward. Multiple subsequence commits or merges (if a branch is recreated off develop branch again for fixing the bugs) makes rolling back the feature from the develop branch very difficult if possible. There are multiple features merging to and being fixed on the develop branch at different times. This creates a big issue when we want to create a release with just some of the features in the develop branch

Testing On Feature Branch

So we thought again and decided we should test features on the feature branches. Before we test, we merge the changes from the develop branch to the feature branch ( catch up with the develop branch ). This is good:

  • You still test the feature with other features in the mainstream
  • Further development ( e.g. bug fix, resolving conflict ) will not pollute the develop branch;
  • You can easily decide not to release the feature until it is fully tested and approved;

However, there are some drawbacks

  • The tester has to do the merging of the code, and if there's any conflict (very likely), he has to ask the developer for help. Our testers specialize in test and are not capable of coding.
  • a feature could be tested without the existence of another new feature. e.g. Feature A and B are both under test at the same time, the two features are unaware of each other because neither of them has been merged to the develop branch. These means you will have to test against the develop branch again when both of the features are merged to the develop branch anyway. And you have to remember to test this in the future.
  • If Feature A and B are both tested and approved, but when merged a conflict is identified, both of the developers for both features believe it is not his own fault/job because his feature branch past the test. There is an extra overhead in communication, and sometimes whoever resolving the conflict is frustrated.

Above is our story. With limited resource, I would like to avoid testing everything everywhere. We are still looking for a better way to cope with this. I would love to hear how other teams handle this kind of situations.

nycynik
  • 6,712
  • 5
  • 54
  • 80
David Lin
  • 12,681
  • 5
  • 43
  • 41
  • 6
    This question seems like it's a better fit for [Programmers](http://programmers.stackexchange.com/), since it doesn't deal with a programming problem, but rather a development process. Can someone migrate it? –  Aug 22 '13 at 05:05
  • 4
    elated question: [How should Git Flow work with QA testing both a release and a new feature?](https://stackoverflow.com/questions/25238846/how-should-git-flow-work-with-qa-testing-both-a-release-and-a-new-feature) – Joshua Goldberg Apr 10 '15 at 16:57
  • 2
    Our model is exactly the same. I'm interested in hearing about how your QA team reports issues on feature branches differently from issues in the field or issues during UAT process (if you have one). We use Atlassian JIRA and we have a different workflow for the two. – void.pointer Sep 18 '15 at 18:33
  • 2
    Deciding the same thing right now. Plus, as our environment is a java spring application, it takes around 20 minutes to build and deploy to test environment. Happy someone asked the same doubts I had. – digao_mb Feb 25 '16 at 12:31
  • The first drawback isn't inherent to the process of testing on feature branches. Tools like Github Enterprise and Bitbucket have the ability to require approval for pull requests and the person responsible for QA can approve signaling to the developer that they are free to merge into develop. – Derek Greer Dec 15 '16 at 16:38
  • A couple of things I would add to the drawbacks: 1) There are issues with either requiring testers to deploy features to their local box or using CI to deploy features to a shared QA environment. 2) Features are often progressive (e.g. story 1 adds feature X with supporting infrastructure, story 2 builds on the infrastructure of story 1 to add feature Y). Only allowing completed features to be merged to develop means you either can't squash feature commits (more junk commits than feature bug fixes causes) or you can't start story 2 until story 1 is complete. – Derek Greer Dec 15 '16 at 17:00
  • It's funny how some questions get closed for seeming broad even though they are about programming and programming languages, and yet something like this is acceptable on here. – Ash May 28 '17 at 11:54

6 Answers6

106

The way we do it is the following:

We test on the feature branches after we merge the latest develop branch code on them. The main reason is that we do not want to "pollute" the develop branch code before a feature is accepted. In case a feature would not be accepted after testing but we would like to release other features already merged on develop that would be hell. Develop is a branch from which a release is made and thus should better be in a releasable state. The long version is that we test in many phases. More analytically:

  1. Developer creates a feature branch for every new feature.
  2. The feature branch is (automatically) deployed on our TEST environment with every commit for the developer to test.
  3. When the developer is done with deployment and the feature is ready to be tested he merges the develop branch on the feature branch and deploys the feature branch that contains all the latest develop changes on TEST.
  4. The tester tests on TEST. When he is done he "accepts" the story and merges the feature branch on develop. Since the developer had previously merged the develop branch on feature we normally don't expect too many conflicts. However, if that's the case the developer can help. This is a tricky step, I think the best way to avoid it is to keep features as small/specific as possible. Different features have to be eventually merged, one way or another. Of course the size of the team plays a role on this step's complexity.
  5. The develop branch is also (automatically) deployed on TEST. We have a policy that even though the features branch builds can fail the develop branch should never fail.
  6. Once we have reached a feature freeze we create a release from develop. This is automatically deployed on STAGING. Extensive end to end tests take place on there before the production deployment. (ok maybe I exaggerate a bit they are not very extensive but I think they should be). Ideally beta testers/colleagues i.e. real users should test there.

What do you think of this approach?

PHLAK
  • 20,477
  • 18
  • 47
  • 52
Aspasia
  • 1,411
  • 1
  • 12
  • 20
  • 2
    How do we make sure that feature1 and feature2 which were tested independently are also good to go together (as mentioned in the question)? – Kumar Deepak Jul 15 '14 at 13:22
  • 2
    we do indirectly, by merging one and then the other one to develop. It is step 4 of the process above and it has to do with chronological order. So if feature 2 is ready to be merged but feature 1 was already merged, the feature 2 developer and tester have to make sure that their merge will work. – Aspasia Jul 15 '14 at 13:31
  • 1
    I think anyway according to [this](http://nvie.com/posts/a-successful-git-branching-model/) git branching model you are not supposed to merge two feature branches with each other. – Aspasia Jul 15 '14 at 14:25
  • 1
    We've run into problems in step 6, especially in crunch times with multiple features being moved to develop, because of non-trivial merges that happen after QA has signed off on the feature branch, despite merging devlop to feature as late as possible. I commented in a little more detail here: http://stackoverflow.com/a/25247382/411282 – Joshua Goldberg Apr 10 '15 at 17:02
  • @JoshuaGoldberg I know what you mean and agree that is the trickiest part. One way I can think of reducing the complexity is create regular, small releases, with very few/even just one features. Or features that don't touch same parts of the code. And anyway the release should be thoroughly tested as a whole before it hits production imho. – Aspasia Jun 01 '15 at 14:24
  • 9
    Do you have a complete TEST Environment (DB, Server, Client, etc) for each feature branch? Or do they share the Environment and just have different names (e.g. app-name_feature1- app-name_feature2, etc.) – hinneLinks Aug 10 '15 at 07:46
  • @hinneLinks one environment should be enough and test one feature branch at the time. – Aspasia Oct 13 '15 at 08:58
  • 1
    So this model still doesn't allow for concurrent testing of features to occur without having to re-test after every merge to develop? – Oletha Sep 09 '16 at 15:18
  • Since we promote builds (we use the same build on each env to test before production) this method does not work well for us. – nycynik Dec 13 '16 at 19:32
  • How do you keep E2E tests updated? Do you just create them when the feature lands on the TEST environment? Or does the developer merge E2E tests? This is assuming that automated unit tests are already defined and merged. Are automated E2E tests required for your sign off? – aug Jun 19 '17 at 21:38
  • @Oletha not sure how you would know that everytihning works before rebasing on develop. But if you know how I am curious! – Aspasia Jul 21 '17 at 09:48
  • @aug I would treat them as all other tests but that depends on your particular setting: who writes those tests? Are they on the same repo? Are they part of the continuous integration? – Aspasia Jul 21 '17 at 09:52
  • @Aspasia if you merge/rebase develop into your feature branch, it will be in the same state as develop will be in after that feature branch is merged into it. :) of course this is not ideal since develop could change before you are ready to merge... – Oletha Jul 21 '17 at 11:09
  • @Oletha that's what I suggested above, I prefer rebase. But yeah if develop changes in the mean time it can be a problem. For a small team it works – Aspasia Jul 21 '17 at 11:40
  • @Aspasia yeah the annoying this is the tests are in a different repo and we have someone who has a role for writing tests. Naturally we have to have good communication with them but sometimes, depending how complex the feature is to test in an automative way, we do manual testing first and then circle back to automate the test cases. I was just curious if you had a different process. – aug Jul 21 '17 at 18:16
  • @GlenThomas this is a pretty old post. The PR/code review would happen in step 4 in order to merge the feature branch to develop. – Aspasia Nov 18 '17 at 16:44
  • @Aspasia so the tester is creating a PR for code they didn't write? – Glen Thomas Nov 20 '17 at 13:11
  • @GlenThomas sorry I meant 3. So the developer rebases develop on the feature branch and then deploys on TEST. Whether the tester will also review the code or not that depends if the testers are also developers or not. In any case when tests and review are done and fixes are made on the future branch, the PR can be merged to develop. – Aspasia Nov 21 '17 at 14:16
  • What happens if there is bug after merging feature branch into develop branches? Should the bug be fixed in a new feature branch or in the develop branch? – thoslin Jan 24 '18 at 05:22
  • @thoslin my opinion is that if branch has already been merged to master it should be treated like a hotfix on master. If not, probably it has to do with one of the futures and should better fixed on there. Please have a look here http://nvie.com/posts/a-successful-git-branching-model/ – Aspasia Jan 24 '18 at 15:47
  • @Aspasia one thing I do not understand, when we are publishing the TEST from feature branches. Wouldn't we need multiple TEST environment, in fact as many as the no of features currently in development? – shbht_twr Aug 03 '18 at 06:25
  • @shbht_twr I think this is very dependent on the team. For example if you have one test environment per team of 5-6 people, it is easy to coordinate and test one feature at the time. Since features are independent once it is done the next is deployed. If the team releases and tests very often then more test environments would be needed or maybe virtual/cloud ad hock environments. – Aspasia Aug 23 '18 at 10:25
  • Hello @Aspasia Thanks for sharing this info. I would like to ask how do you deploy a feature branch for testing? I am using GCP and trying to figure out this thing. Thanks – Bill May 22 '21 at 20:17
42

Before test, we merge the changes from the develop branch to the feature branch

No. Don't, especially if 'we' is the QA tester. Merging would involve resolving potential conflicts, which is best done by developers (they know their code), and not by QA tester (who should proceed to test as quickly as possible).

Make the developer do a rebase of his/her feature branch on top of devel, and push that feature branch (which has been validated by the developer as compiling and working on top of the most recent devel branch state).
That allows for:

Each time the tester detects bug, he/she will report it to the developer and delete the current feature branch.
The developer can:

  • fix the bug
  • rebase on top of a recently fetched develop branch (again, to be sure that his/her code works in integration with other validated features)
  • push the feature branch.

General idea: make sure the merge/integration part is done by the developer, leaving the testing to the QA.

VonC
  • 1,042,979
  • 435
  • 3,649
  • 4,283
  • Are you saying "don't use merge, use rebase instead"? If so, I'm confused, given the Git FAQ on the difference between the two: https://git.wiki.kernel.org/index.php/GitFaq#What_is_the_difference_between_a_merge_and_a_rebase.3F – Vicki Laidler Sep 21 '13 at 03:54
  • 1
    @VickiLaidler yes, if the feature branch is rejected by QA, the developer has to rebase, not merge (http://stackoverflow.com/a/804178/6309) – VonC Sep 21 '13 at 08:06
  • @VonC: Wow, I'm very surprised nobody else is grokking this. What you recommend is *clearly* the right way to do this. I'd even go so far as to argue that you don't *need* to rebase every time, as long as the feature branch was started at a stable point, and isn't too far behind. But if we are going to insist that we sync the feature branch with develop, then **absolutely rebase, don't merge**. Junio Hamano's comment on this article explains this quite well: http://kentnguyen.com/development/visualized-git-practices-for-team/#comment-423841808 – Dan Moulding Mar 13 '15 at 17:01
  • @DanMoulding I agree. I am used with that kind of workflow (rebase locally, *then* push), as it is enforced by most other modern tool: I am using RTC right now (Rational Ream Concert: https://jazz.net/products/rational-team-concert/) which has the same workflow. I described the Git one in http://stackoverflow.com/a/457988/6309 and http://stackoverflow.com/a/804178/6309 a few years ago. – VonC Mar 13 '15 at 17:54
  • 1
    @VonC I completely agree but there are some issues: 1) Deleting the branch impacts other tooling, like Stash Pull Requests (deleting branch closes the PR). Prefer force pushing. 2) If it's a big feature branch where during its lifetime two people collaborated, merges would have been preferred over rebasing. Rebasing it at the end creates conflict nightmare as the merge commits will be removed, and if code depended on those merge changes, it is non trivial to fix – void.pointer Sep 18 '15 at 18:31
  • @void.pointer I agree this workflow doesn't fit all situation. And I mentioned force push before (http://stackoverflow.com/a/26753583/6309). – VonC Sep 18 '15 at 21:07
  • 1
    Looking back at my answer I would also do a rebase and not a merge for cleaner history. – Aspasia Oct 13 '15 at 08:59
  • "No. Don't, especially if 'we' is the QA tester." No reason provided, therefore this review is not useful – Steve Harrison May 02 '17 at 19:01
  • @SteveHarrison Right: I have added the reason (which is repeated in the last sentence of this answer) – VonC May 02 '17 at 19:05
  • "General idea: make sure the merge/integration part is done by the developer, leaving the testing to the QA." *is not a reason* for "Before test, we(QA Tester) merge the changes from the develop branch to the feature branch" – Steve Harrison May 02 '17 at 19:48
  • Rebasing should be only used on private developer histories (i.e. histories that haven't been shared with other developers yet or that were worked on by other developers). This may not always be sensible for all environments. – bompf May 31 '17 at 15:14
  • @VonC For your first point the feature branch is already rebased on top of develop, no merge conflicts are expected. Who does all this is dependent on the team setup of course. Our QAs were developers as well. – Aspasia Jul 21 '17 at 09:56
  • @Aspasia If the feature branch is already rebased on top of develop, then you don't merge develop to feature. – VonC Jul 21 '17 at 12:21
  • @VonC sorry I meant it the other way round obviously I think you know what I mean, its the developers responsibility to rebase develop on top of the feature branch before creating the PR – Aspasia Jul 24 '17 at 09:12
  • @Aspasia Strange: rebasing develop would change its history, which is not practical... – VonC Jul 24 '17 at 09:15
  • @VonC oh the confusion. How I normally work is: rebase the feature branch like: git rebase -i develop, decide which commits-messages I keep and then create a pull request. If that goes well we merge the pr. – Aspasia Jul 25 '17 at 20:26
  • @Aspasia OK, this time, I get it ;) a `git rebase -i develop` is *not* "rebasing `develop`", it is "rebasing (the current branch) *on top of* `develop`". If that current branch is `feature`, that makes sense! – VonC Jul 25 '17 at 21:05
  • @VonC I know, that's what happens when you type fast between other things. So to reply to your "If the feature branch is already rebased on top of develop, then you don't merge develop to feature.": no but you do merge the pr. – Aspasia Jul 26 '17 at 10:43
  • @Aspasia is the PR done between feature and develop? As in: 'I will apply that PR from feature to develop"? – VonC Jul 26 '17 at 10:54
  • @VonC yes between feature and develop. What is called "merge pull request" in github. Why are you asking this question? – Aspasia Jul 26 '17 at 11:40
  • @Aspasia just to make sure what I need to amend in my answer: it is still a merge from feature to master, but not a fast-forward one as I describe: rather, a pull request to be accepted/merged. – VonC Jul 26 '17 at 11:50
  • @VonC yes what I am talking about is merging a PR. However, if you have rebased before merging the PR does exactly the same in the background, a fast forward like described here https://help.github.com/articles/about-pull-request-merges/#rebase-and-merge-your-pull-request-commits. Personally I prefer PRs for several reasons. One of them is they can be nicely connected to github issues and automatically close them on merge. Other reasons are the descriptions can be useful for the reviewer to understand what has been done and what needs to be checked and they are very easy to revert if needed. – Aspasia Jul 27 '17 at 09:40
  • 1
    @Aspasia Good points. I have included pull-requests in the answer for more visibility. – VonC Jul 27 '17 at 11:12
  • @VonC " Each time the tester detects bug, he/she will report it to the developer and delete the current feature branch. " What is the point of deleting the current feature branch? – Wenneguen Feb 05 '18 at 17:58
  • @Wenneguen Just making sure the branch is not merged by mistake. (I suppose that it is what I was going for 5 years ago) – VonC Feb 05 '18 at 18:05
13

The best approach is continuous integration, where the general idea is to merge the feature branches into the developer branch as frequently as possible. This reduces on the overhead of merging pains.

Rely on automated tests as much as possible, and have builds automatically kick off with unit tests by Jenkins. Have the developers do all the work with merging their changes into the main branch and provide unit tests for all their code.

The testers/QA can take participate in code reviews, check off on unit tests and write automated integration tests to be added to the regression suite as features are completed.

For more info check out this link.

Johnny Z
  • 11,190
  • 4
  • 24
  • 35
9

We use what we call "gold", "silver", and "bronze". This could be called prod, staging, and qa.

I've come to call this the melting pot model. It works well for us because we have a huge need for QA in the business side of things since requirements can be hard to understand vs the technicals.

When a bug or feature is ready for testing it goes into "bronze". This triggers a jenkins build that pushes the code to a pre-built environment. Our testers (not super techies by the way) just hit a link and don't care about the source control. This build also runs tests etc. We've gone back and forth on this build actually pushing the code to the testing\qa environment if the tests (unit, integration, selenium ) fail. If you test on a separate system ( we call it lead ) you can prevent the changes from being pushed to your qa environment.

The initial fear was that we'd have lots of conflicts between this features. It does happen were feature X makes it seem like feature Y is breaking, but it is infrequent enough and actually helps. It helps get a wide swath of testing outside what seems is the context of the change. Many times by luck you will find out how your change effects parallel development.

Once a feature passes QA we move it into "silver" or staging. A build is ran and tests are run again. Weekly we push these changes to our "gold" or prod tree and then deploy them to our production system.

Developers start their changes from the gold tree. Technically you could start from the staging since those will go up soon.

Emergency fixes are plopped directly into the gold tree. If a change is simple and hard to QA it can go directly into silver which will find its way to the testing tree.

After our release we push the changes in gold(prod) to bronze(testing) just to keep everything in sync.

You may want to rebase before pushing into your staging folder. We have found that purging the testing tree from time to time keeps it clean. There are times when features get abandoned in the testing tree especially if a developer leaves.

For large multi-developer features we create a separate shared repo, but merge it into the testing tree the same when we are all ready. Things do to tend bounce from QA so it is important to keep your changesets isolated so you can add on and then merge/squash into your staging tree.

"Baking" is also a nice side effect. If you have some fundamental change you want to let sit for a while there is a nice place for it.

Also keep in mind we don't maintain past releases. The current version is always the only version. Even so you could probably have a master baking tree where your testers or community can bang on see how various contributors stuff interact.

Eric Twilegar
  • 480
  • 4
  • 6
3

In our company we cant use agile development and need approval for every change by business, this causes a lot of issues.

Our approach for working with GIT is this;

We have implemented "Git Flow" in our company. We using JIRA and only approved JIRA Tickets should be go to production. For Test approval we exted it with a created a seperate Test-Branch.

Steps for processing a JIRA Tickets are:

  1. Create a new Branch from Develop-Branch
  2. Do the code Changes on the Feature-Branch
  3. Pull from Feature the Changes to the Test/QA Branch
  4. After business approval we pull the change from feature branch into develop
  5. The develop goes frequently in a release and then finally master branch

Splitting each request in an own feature ensures, only approved changes went to production.

The complete process looks like this: enter image description here

Christian Müller
  • 712
  • 5
  • 14
  • So this process only allows for a single feature to be tested at once? That seems like a huge bottleneck for a big dev team. – Kane Oct 21 '20 at 12:09
  • You can test multiple features in a testsystem merging them, both in the "test" branch. And if you have multiple test systems, you can create multiple "test-branches" – Christian Müller Oct 22 '20 at 07:58
  • That's the issue ... having multiple "test systems". Nobody ever talks about that bottleneck. Also, testing multiple features in a single test branch is risky ... as you don't know what impact feature A will have on feature B. – Kane Oct 23 '20 at 14:20
  • You are correct, deciding to have multiple test branches is risky and should considered carefully. The use is, if you have long term changes to be tested seperatly to short term bug fixes. Updating the Test systems frequently from Develop is important to do, after a release, or you may get in trouble later, when the larger features go live (like a new year release or something). But for long term changes, since they have usually larger impacts, to make an regression test also on the Release branch before go live. – Christian Müller Oct 30 '20 at 15:22
  • 1
    @ChristianMüller your approach is very interesting. I was just curious on knowing how do you handle database scripts considering the differences between feature branch and develop branch? – Felicity Feb 16 '21 at 15:48
  • @Felicity: That can be a little bit problematic, depending what you are using. I have two approaches for MS SQL Server. (1) With Database First Approach on SQL Server you can use DACPAC -> https://docs.microsoft.com/de-de/sql/relational-databases/data-tier-applications/data-tier-applications?view=sql-server-ver15 / (2) Alternate with code first you can use the migrations concept: https://docs.microsoft.com/de-de/ef/ef6/modeling/code-first/migrations/ – Christian Müller Feb 18 '21 at 09:23
1

I would not rely on manual testing alone. I would automate the testing of each feature branch with Jenkins. I setup a VMWare lab to run Jenkins tests on Linux and Windows for all browsers. It's truly an awesome cross browser, cross platform testing solution. I test functional/integration with Selenium Webdriver. My selenium tests run under Rspec. And I wrote them specially to be loaded by jRuby on Windows. I run traditional unit tests under Rspec and Javascript tests under Jasmine. I setup headless testing with Phantom JS.

Natus Drew
  • 1,616
  • 1
  • 19
  • 20