$\DeclareMathOperator{\ex}{\mathbb E}\DeclareMathOperator{\Var}{Var}\DeclareMathOperator{\Cov}{Cov}$If there are two giant families, one with all the women in it and one with all the men in it, then men will have more brothers on average. So you cannot say anything in general, even assuming that there are the same number of men and women.
So we will need to make some assumptions. In order to find out what the correct assumptions are, I shall define some notation. For convenience, I shall use probabilistic notation, but we are really just talking about counting.
Let $M$ be the random variable corresponding to the number of men in a family chosen at random from the set of families, and let $W$ be the quantity corresponding to the number of women in a randomly chosen family. Let $\mathcal M$ denote the total number of men, let $\mathcal W$ denote the total number of women and let $\mathcal F$ denote the total number of families.
Important Note: The families themselves should be treated as constants. They are not random samples drawn from some kind of
distribution or anything like that. When I use probabilistic
notation, it is purely for the sake of convenience - I am interpreting the question combinatorially, and it so happens that probabilistic constructs such as sample variance do a good job of capturing certain combinatorial quantities that are relevant in this question.
If you like, we are working over the discrete measure space $(F,
\mathcal P(F), \mathbb P)$, where $F$ is the set of families and
$\mathbb P(A)=|A|/|F|$ for any $A\subset F$. $M$ is then the random
variable defined by $M(f)=\textrm{number of men in $f$}$, while
$W(f)=\textrm{number of women in $f$}$. $\ex,\Var,\Cov$ will all take
their usual meanings as population mean, population variance and population
covariance.
We want to compute the average number of brothers that each man has. To do this, we shall double-count the set $A$ of pairs $(m_1,m_2)$ such that $m_1$ and $m_2$ are brothers.
The first way we count this set will be by family. For a family $f$, let $m_f$ denote the number of men in family $f$. Then we have
\begin{align}
|A|&=\sum_f m_f(m_f-1)\\
&=\mathcal F\ex(M(M-1))
\end{align}
since in each family $f$, we have $m_f$ choices for the first brother and $m_f-1$ choices for the second brother.
The second way to count this set will be by man. For a man $m$, denote by $b_m$ the number of brothers that $m$ has. Then we have
$$
|A|=\sum_m b_m
$$
Therefore:
$$
\sum_m b_m=\mathcal F\ex(M(M-1))
$$
Then:
\begin{align}
\textrm{Average number of brothers a man has}&=\sum_m b_m/\mathcal M\\
&=\frac{\mathcal F}{\mathcal M}\ex(M(M-1))\\
&=\frac{\ex(M(M-1))}{\ex M}
\end{align}
A similar argument gives us
$$
\textrm{Average number of brothers a woman has}=\frac{\ex(MW)}{\ex W}
$$
In this second case, if we write $w_f$ for the number of women in family $f$, then the number of pairs $(w,m)$ such that $w$ and $m$ are sister and brother is $m_fw_f$.
We would like to show that this second quantity is bigger than the first. Let's try and rewrite each quantity first.
\begin{align}
\frac{\ex(M(M-1))}{\ex M}&=\frac1{\ex M}\left(\ex M^2-\ex M\right)\\
&=\frac1{\ex M}\left(\Var M+(\ex M)^2-\ex M\right)\\
&=\frac{\Var M}{\ex M} + \ex M - 1
\end{align}
while
\begin{align}
\frac{\ex(MW)}{\ex W}&=\frac{1}{\ex W}\left(\Cov(M,W)+\ex M\ex W\right)\\
&=\frac{\Cov(M,W)}{\ex W} + \ex M
\end{align}
Therefore, in order to ensure that the average number of brothers a woman has is greater than the average number of brothers a man has, we need to assume that:
$$
\frac{\Cov(M,W)}{\ex W}>\frac{\Var M}{\ex M}-1
$$
We can turn this into a condition saying that the number of men per family has to have small variance
$$
\Var M<\ex M+\frac{\mathcal M}{\mathcal W}\Cov(M, W)
$$
Is this a reasonable assumption to make? Assuming that the number of men in a family is independent of the number of women in that family, we should expect $\Cov(M, W)$ to be small. And if we assume that the total number of men is roughly equal to the to the number of women, then $\mathcal M/\mathcal W$ will be roughly equal to $1$. So women have more brothers if and only if the variance in the number of men per family is less than $\ex M$.
This is a surprising result, since there's no reason to suppose that the variance in the number of men per family should be less than $\ex M$. In fact, numerical experiments indicate that $\Var M$ is quite often larger than $\ex M$, which means that in fact it will be men who have more brothers than women.
So the answer is that it depends on how large the families are, but your intuition that women will have more brothers is not true in general, even under fairly strong assumptions. If the number of men per family varies greatly from family to family, it is in fact men who have more brothers on average.
This surprising result can easily be confirmed with numerical experiment.
What's the explanation? Well, let's consider a situation in which the number of men per family varies greatly from family to family. Suppose we assume also that the number of men per family is uncorrelated with the number of women per family.
What this means is that there are going to be a significant number of families with lots of men and very few women, and a significant number of families with lots of women and very few men.
Now, within any given family, we know that the women will have more brothers than the men. But look at the overall contribution to the average:
- The first type of family gives us lots of men with lots of brothers, and a small number of women with lots of brothers.
- The second type of family gives us lots of women with very few brothers, and a small number of men with very few brothers.
So on average, the men will tend to have lots of brothers, and the women will tend to have very few brothers, as long as the variance in the number of men is large enough to counteract the extra brother that each woman has. With large enough families, that one extra brother counts for less and less, and the variance effect takes over, giving you precisely the opposite effect from what you expected.
Let's take one last look at the result. We found that the important factor was that the variance in the number of men per family should not be too large. What if this value were equal to zero? That would mean that every family had the same number $a$ of men, so then it would be true that women have more brothers, since every man would have $a-1$ brothers and every woman would have $a$ brothers.
The dependence on the covariance is interesting, too. By the Cauchy-Schwarz inequality, we have $\Cov(M,W)\le\sqrt{\Var M\Var W}$, and if we assume that $\Var M$ and $\Var W$ are roughly the same, then we have $\Cov(M,W)\le\Var M$. This extreme value occurs if $W=\lambda M$ for some positive constant $\lambda$. In that case, a simple counting argument shows us that women will have an average of one more brother than men.