1

I am having trouble understanding how the following ewm() function is working from trial and reading the docs, can anyone help explain?

I am trying to use the following line to exponentially weight correlation within each rolling window.

df['col'].ewm(alpha=0.02, min_periods=10).corr(df['col2'])

My question is: Does this exponentially weight within windows of 10 rows on a rolling basis? if not, how can this be done?

np2020
  • 58
  • 6

1 Answers1

1

min_periods only make sure you have at least 10 data points (rows) before making the calculation. It wouldn't change the outputs where the min_periods is met. Here's an example:

s = pd.Series(np.random.rand(20))

# with min_periods
s1 = s.ewm(alpha=0.5, min_periods=5).corr(s)

# without min_periods
s2 = s.ewm(alpha=0.5).corr(s)

# compare the results
(s1 == s2)

Output:

0     False
1     False
2     False
3     False
4      True
5      True
6      True
7      True
8      True
9      True
10     True
11     True
12     True
13     True
14     True
15     True
16     True
17     True
18     True
19     True
dtype: bool

As you can see, the first 4 rows are False because s1 also has first items being NaN due to min_periods.

TLDR: No, it only masks the first few items as NaN. It stills computes the weighted on the entire column, doesn't shift anything.

Quang Hoang
  • 117,517
  • 10
  • 34
  • 52
  • Thanks for that explanation, I have updated the question slightly as I'm still unsure how I can use these pandas' functions to exponentially weight within each rolling window. – np2020 Mar 05 '21 at 15:48
  • @np2020 See [this answer](https://stackoverflow.com/questions/58012189/is-there-a-way-to-get-pandas-ewm-to-function-on-fixed-windows/58013016#58013016). It is for `mean()`, but I think you can make it work for `corr`. – Quang Hoang Mar 05 '21 at 15:55