regular expression pyspark dataframe column

Question

My dataframe looks like this.

I have a pyspark dataframe and I want to split column A into A1 and A2 like this using regex but that didn't work.

A                 |   A1           | A2
20-13-2012-monday    20-13-2012     monday
20-14-2012-tues      20-14-2012     tues
20-13-2012-wed       20-13-2012     wed

My code looks like this

import re
from pyspark.sql.functions import regexp_extract   
reg = r'^([\d]+-[\d]+-[\d]+)'
df=df.withColumn("A1",re.match(reg, df.select(['A'])).group())
df.show()

You're mixing `re` from the python library with spark. [`pyspark.sql.functions.split`](http://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.functions.split) can split on regex: `df=df.withColumn("A1",split(col("A"), reg))` — pault, Nov 20 '18 at 06:05
I am getting output like -- [ , monday]. I want just monday. No comma no [] — Emma, Nov 20 '18 at 13:09
This is a useful post: [Reference: what does this regex mean?](https://stackoverflow.com/questions/22937618/reference-what-does-this-regex-mean). — pault, Nov 20 '18 at 14:10

score 2 · Accepted Answer · answered Nov 21 '18 at 02:08

You can use the regex as an udf and achieve the required output like this:

>>> import re
>>> from pyspark.sql.types import *
>>> from pyspark.sql.functions import udf

>>> def get_date_day(a):
...   x, y = re.split('^([\d]+-[\d]+-[\d]+)', a)[1:]
...   return [x, y[1:]]

>>> get_date_day('20-13-2012-monday')
['20-13-2012', 'monday']

>>> get_date_day('20-13-2012-monday')
['20-13-2012', '-monday']
>>> get_date_udf = udf(get_date_day, ArrayType(StringType()))


>>> df = sc.parallelize([('20-13-2012-monday',), ('20-14-2012-tues',), ('20-13-2012-wed',)]).toDF(['A'])
>>> df.show()
+-----------------+
|                A|
+-----------------+
|20-13-2012-monday|
|  20-14-2012-tues|
|   20-13-2012-wed|
+-----------------+

>>> df = df.withColumn("A12", get_date_udf('A'))
>>> df.show(truncate=False)
+-----------------+--------------------+
|A                |A12                 |
+-----------------+--------------------+
|20-13-2012-monday|[20-13-2012, monday]|
|20-14-2012-tues  |[20-14-2012, tues]  |
|20-13-2012-wed   |[20-13-2012, wed]   |
+-----------------+--------------------+

>>> df = df.withColumn("A1", udf(lambda x:x[0])('A12')).withColumn("A2", udf(lambda x:x[1])('A12'))
>>> df = df.drop('A12')
>>> df.show(truncate=False)
+-----------------+----------+------+
|A                |A1        |A2    |
+-----------------+----------+------+
|20-13-2012-monday|20-13-2012|monday|
|20-14-2012-tues  |20-14-2012|tues  |
|20-13-2012-wed   |20-13-2012|wed   |
+-----------------+----------+------+

Hope this helps!

regular expression pyspark dataframe column

1 Answers1