remove a single quote in a string with pySpark using regex

Question

I need to remove a single quote in a string. The column name is Keywords. I have an array hidden in a string. So I need to use Regex within Spark Dataframe to remove a single quote from the beginning of the string and at the end. The string looks like this:

Keywords=
'
  [
      "shade perennials"," shade loving perennials"," perennial plants"," perennials"," perennial flowers"," perennial plants for shade"," full shade perennials"
  ]
'

I have tried the following:

remove_single_quote = udf(lambda x: x.replace(u"'",""))
cleaned_df = spark_df.withColumn('Keywords', remove_single_quote('Keywords'))

But the single quote is still there, I have also tried (u"\'","")

Ghost · Answer 1 · 2019-12-17T10:34:01.257

2

from pyspark.sql.functions import regexp_replace

new_df = data.withColumn('Keywords', regexp_replace('Keywords', "\'", ""))

edited Dec 17 '19 at 10:34

answered Dec 17 '19 at 10:10

Ghost

304
2
8

Strick · Answer 2 · 2019-12-18T12:59:10.113

1

Try regexp_replace

from pyspark.sql.functions import regexp_replace,col
    cleaned_df = spark_df.withColumn('Keywords', regexp_replace('Keywords',"\'",""))

OR

from pyspark.sql import functions as f
    cleaned_df = spark_df.withColumn('Keywords', f.regexp_replace('Keywords',"\'",""))

I have not tested It but should work

import ast

    cleaned_df = spark_df.withColumn('Keywords',ast.literal_eval('Keywords'))

Please refer

edited Dec 18 '19 at 12:59

answered Dec 16 '19 at 14:39

Strick

1,010
4
12

1

I ran both methods, but the single quote is still appearing in my rows :/ – Sonya Dec 16 '19 at 15:15
As an alternative you can convert this string into list and then access the first 0th element – Strick Dec 16 '19 at 15:49
1

Infact you dont even need to convert into list you can directly access 0th element from string – Strick Dec 16 '19 at 15:51
Updated answer please chk – Strick Dec 16 '19 at 16:04

remove a single quote in a string with pySpark using regex

2 Answers2