Select dictionary elements in a Pandas series

Question

I have sample data which looks like the following (these are two separate rows delimited by tabs):

Details
[{'name': 'Irrelevant_Data',
  'parentName': 'Irrelevant_Scrape',
  'parentId': '2662610',
  'id': '2684157'},
 {'name': 'Irrelevant_Data',
  'parentName': 'Irrelevant_Scrape',
  'parentId': '068111',
  'id': '291005'}]
[{'name': 'Desired_Data',
  'parentName': 'Relevant_Scrape',
  'parentId': '6123777',
  'id': '31568812'},
 {'name': 'Desired_Data2',
  'parentName': 'Relevant_Scrape',
  'parentId': '6123777',
  'id': '2892718'},
 {'name': 'Irrelevant',
  'parentName': 'Irrelevant_Scrape',
  'parentId': '068111',
  'id': '8001822'}]

It's stored in a Pandas DataFrame series in one column (let's call the column "Details"). I want to select only those "name" elements whose "parentName" in the same row = "Relevant_Scrape."

I'm familiar with the different data structures in Python and also am somewhat familiar with Pandas, but the combination of the two is throwing me off. When I try to loop through the series, my data is transformed into a string, making extraction much harder.

import pandas as pd
from pandas import DataFrame, read_csv

df = pd.read_csv('dataset.csv', sep = '\t')
for row in df['Details']:
    if "Relevant_Scrape" in "parentname":
        print("name")

Thank you in advance.

Edit 2: expanded sample

queryName   date    summary tagging Details
query1  3/31/2016   negative    ['Dummy - Dummy']   [{'name': 'Irrelevant_Data', 'parentName': 'Irrelevant_Scrape', 'parentId': '2517840', 'id': '2565351'}]
query2  3/26/2016   positive    ['Dummy', 'Dummy', 'Dummy'] [{'name': 'Irrelevant_Data', 'parentName': 'Irrelevant_Scrape', 'parentId': '2662610', 'id': '2684157'}, {'name': 'Irrelevant_Data', 'parentName': 'Irrelevant_Scrape', 'parentId': '2517840', 'id': '2565351'}]
query3  3/26/2016   neutral ['Dummy']   [{'name': 'Irrelevant_Data', 'parentName': 'Irrelevant_Scrape', 'parentId': '2662610', 'id': '2684157'}, {'name': 'Irrelevant_Data', 'parentName': 'Irrelevant_Scrape', 'parentId': '2517840', 'id': '2565351'}]
query4  3/19/2016   positive    ['Dummy', 'Dummy']  [{'name': 'Relevant_Data', 'parentName': 'Relevant_Scrape', 'parentId': '2892458', 'id': '2892601'}, {'name': 'Relevant_Data', 'parentName': 'Relevant_Scrape', 'parentId': '2892458', 'id': '2892718'}, {'name': 'Irrelevant_Data', 'parentName': 'Irrelevant_Scrape', 'parentId': '2517840', 'id': '2565351'}]

`When I try to loop through the series...` What loop? I don't see any loop... — Mad Physicist, Jul 05 '16 at 14:24
Can you show us what the actual CSV looks like? The two rows you copied are lists and dicts, not what you're reading into Pandas using `read_csv` — ƘɌỈSƬƠƑ, Jul 05 '16 at 14:32
I've used regex to do this kind of extraction before. added it in just in case someone has a suggestion for how to utilize regex for string data extraction. — dataelephant, Jul 05 '16 at 14:38
You can use `ast` module to convert a string back to a list/dict, see: [this answer](http://stackoverflow.com/a/1894296/5717589). Side note: `pandas.DataFrame` is not designed to store lists, so if you are using this `df`extensively, you may consider changing the way you store your data. — ptrj, Jul 05 '16 at 23:23
Any follow up on this? I have a very similar problem, where I need a derived Series consisting only of values from a specific key in the original Series. I can't iterate, since the Series is large (~ 10 million rows). — darthbhyrava, Feb 20 '20 at 10:04
[This thread](https://stackoverflow.com/q/34157811/7109869) may be of help. — Gonçalo Peres 龚燿禄, May 29 '21 at 00:58

Select dictionary elements in a Pandas series

0 Answers0