How to use "with open as f" for all files within a directory

Question

I should preface this by saying that I am not a pro, so I apologize if I am missing something really obvious, but I want to be able to open a bunch of csvs in a directory to perform the same operation on them. I've been able to do this with other files (like txt files, to parse them, for example), but not with the "with open as f" structure I am using below

I have code where if I specify the file I want to open (like this):

import csv
numlist = []
rangelist = []
designlist = []

with open("full file name", "r+") as f:
 reader = csv.reader(f, delimiter="^")
 for line in f:
  numlist.append(line)

numlist = numlist[1:] # get rid of first line (header)

for item in numlist: 
 if item[0] == 'D':#evaluate if first character in item is "D"
  designlist.append(item)

with open("design" + "test" + ".csv", "w") as f:
 for item in designlist:
  f.writelines(item)#('%s\n' % item)

It works fine, it prints the CSV exactly how I want it to. But when I try to do it for all folders in a file (like this):

import csv
import os
numlist = []
rangelist = []
designlist = []

directory_in_str = 'my directory'
directory = os.fsencode(directory_in_str)

for file in os.listdir(directory):
 filename = os.fsdecode(file)
 full_name = directory_in_str + filename
 with open(full_name, "r+") as f:
  reader = csv.reader(f, delimiter="^")
  for line in f:
   numlist.append(line)

 numlist = numlist[1:] # get rid of first line (header)

 for item in numlist: 
  if item[0] == 'D':#evaluate if first character in item is "D"
   designlist.append(item)

 with open("design" + filename + ".csv", "w") as f:
  for item in designlist:
   f.writelines(item)#('%s\n' % item)

I get the following error:

Traceback (most recent call last):
  File "the python file", line 16, in <module>
    for line in f:
  File "root for where python is on my computer\Python36\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 3223: character maps to <undefined>

I've checked to make sure the files are valid, that the "type" of my variables are strings, but it seems like it isn't accepting the output of my loop.

I've used variants of that same loop to cycle through files in a directory before, and it works fine, but haven't tried it in a "with open as f" statement before. Is there something about that structure that I am missing?

This has nothing to do with using `with` or not. You're getting a `UnicodeDecode` error, which means that one of your files contains characters that can't be decoded using your selected codec. See, e.g., https://stackoverflow.com/questions/14630288/unicodeencodeerror-charmap-codec-cant-encode-character-maps-to-undefined — larsks, Aug 18 '17 at 22:50
@larsks I am sure this is going to sound like a dumb question, but what exactly do I want to be decoding/encoding? I've tried the file name itself, the f object, the line, and the item, and I get errors that I either can't decode what I am trying to decode, or it doesn't solve the actual issue (and I get the same error) — John Doe, Aug 18 '17 at 23:56
If you can update your question to include a complete reproducer (sufficient code *and data* that someone reading it can copy-and-paste things locally and reproduce the problem), we can try addressing your *specific* problem rather than the general question of "what do I do about a UnicodeDecodeError?" (because there are already many answers to that here on SO and elsewhere). — larsks, Aug 19 '17 at 00:49

How to use "with open as f" for all files within a directory

0 Answers0