How to create a new directory in which to save my weather files in python?

Question

I am tasked with having to manually create a new directory (right-click, create new folder) in which to save my 48 weather files. The set-up I have is I am getting weather data from 4 cities from the web using: wunderground.com

Specifically, from this URL: https://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=KMDLAURE5&year=2018&month=2&graphspan=month&format=1

Am tasked with grabbing the data from that website, clean up the data, then save it.

In the problem I must access data from the internet 48 times, because their are 4 stations (cities) and 12 months. So I think I must create a file for each month and save all its monthly recorded temperature data. Then have all of those files stored into one directory file. I created a function for this:

import urllib

def accessData(ID, Month):
 url="https://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=" + str(ID) + "&year=2017=7&month=" + str(Month) + "&graphspan=month&format=1"

 infile = urllib.urlopen(url)
 readline = infile.readlines()
 infile.close()
 return readline

Now I was given a separate file called stations.csv with contains the data:

KCASANFR131,37.778,-122.408

KDCWASHI48,38.913,-77.031

IBRITISH359,49.256,-123.245

KNYNEWYO639,40.755,-74.007

I know that, for example KCASANFR131 is the station ID but what 37.778,-122.408 is? Unsure of what that represents.

Also, should I create a list that stores the station IDs? and then call them in my nested loop, or is there a way to call those IDs from the csv file itself?

Now that I have the function it makes sense to create a nested loop outside of the function and then to call the function inside the nested loop. For for each station ID, that will be the outer loop, and the 12 months that will be the inner loop.

After it completes one full iteration it should return a list containing the results of that web request.

Here is the code for that (removed the header with an if-statement:

if I were to store that ID data in a list

stationID = [KCASANFR131, KDCWASHI48, IBRITISH359, KNYNEWYO639]

for i in range(1,13):
    data = accessDat(i)
    filename = "0{}.2017.csv".format(i)
    outfile = open(filename, 'w')
    row_count = len(data)

      for j in range (2, row_count):
        if(data[j] != '<br>\n' and data[j] != 'n'):
          outfile.write(data[j])

outile.close()

Now what I am trying to do is keep that data separate from my scripts. So I want to save each file into a pre-made data directory using the format ­­.csv Example: KDCWASHI48­04­2017.csv

Also I need to have leading zeros, so the month should always be 2 characters long. For example, January will look like 01, and December will be 12, how does one do that using the str.zfill(2) method?

Answer 1

You've got most of it covered.

"1".zfill(2)

will result in 01 as you probably already guessed.

in the line

outfile = open(filename, 'w')

it would be good to give the specific path of your data directory.

outfile = open(os.path.join([*data_dir_path, month_path, filename]), 'w')

have a look at the post

How can I create a directory if it does not exist?

if you want to know how to create a directory if it doesnt exist.

I hope this solves all the issues, and Antry mentioned correctly that 49.256,-123.245 are Latitude/Longitude.

Answer 2

Structure your Code

Here are a couple guidelines to help you on your adventure.

When developing a programming, you usually want to design it like you are making a factory, elegant individual modules inside a chain of command. Instead of rigid/absolute code with no maintainability, scalability or abstraction of it's concepts

Think like legos, components and little bricks, to each their type and task.

Data-Structure

Start with your data-structures, they are the foundation of what you are working on, simple to make, you only need to know what you are working on to make them.

class Station:
    def __init__(self, _id, _lat, _long):
        self.id = _id
        self.lat = _lat
        self.long = _long

Here we have created a class which can be the home of the information from stations.csv this will let us manipulate PythonObjects instead of only iterating through lists (even if we ultimately will, we don't have to deal directly with it when we need it), this will also make your code a lot clearer and a lot easier to read, especially for people other than you.

Factories

Progress on to adding functionalities to your Classes through methods.

We'll start off by creating a factory class, which we could call Utility which would be responsible for importing and populating your classes for exemple, by doing this, we abstract our information layer from our logic layer, so you will never have to manipulate the parsing inside your main programs loop (I am not literally talking about __main__).

Let's start with parsing our Station information from our CSV:

import csv

stations = []

with open('stations.csv', 'r') as f:
    reader = csv.reader(f) 
    for row in reader:     
        clean_row = row.split(“,”)        
        stations.append(Station(clean_row[0], clean_row[1], clean_row[2])

(This does not take into account the possibility that you have your column names on your first row and assumes they are not.)

Now we have populated our list of stations with Station python objects, which we can use to access the data directly as such:

print(stations[0].id)

[...] To be continued; if requested. (kinda high on work)