how to specify s3 config for spectrify python package?

Question

how to specify this s3_config object for python spectrify package ?

from spectrify.export import RedshiftDataExporter
RedshiftDataExporter(sa_engine, s3_config).export_to_csv('my_table')

Answer 1

The documentation for the spectrify package is not great, but a quick read through the code and the examples will show you that s3_config can be an instance of S3Config or SimpleS3Config, both defined by the spectrify package. Here's an example:

csv_path_template = 's3://my-bucket/my-table/csv/{start.year}/{start.month:02d}/{start.day:02d}'
spectrum_path_template = 's3://my-bucket/my-table/spectrum/partition_key={start}'

# Construct a S3Config object with the source CSV folder and
# destination Spectrum/Parquet folder on S3.
csv_path = csv_path_template.format(start=start_date)
spectrum_path = spectrum_path_template.format(start=start_date)
s3_config = SimpleS3Config(csv_path, spectrum_path)