I work with a variety of data sources (Salesforce, Google, CSVs, other REST APIs etc.) and types (tabular, key-value) and have a number of different Python wrappers that expose these feeds.
These are difficult to maintain, and at present, don't follow a consistent interface/convention - which I recognise is a bad place to be. The list of sources from which I need to retrieve data is growing all the time, and so I want to find/come up with a better solution.
My gut feel says to try and design some kind of 'generic' data gateway, built in Python, that can provide a consistent interface to the data I require, regardless of underlying source and method of retrieval (e.g. HTTP request vs. database call). Problem is, I'm struggling to find any concrete examples of this type of setup in use, which to me seems odd, as this appears to be a common problem for many organisations, where data comes from many sources with lots of different schemas and structures.
I'd appreciate any suggestions on how to possibly begin thinking about this, and/or directions towards existing solutions that may help.