Last time I
promised to blog about Azure Data Factory Data Flows, but decided to do this
first. My business problem was to process files on On-Premise file share with
SSIS without moving original files anywhere. Challenge was that with SSIS that
is not easily done without possibility to move the original file. Maybe with
log table and custom code.
to use Azure Data Factory (ADF) and Azure Blob Storage to tackle this
challenge. Even ADF is missing couple of critical features, but I managed to
use workarounds. I wanted to write about this, because I couldn’t find any good
instructions for this process in one place.
In ADF there
is a nice component called “Get Metadata”. Basically you could do this also
with SSIS script component. I decided to use File Creation Date and store that
into BatchControl table for the next run.