๐ File-Based Sources to Snowflake Using Fivetran
๐ Overviewโ
Fivetran supports automated ingestion of structured and semi-structured files (CSV, JSON, Excel, etc.) from cloud storage platforms directly into Snowflake. This enables seamless data ingestion pipelines with minimal setup and maintenance.
๐ Supported File Storage Platformsโ
- Amazon S3
- Google Cloud Storage (GCS)
- Microsoft Azure Blob Storage
- SFTP
- FTPS
- Dropbox
- Box
๐ Data Flow: File Storage โ Fivetran โ Snowflakeโ
- Files dropped into cloud storage (e.g., S3 bucket)
- Fivetran connector monitors and ingests new/updated files
- Files parsed and loaded into Snowflake tables
- Auto-scheduling and deduplication supported
๐ ๏ธ Steps to Configureโ
โ Step 1: Set Up File Storage Sourceโ
- Go to Fivetran Dashboard โ Add Connector
- Choose file source (e.g., S3)
- Provide access credentials:
- IAM Role / Access Key for AWS
- Shared keys for Azure
- OAuth for Dropbox or Box
- Define file path or wildcard (e.g.,
invoices/*.csv)
โ Step 2: Configure Destination (Snowflake)โ
- Provide:
- Account Name (e.g.,
xy12345.ap-south-1) - Warehouse
- Database
- Schema
- Role and Credentials
- Account Name (e.g.,
โ Step 3: Schema Mappingโ
- Auto-detect headers and data types (CSV, JSON)
- Optionally map columns manually
- Support for custom delimiter, encoding, line terminator, etc.
โ Step 4: Sync Settingsโ
- Set sync frequency
- Choose file processing strategy:
- Process once and ignore future changes
- Rescan and update if file content changes
๐ Example: Folder Structure on S3โ
s3://your-bucket/data/
โโโ employees_2024_01.csv
โโโ employees_2024_02.csv
โโโ employees_2024_03.csv
๐งฉ Key Featuresโ
| Feature | Description |
|---|---|
| ๐ Auto-Ingestion | No-code, continuous sync |
| ๐ง Intelligent Parsing | Schema detection and type conversion |
| โฑ๏ธ Scheduled Syncs | Customizable frequency |
| ๐งน Data Cleaning | Null trimming, deduplication |
| ๐ Secure Connection | Encrypted file access and credential management |
| ๐ ๏ธ Flexible Formats | CSV, JSON, TSV, Excel, Parquet |