Skip to main content

๐Ÿ“‚ File-Based Sources to Snowflake Using Fivetran

๐Ÿ”— Overviewโ€‹

Fivetran supports automated ingestion of structured and semi-structured files (CSV, JSON, Excel, etc.) from cloud storage platforms directly into Snowflake. This enables seamless data ingestion pipelines with minimal setup and maintenance.


๐ŸŒ Supported File Storage Platformsโ€‹

  • Amazon S3
  • Google Cloud Storage (GCS)
  • Microsoft Azure Blob Storage
  • SFTP
  • FTPS
  • Dropbox
  • Box

๐Ÿ”„ Data Flow: File Storage โ†’ Fivetran โ†’ Snowflakeโ€‹

  1. Files dropped into cloud storage (e.g., S3 bucket)
  2. Fivetran connector monitors and ingests new/updated files
  3. Files parsed and loaded into Snowflake tables
  4. Auto-scheduling and deduplication supported

๐Ÿ› ๏ธ Steps to Configureโ€‹

โœ… Step 1: Set Up File Storage Sourceโ€‹

  • Go to Fivetran Dashboard โ†’ Add Connector
  • Choose file source (e.g., S3)
  • Provide access credentials:
    • IAM Role / Access Key for AWS
    • Shared keys for Azure
    • OAuth for Dropbox or Box
  • Define file path or wildcard (e.g., invoices/*.csv)

โœ… Step 2: Configure Destination (Snowflake)โ€‹

  • Provide:
    • Account Name (e.g., xy12345.ap-south-1)
    • Warehouse
    • Database
    • Schema
    • Role and Credentials

โœ… Step 3: Schema Mappingโ€‹

  • Auto-detect headers and data types (CSV, JSON)
  • Optionally map columns manually
  • Support for custom delimiter, encoding, line terminator, etc.

โœ… Step 4: Sync Settingsโ€‹

  • Set sync frequency
  • Choose file processing strategy:
    • Process once and ignore future changes
    • Rescan and update if file content changes

๐Ÿ“ Example: Folder Structure on S3โ€‹

s3://your-bucket/data/
โ”œโ”€โ”€ employees_2024_01.csv
โ”œโ”€โ”€ employees_2024_02.csv
โ””โ”€โ”€ employees_2024_03.csv

๐Ÿงฉ Key Featuresโ€‹

FeatureDescription
๐Ÿ”„ Auto-IngestionNo-code, continuous sync
๐Ÿง  Intelligent ParsingSchema detection and type conversion
โฑ๏ธ Scheduled SyncsCustomizable frequency
๐Ÿงน Data CleaningNull trimming, deduplication
๐Ÿ” Secure ConnectionEncrypted file access and credential management
๐Ÿ› ๏ธ Flexible FormatsCSV, JSON, TSV, Excel, Parquet

๐Ÿ“˜ Additional Resourcesโ€‹