Casos d'Ús en Big Data

9.1. Ingesta en temps real

Architecture:

External System → [Webhook] → N8N → [Validate] → [Transform] → [Load to Data Lake]
                                ↓
                          [Redis: Buffer]
                                ↓
                        [Batch every 60s]
                                ↓
                         [S3: Parquet files]

Implementació:

[Webhook Trigger]
  Path: /events/ingest
  ↓
[Validate Schema]
  ↓
[Redis: Add to Buffer]
  Key: events:buffer:{{$now.toFormat('yyyyMMddHHmm')}}
  Value: {{$json}}
  ↓
[Respond: 202 Accepted]

---

[Schedule: Every 60s]
  ↓
[Redis: Get Buffered Events]
  ↓
[IF: Count > 0]
  ↓
[Transform to Parquet]
  ↓
[S3: Upload]
  Path: s3://data-lake/events/{{$now.toFormat('yyyy/MM/dd/HH')}}/batch_{{$now.toUnixInteger()}}.parquet
  ↓
[Redis: Clear Buffer]

9.2. Data Warehouse Pipeline

Complete ETL Pipeline:

[Schedule: Daily 2AM]
  ↓
[Extract from Sources]
  ├─ [Salesforce API]
  ├─ [PostgreSQL DB]
  └─ [S3 Files]
  ↓
[Merge All Sources]
  ↓
[Transform & Clean]
  ├─ Deduplicate
  ├─ Validate
  ├─ Enrich
  └─ Normalize
  ↓
[Load to Staging]
  PostgreSQL: staging.raw_data
  ↓
[dbt: Transform in DWH]
  Run models:
    - staging models
    - fact tables
    - dimension tables
  ↓
[Data Quality Checks]
  ↓
[IF: All Pass]
  ├─ true → [Promote to Production]
  └─ false → [Rollback + Alert]
  ↓
[Update Metadata]
  ↓
[Send Success Report]

9.3. ML Operations

ML Pipeline:

[Schedule: Weekly]
  ↓
[Fetch Training Data]
  BigQuery: last 90 days
  ↓
[Data Preparation]
  - Feature engineering
  - Train/test split
  ↓
[Trigger Training Job]
  SageMaker/Vertex AI
  ↓
[Wait for Completion]
  ↓
[Evaluate Model]
  Metrics: accuracy, F1, AUC
  ↓
[IF: Metrics > Threshold]
  ├─ true → [Deploy New Model]
  │          ↓
  │        [Update Model Registry]
  │          ↓
  │        [Switch Production Traffic]
  │
  └─ false → [Alert ML Team]
              ↓
            [Keep Current Model]

9.4. Automatització de Reports

[Schedule: Daily 8AM]
  ↓
[BigQuery: Run Analytics Queries]
  - Daily sales by region
  - Top products
  - Customer growth
  ↓
[Generate Visualizations]
  Create charts with Chart.js
  ↓
[Google Sheets: Update Dashboard]
  Spreadsheet ID: abc123
  Range: 'Daily Metrics'!A1:Z100
  ↓
[Generate PDF Report]
  Template: company_report.html
  Data: {{$json.metrics}}
  ↓
[Send Email to Stakeholders]
  To: executives@company.com
  Subject: Daily Business Report - {{$today}}
  Attachments: report.pdf
  ↓
[Slack: Post Summary]
  Channel: #daily-metrics
  Message: "📊 Daily report ready!"