Skip to content

Commit 1cd6838

Browse files
committed
feat: Implement Day 12B Great Expectations Cloud integration with comprehensive data quality validation framework
1 parent 9957803 commit 1cd6838

File tree

8 files changed

+1208
-0
lines changed

8 files changed

+1208
-0
lines changed

day12/README.md

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,34 @@
66

77
---
88

9+
## 🎯 Two Implementations: A/B Comparison
10+
11+
This project demonstrates **two approaches** to data quality validation, showcasing both custom framework development and enterprise tool expertise:
12+
13+
| Aspect | **Day 12A** (Custom Framework) | **Day 12B** (GE Cloud) |
14+
|--------|--------------------------------|------------------------|
15+
| **Location** | [`day12/`](.) | [`day12b/`](../day12b) |
16+
| **Approach** | Custom validation framework | Great Expectations Cloud |
17+
| **When to Use** | Dependency issues, air-gapped environments, learning GE concepts | Production enterprise deployments, team collaboration |
18+
| **Dependencies** | Minimal (pandas, faker) | Full GE Cloud SDK |
19+
| **Data Docs** | Local HTML generation | Cloud-hosted at app.greatexpectations.io |
20+
| **Expectations** | Custom Python classes (10 expectations) | Native GE expectations (8 native) |
21+
| **Validation Results** | Local JSON files | GE Cloud storage + UI |
22+
| **Team Collaboration** | Single developer | Multi-user with GE Cloud |
23+
| **Setup Time** | <5 minutes | ~15 minutes (Cloud account required) |
24+
| **Portfolio Message** | "Can build validation frameworks from scratch" | "Can use enterprise data quality tools" |
25+
26+
**💡 Why Both?**
27+
- **Day 12A** demonstrates problem-solving when dependencies fail (applied PIVOT RULE)
28+
- **Day 12B** shows proper enterprise GE Cloud integration for production use
29+
- Together they prove both **deep conceptual understanding** AND **practical tool expertise**
30+
31+
**👉 Quick Start:**
32+
- **For Custom Framework:** Continue reading below, or jump to [Day 12A Quick Start](#how-to-use-this-project)
33+
- **For GE Cloud:** See [Day 12B Setup Guide](../day12b/README_12B.md)
34+
35+
---
36+
937
## Navigation
1038

1139
### Quick Access (By Role)

day12b/.env.example

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
# ==============================================================================
2+
# DAY 12B - GREAT EXPECTATIONS CLOUD CONFIGURATION
3+
# ==============================================================================
4+
# Copy this file to ../config/.env and fill in your GE Cloud credentials
5+
6+
# Great Expectations Cloud Credentials
7+
# Get these from: https://app.greatexpectations.io/settings/tokens
8+
DAY12B_GE_CLOUD_ORG_ID=your-organization-id-here
9+
DAY12B_GE_CLOUD_ACCESS_TOKEN=your-access-token-here
10+
DAY12B_GE_CLOUD_BASE_URL=https://app.greatexpectations.io
11+
12+
# Logging
13+
DAY12B_LOG_LEVEL=INFO
14+
15+
# Notification Settings
16+
DAY12B_NOTIFY_ON_FAILURE=true
17+
DAY12B_SLACK_WEBHOOK_URL=https://hooks.slack.com/services/YOUR/WEBHOOK/URL
18+
19+
# ==============================================================================
20+
# GETTING YOUR GE CLOUD CREDENTIALS
21+
# ==============================================================================
22+
# 1. Sign up at https://greatexpectations.io/cloud
23+
# 2. Navigate to Settings → Access Tokens
24+
# 3. Create a new token with "Data Context" permissions
25+
# 4. Copy your Organization ID from the URL or Settings page
26+
# 5. Paste both values above
27+
28+
# ==============================================================================
29+
# USAGE
30+
# ==============================================================================
31+
# 1. Fill in your GE Cloud credentials above
32+
# 2. Copy to config directory: cp day12b/.env.example ../config/.env
33+
# 3. Run setup: python3 day12b_SETUP_cloud.py
34+
# 4. Create expectations: python3 day12b_CREATE_expectations.py
35+
# 5. Run validation: python3 day12b_RUN_validation_cloud.py
36+
37+
# ==============================================================================
38+
# COMPARISON: DAY 12A vs DAY 12B
39+
# ==============================================================================
40+
# Day 12A: Custom validation framework (no GE Cloud)
41+
# - Pros: Works without external dependencies, demonstrates deep understanding
42+
# - Cons: Limited features, no cloud Data Docs, manual maintenance
43+
#
44+
# Day 12B: Great Expectations Cloud (this version)
45+
# - Pros: Enterprise features, cloud Data Docs, team collaboration, managed infrastructure
46+
# - Cons: Requires GE Cloud account, external dependency

0 commit comments

Comments
 (0)