The short answer
In large-scale data lakes built on Google Cloud Storage (GCS), storing all historical files, raw landing data, and temporary scratch files in the default storage tier creates unnecessary infrastructure expenses.
Google Cloud Storage provides four tiered storage classes to balance access frequency against storage price (check current pricing).
The four GCS storage classes
- Standard Storage: Designed for actively read, frequently queried data (such as active silver and gold analytics tables queried multiple times daily). Incurs the highest monthly storage fee per gigabyte, but zero retrieval fees and no minimum retention duration.
- Nearline Storage: Tailored for data accessed less than once a month (such as raw ingestion backups or monthly financial batch runs). Offers lower storage costs, with a 30-day minimum storage commitment and a per-gigabyte retrieval fee.
- Coldline Storage: Designed for data accessed less than once a quarter (such as disaster recovery backups or historical compliance tables). Further reduces monthly storage pricing, with a 90-day minimum retention duration.
- Archive Storage: Intended for cold archival and regulatory audits accessed less than once a year. Offers the lowest storage rate, with a 365-day minimum retention commitment and higher retrieval fees.
Automated Object Lifecycle Management
Rather than writing custom Python or Airflow scripts to scan buckets and migrate aging objects, configure GCS Object Lifecycle Management rules:
{
"rule": [
{
"action": {"type": "SetStorageClass", "storageClass": "NEARLINE"},
"condition": {"age": 30, "matchesPrefix": ["raw/"]}
},
{
"action": {"type": "SetStorageClass", "storageClass": "COLDLINE"},
"condition": {"age": 90, "matchesPrefix": ["raw/"]}
},
{
"action": {"type": "Delete"},
"condition": {"age": 365, "matchesPrefix": ["staging/temporary/"]}
}
]
}Google Cloud Storage automatically inspects object age and executes storage class transitions and object deletions asynchronously in the background.