The short answer
Without workload management, a single massive query or complex data science aggregation can consume all memory and CPU on a Redshift cluster, queuing up critical executive dashboards. Workload Management (WLM) defines rules and queues to isolate workloads and allocate resources predictably.
Cluster Memory Pool (100%) +----------------------------------------------------+ | ETL Queue (40% memory, max concurrency: 3) | | Long-running COPY, CTAS, and batch transformations| +----------------------------------------------------+ | BI Dashboard Queue (50% memory, concurrency: 15) | | Sub-second analyst queries, high priority | +----------------------------------------------------+ | Default / Ad-hoc Queue (10% memory) | +----------------------------------------------------+
Manual WLM versus Automatic WLM
- Manual WLM: You configure fixed query queues, set the maximum number of concurrent queries per queue, and assign a percentage of memory to each queue. You route queries using user groups or query group session variables.
- Automatic WLM: Redshift uses machine learning to manage memory and concurrency dynamically. You set priority levels (Critical, High, Normal, Low) for each queue. Redshift allocates more memory to memory-intensive queries and gives execution preference to higher-priority queues.
Short Query Acceleration (SQA)
Short Query Acceleration uses machine learning to inspect incoming queries and predict their execution duration. If an interactive query will finish in a few seconds, SQA pulls it out of the queue and executes it ahead of long-running queries, preventing dashboards from waiting behind multi-hour transformations.
Concurrency scaling
When query traffic spikes past cluster limits, Concurrency Scaling automatically spins up transient, serverless clusters behind the scenes to process read-only queries with zero downtime. Users earn 1 hour of free concurrency scaling credits for every 24 hours their cluster operates.