Parse log lines into columns

Pandas data engineering interview problem. Difficulty: intermediate. Pattern: String Functions. About 16 minutes. Part of the Pro drill bank.

Extract structured IP, timestamp, HTTP method, URL path, status code, and payload bytes from raw log strings. Treat this as a production helper: match the contracted return shape, including empty and duplicate inputs.

Given a DataFrame logs with a single string column raw_log: Each row contains an access log in standard format: 192.168.1.1 - [10/Oct/2024:13:55:36] "GET /api/v1/users HTTP/1.1" 200 2326 Parse each log line into structured columns: ip: client IP address (string) timestamp: text between square brackets (string) method: HTTP request method such as GET or POST (string) path: request URL path (string) status: HTTP status code (integer) bytes: response body size in bytes (integer) Reset the index. Assign the extracted DataFrame to result.

Requirements

  • Output columns must match: ip, timestamp, method, path, status, bytes.
  • status and bytes must be integer typed.

Constraints

  • raw_log matches the Apache combined access log pattern.
  • Lines contain valid integers for status and bytes.

Examples

Input: logs raw_log 192.168.1.1 - [10/Oct/2024:13:55:36] "GET /api/v1/users HTTP/1.1" 200 2326 10.0.0.2 - [10/Oct/2024:13:55:38] "POST /api/v1/login HTTP/1.1" 401 512 Output: ip | timestamp | method | path | status | bytes 192.168.1.1 | 10/Oct/2024:13:55:36 | GET | /api/v1/users | 200 | 2326 10.0.0.2 | 10/Oct/2024:13:55:38 | POST | /api/v1/login | 401 | 512 Each component is parsed into its dedicated typed column.

Topics: lakebench, pandas, extract, regex.

More interview problems · All interview problems · Learn data engineering

intermediate

Parse log lines into columns

Interview-style drill: Extract structured IP, timestamp, HTTP method, URL path, status code, and payload bytes from raw log strings.

Given a DataFrame `logs` with a single string column `raw_log`: Each row contains an access log in standard format: `192.168.1.1 - [10/Oct/2024:13:55:36] "GET /api/v1/users HTTP/1.1" 200 2326` Parse each log line into structured columns: - `ip`: client IP address (string) - `timestamp`: text between square brackets (string) - `method`: HTTP request method such as GET or POST (string) - `path`: request URL path (string) - `status`: HTTP status code (integer) - `bytes`: response body size in bytes (integer) Reset the index. Assign the extracted DataFrame to `result`.