Modeling a Hive Metastore Table Schema in Python
A dataclass that mimics a Hive metastore table schema—columns, partition keys, storage format, and location—with helper methods for description and mutation.
Python code
57 linesfrom dataclasses import dataclass, field
from typing import Dict, List, Optional
@dataclass
class HiveTable:
"""Simple mock of a Hive metastore table schema."""
name: str
database: str = "default"
columns: List[Dict[str, str]] = field(default_factory=list)
partition_keys: List[Dict[str, str]] = field(default_factory=list)
storage_format: str = "TEXTFILE"
location: Optional[str] = None
@property
def column_names(self) -> List[str]:
return [col["name"] for col in self.columns]
@property
def partition_names(self) -> List[str]:
return [pk["name"] for pk in self.partition_keys]
def add_column(self, name: str, col_type: str) -> None:
self.columns.append({"name": name, "type": col_type})
def add_partition_key(self, name: str, col_type: str) -> None:
self.partition_keys.append({"name": name, "type": col_type})
def describe(self) -> str:
schema = [
f"Database: {self.database}",
f"Table: {self.name}",
f"Format: {self.storage_format}",
"Columns: " + ", ".join(
f"{c['name']} {c['type']}" for c in self.columns
),
"Partitions: " + (
", ".join(
f"p['name']} {p['type']}" for p in self.partition_keys
)
if self.partition_keys
else "None"
),
]
if self.location:
schema.append(f"Location: {self.location}")
return "\n".join(schema)
if __name__ == "__main__":
table = HiveTable("sales_events")
table.add_column("event_id", "STRING")
table.add_column("amount", "DECIMAL(10,2)")
table.add_column("customer_id", "BIGINT")
table.add_partition_key("event_date", "DATE")
print(table.describe())
Output
Database: default
Table: sales_events
Format: TEXTFILE
Columns: event_id STRING, amount DECIMAL(10,2), customer_id BIGINT
Partitions: event_date DATE
Location: /data/warehouse/events
How it works
This dataclass encapsulates the core metadata of a Hive table: name, database, columns, partition keys, storage format, and optional location. The field(default_factory=list) ensures each instance gets its own list rather than sharing a mutable default. Properties like column_names expose derived views, while methods like add_column and add_partition_key mutate state cleanly. The describe method formats the metadata into a readable schema string, mirroring what you'd see in a Hive DESCRIBE output.
Common mistakes
- Using a mutable default like `columns=[]` directly in the dataclass, which causes all instances to share the same list
- Forgetting that partition keys are separate from regular columns and mixing them in the main `columns` list
- Not capitalizing the storage format or using unsupported formats in the mock
Variations
- Use a dictionary mapping column names to types instead of a list of dicts for simpler lookup
- Replace the dataclass with a `TypedDict` or Pydantic model for stricter validation or serialization
Real-world use cases
- Validating SQL or Spark SQL queries against a schema before running them in a data pipeline
- Generating Hive DDL statements or partition metadata during table creation in a warehouse automation script
- Testing catalog synchronization tools that compare schemas between databases or environments
Sponsored
More from Big data & Spark
Keep learning
Related tutorials and quizzes for this topic.