Modeling a Hive Metastore Table Schema in Python

A dataclass that mimics a Hive metastore table schema—columns, partition keys, storage format, and location—with helper methods for description and mutation.

Easy Python 3.9+ Aug 9, 2026 Big data & Spark 13 views 0 copies

Python code

57 lines
Python 3.9+
from dataclasses import dataclass, field
from typing import Dict, List, Optional


@dataclass
class HiveTable:
    """Simple mock of a Hive metastore table schema."""
    name: str
    database: str = "default"
    columns: List[Dict[str, str]] = field(default_factory=list)
    partition_keys: List[Dict[str, str]] = field(default_factory=list)
    storage_format: str = "TEXTFILE"
    location: Optional[str] = None

    @property
    def column_names(self) -> List[str]:
        return [col["name"] for col in self.columns]

    @property
    def partition_names(self) -> List[str]:
        return [pk["name"] for pk in self.partition_keys]

    def add_column(self, name: str, col_type: str) -> None:
        self.columns.append({"name": name, "type": col_type})

    def add_partition_key(self, name: str, col_type: str) -> None:
        self.partition_keys.append({"name": name, "type": col_type})

    def describe(self) -> str:
        schema = [
            f"Database: {self.database}",
            f"Table: {self.name}",
            f"Format: {self.storage_format}",
            "Columns: " + ", ".join(
                f"{c['name']} {c['type']}" for c in self.columns
            ),
            "Partitions: " + (
                ", ".join(
                    f"p['name']} {p['type']}" for p in self.partition_keys
                )
                if self.partition_keys
                else "None"
            ),
        ]
        if self.location:
            schema.append(f"Location: {self.location}")
        return "\n".join(schema)


if __name__ == "__main__":
    table = HiveTable("sales_events")
    table.add_column("event_id", "STRING")
    table.add_column("amount", "DECIMAL(10,2)")
    table.add_column("customer_id", "BIGINT")
    table.add_partition_key("event_date", "DATE")

    print(table.describe())

Output

stdout
Database: default
Table: sales_events
Format: TEXTFILE
Columns: event_id STRING, amount DECIMAL(10,2), customer_id BIGINT
Partitions: event_date DATE
Location: /data/warehouse/events

How it works

This dataclass encapsulates the core metadata of a Hive table: name, database, columns, partition keys, storage format, and optional location. The field(default_factory=list) ensures each instance gets its own list rather than sharing a mutable default. Properties like column_names expose derived views, while methods like add_column and add_partition_key mutate state cleanly. The describe method formats the metadata into a readable schema string, mirroring what you'd see in a Hive DESCRIBE output.

Common mistakes

  • Using a mutable default like `columns=[]` directly in the dataclass, which causes all instances to share the same list
  • Forgetting that partition keys are separate from regular columns and mixing them in the main `columns` list
  • Not capitalizing the storage format or using unsupported formats in the mock

Variations

  1. Use a dictionary mapping column names to types instead of a list of dicts for simpler lookup
  2. Replace the dataclass with a `TypedDict` or Pydantic model for stricter validation or serialization

Real-world use cases

  • Validating SQL or Spark SQL queries against a schema before running them in a data pipeline
  • Generating Hive DDL statements or partition metadata during table creation in a warehouse automation script
  • Testing catalog synchronization tools that compare schemas between databases or environments

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Big data & Spark

Related tutorials and quizzes for this topic.