hmsclient

hmsclient is a Python client for the Apache Hive Metastore, built directly on top of Thrift-generated stubs for the ThriftHiveMetastore service. Rather than requiring callers to hand-construct Thrift structs and manage socket transports themselves, it wraps the generated client in a single HMSClient class with context-manager semantics (with client as c:) and a handful of convenience helpers for common metastore operations.

The package bundles pre-generated Python bindings for both the hive_metastore and fb303 Thrift services, along with a generate.py script that can re-run the Thrift compiler against newer .thrift IDL files when the upstream Hive Metastore protocol changes. This makes it a thin, purpose-built adapter for teams that need to query or mutate Hive Metastore state (databases, tables, partitions) from Python without writing raw Thrift RPC code by hand.

What You Get

• An HMSClient class that extends the generated ThriftHiveMetastore.Client with open()/close()/context-manager support so a TSocket/TBufferedTransport/TBinaryProtocol connection is set up and torn down automatically
• Partition helpers — make_partition() derives a Partition object’s storage descriptor from its parent table, and add_partition()/drop_partitions()/drop_all_partitions() wrap the raw Thrift calls for common partition lifecycle operations
• Schema helpers — make_schema() builds a list of FieldSchema objects from short name:type strings, and parse_schema() converts FieldSchema objects back into readable name\ttype strings
• check_for_named_partition() turns the metastore’s NoSuchObjectException into a plain boolean existence check, avoiding a manual try/except at every call site
• Pre-generated Python bindings for the full hive_metastore and fb303 Thrift services (types, exceptions, and RPC client classes), plus generate.py to regenerate them from newer upstream .thrift files
• Configuration via HMS_HOST/HMS_PORT environment variables or explicit constructor arguments, defaulting to localhost:9083

Common Use Cases

• Scripting Hive Metastore partition maintenance (adding, checking, or dropping partitions) from Python jobs instead of shelling out to hive or beeline
• Building internal tooling or Airflow/orchestration tasks that need to read or write Hive table and partition metadata programmatically
• Polling the metastore’s notification event log (get_current_notification_id()) to track schema or partition changes over time
• Regenerating the Thrift client bindings against a newer Hive Metastore IDL version when upgrading a Hive cluster

Under The Hood

Architecture The package centers on a single HMSClient class (hmsclient/hmsclient.py) that subclasses the Thrift-generated ThriftHiveMetastore.Client and layers on context-manager semantics plus a handful of convenience methods — make_schema, parse_schema, make_partition, add_partition, check_for_named_partition, drop_partitions, drop_all_partitions, and get_current_notification_id. The hmsclient/genthrift/ package holds the raw Thrift-compiler output for both the hive_metastore and fb303 services (types, exceptions, and RPC client classes), regenerable via the root-level generate.py script against newer .thrift IDL files. It’s a flat, single-purpose wrapper with no plugin system or dependency injection — data flow is a direct translation of Python method calls into Thrift RPC calls over a TBufferedTransport/TSocket pair, so any change to the upstream Hive Metastore Thrift schema would need to propagate through the generated stubs and directly affect the wrapper’s assumptions.

Tech Stack Pure Python 3 (targeting 3.5/3.6 per its classifiers), packaged with a classic setup.py/setuptools layout rather than pyproject.toml. Runtime dependencies are thrift (the Apache Thrift Python bindings used for the RPC transport and protocol layers) and click (used by the standalone generate.py CLI, not by the client library itself). There’s no web framework, ORM, or database driver involved — communication happens directly over a raw socket to the Hive Metastore server (default port 9083) using TBinaryProtocol. Regeneration of the bundled Thrift stubs is a separate, manually-invoked process that fetches hive_metastore.thrift and fb303.thrift from upstream and re-runs the Thrift compiler.

Code Quality No test files exist anywhere in the repository, and there is no CI configuration (no GitHub Actions workflow, no .travis.yml, no tox.ini). Naming follows conventional Python snake_case, and several public methods carry Sphinx-style docstrings with :param/:type/:rtype annotations, but there are no type hints and no linter or formatter configuration. Error handling is minimal and selective — check_for_named_partition catches the specific NoSuchObjectException from the generated Thrift types to produce a boolean, while other methods simply propagate whatever exceptions the underlying Thrift layer raises.

What Makes It Unique The library’s value is a narrow but real convenience layer over an auto-generated RPC client: a context manager that opens and closes the socket transport automatically, plus helpers like make_schema (short name:type strings instead of hand-built FieldSchema objects) and make_partition (deriving a partition’s storage descriptor from its parent table) that remove boilerplate teams would otherwise write by hand against the raw generated Thrift client. It doesn’t introduce a novel technical approach — it’s a standard adapter pattern applied to one specific external service’s RPC interface.