Managing Configurations in Python Projects

Jan 22, 2025

When building a Python project, managing configurations can quickly become a headache. You have different environments like development, testing, and production, each with its own set of configurations. You want to keep things organized, avoid hardcoding sensitive information, and make it easy to switch between environments. In this post, I’ll walk you through how I tackled this problem using Pydantic and dotenv files. Let’s dive in!

The Problem: Managing Multiple Environments

In any non-trivial project, you’ll likely have at least three environments:

• Development (dev): Where you write and test your code locally
• Testing (test): Where automated tests run
• Production (prod): Where your application runs in the real world

Each environment has its own configuration. For example, database credentials, API keys, and file paths might differ between development and production. Hardcoding these values is a no-go—it’s error-prone and insecure. Instead, we want to manage these configurations externally, typically using environment variables.

ENV=dev
DEEPSEEK_API_KEY=your_api_key_here
FILE_UPLOAD_DIR=~/uploads
POSTGRES_USER=dev_user
POSTGRES_PASSWORD=dev_password
QDRANT_HTTP_PORT=6333

You should NEVER commit these dotenv files to Git. They often contain sensitive information like API keys and passwords. However, you should include a .env.example file with placeholder values. This way, other developers know what environment variables they need to set up.

Determining the Current Environment

To decide which environment to use, I introduced a special environment variable called ENV. This variable can be set to dev, test, or prod. Based on its value, the application will load the corresponding dotenv file (e.g., .env.dev, .env.test, or .env.prod).

Here’s how I implemented this in the find_dotenv.py file:

import os
import dotenv

def find_dotenv() -> str:
    # Get the value of ENV from the environment variables
    env = os.getenv("ENV")

    # Default to dev
    if env is None:
        env = "dev"

    # Select different env files based on ENV
    match env:
        case "dev":
            env_filename = ".env.dev"
        case "test":
            env_filename = ".env.test"
        case "prod":
            env_filename = ".env.prod"
        case _:
            raise ValueError(f"unknown env: {env}")

    # Find the dotenv file
    env_filepath = dotenv.find_dotenv(env_filename)

    return env_filepath

This function checks the ENV variable and selects the appropriate dotenv file. If ENV is not set, it defaults to the development environment.

Grouping Configurations with Pydantic

Dotenv files are great, but they don’t support nested fields. For example, you can’t directly group all PostgreSQL-related configurations under a POSTGRES key. To solve this, I used Pydantic, a powerful library for data validation and settings management.

Pydantic allows you to define configuration classes that can load values from environment variables. Here’s how I grouped PostgreSQL configurations:

from pydantic_settings import BaseSettings, SettingsConfigDict

class PostgresConfig(BaseSettings):
    user: str
    password: str
    db: str
    port: int
    data_dir: str
    uri: str

    model_config = SettingsConfigDict(
        env_file_encoding="utf-8",
        env_prefix="POSTGRES_",
        extra="ignore",
    )

Notice the env_prefix="POSTGRES_" line. This tells Pydantic to look for environment variables that start with POSTGRES_. For example, POSTGRES_USER, POSTGRES_PASSWORD, etc. This way, you can keep all PostgreSQL-related configurations neatly grouped together.

Similarly, I created a QdrantConfig class for Qdrant-related settings:

from pydantic_settings import BaseSettings, SettingsConfigDict

class QdrantConfig(BaseSettings):
    http_port: int
    grpc_port: int
    uri: str

    model_config = SettingsConfigDict(
        env_file_encoding="utf-8",
        env_prefix="QDRANT_",
        extra="ignore",
    )

Loading the Configuration

Now that we have our configuration classes, we need a way to load them. This is where the load_config function comes in. It loads the appropriate dotenv file based on the ENV variable and initializes the configuration classes.

Here’s the load_config function from the config.py file:

def load_config() -> Config:
    # Find the dotenv file based on the ENV
    env_filepath = find_dotenv()

    # Load the qdrant configuration
    qdrant_config = load_qdrant_config()

    # Load the postgres configuration
    postgres_config = load_postgres_config()

    # Load the configuration
    config = Config(
        qdrant=qdrant_config,
        postgres=postgres_config,
        _env_file=env_filepath,
    )

    return config

This function does the following:

• Finds the correct dotenv file using the find_dotenv function from python-dotenv package
• Loads the Qdrant and PostgreSQL configurations using their respective load_*_config functions
• Initializes the main Config class with these configurations

Putting It All Together

Finally, here’s the main Config class that ties everything together:

from pydantic_settings import BaseSettings, SettingsConfigDict
from pydantic import Field, field_validator
from pathlib import Path

class Config(BaseSettings):
    env: Env = Field(default=Env.DEV)
    file_upload_dir: Path
    deepseek_api_key: str
    qdrant: QdrantConfig = Field(default_factory=QdrantConfig)
    postgres: PostgresConfig = Field(default_factory=PostgresConfig)

    model_config = SettingsConfigDict(
        env_file_encoding="utf-8",
        extra="ignore",
    )

    @field_validator("file_upload_dir", mode="after")
    @classmethod
    def resolve_path(cls, path: Path) -> Path:
        return path.expanduser().resolve()

This class includes:

• A default environment (Env.DEV)
• A file_upload_dir field that resolves to an absolute path
• Nested configurations for Qdrant and PostgreSQL

Why This Design?

This design has several advantages:

• Environment-Specific Configurations: By using the ENV variable, you can easily switch between different environments without changing any code.
• Security: Sensitive information is stored in dotenv files, which are not committed to Git.
• Organization: Configurations are grouped logically using Pydantic classes, making the code easier to maintain.
• Flexibility: You can add more configurations by simply creating new Pydantic classes and adding them to the Config class.

Final Thoughts

By using dotenv files and Pydantic, you can keep your configurations organized, secure, and environment-specific. This approach has worked well for me, and I hope it helps you too!