As a tech enthusiast, I have a habit of accumulating “retired” hardware. Over the years, several iPhones and Macs have ended up in drawers, simply collecting dust. I wanted to change that—not just to give them a second life, but to reclaim my digital sovereignty.

My goal wasn’t just to build a server; it was to build a private ecosystem. I wanted to host my own personal ChatGPT interface (via Open WebUI) and my own private “Google Drive/Dropbox” (via OpenCloud). I wanted my data to live on my hardware, under my control, rather than in a corporate cloud.

I looked at a 2017 MacBook Pro sitting in my drawer. It still ran, even if the keyboard was a bit temperamental. I thought: Why not turn this into a “free” Kubernetes cluster?

So, the journey began.

The Setup: From Silicon to Kubernetes

The first step was the basics. I installed Ubuntu 24.04 Desktop, using an old Raspberry Pi SD card to handle the BIOS installation since I didn’t have a spare USB drive handy. It was a slow start, but we were moving.

The first real hurdle was the hardware itself—specifically, getting the Broadcom Wi-Fi drivers to behave. After a frustrating session of trying to modify official driver source code with Claude’s help, I eventually found a script that actually worked: https://gist.github.com/torresashjian/e97d954c7f1554b6a017f07d69a66374. I had to ensure this script runs on every restart, so as long as the Mac is powered, the Wi-Fi stays up.

Once I was online, I wanted a setup that felt modern. I enabled SSH for remote management, but for the actual updates, I wanted a “GitOps” workflow. I deployed k3s for lightweight orchestration and connected it to Flux CD. This meant that instead of manual configuration, Flux would pull directly from my GitLab repository and automatically apply any changes to the cluster.

The “Genius” Plan (and the CG-NAT Reality Check)

Once the core system was humming, I decided to get “clever.”

I wanted to host my services so they were accessible from anywhere, but I didn’t want to pay my ISP for a static IP. I’ll be smart about this, I thought. I’ll just write a simple Python script that checks my public IP every 15 minutes and updates my domain’s A records via an API. No static IP? No problem!

I even set up a takk job to automate this. I felt like a networking wizard.

I was wrong.

As it turns out, my network provider was using CG-NAT (Carrier-Grade NAT). My “clever” script was perfectly updating my domain to point to an IP address that didn’t actually lead back to my house. I was essentially shouting into a void.

Thankfully, a quick, slightly humbling phone call to my network provider solved it—they opened up a port for me, finally allowing the outside world to actually reach my MacBook.

The Pivot to Public: Why HTTPS Matters

Now, I could have theoretically kept everything on my local network and accessed it via a VPN. However, I ran into a practical roadblock: OpenCloud requires HTTPS to function properly.

To get real SSL certificates and a professional setup, I needed a proper public endpoint. This led me to Domeneshop.no. They are a fantastic local DNS provider that not only offers a great API for my IP-update script but also integrates beautifully with Let’s Encrypt through a webhook adaptor. This meant that once my DNS was pointing to my host, the cluster could automatically handle SSL certificates, making the whole experience seamless and secure.

Managing the Chaos with Takk

With the network and DNS stable, I started deploying the “big” services: OpenCloud for my files and Open WebUI for my AI.

However, as I added more services—requiring Postgres databases, Redis caches, MinIO storage, and complex SSL certificates—the Kubernetes YAML files started to become a nightmare. I found myself constantly fighting with incorrect service names, misconfigured environment variables, and metadata mismatches.

This is where the project shifted from “playing with k8s” to “building a professional system.” To manage this complexity, I turned to takk—an infrastructure-as-code framework that I maintain.

Instead of writing hundreds of lines of brittle YAML, I define my infrastructure in Python. takk handles the heavy lifting: it generates the k8s configs, ensures all secrets and environment variables are consistent across services, and manages the relationships between third-party resources like databases and the apps that need them.

from typing import Annotated
from pydantic import AnyUrl, Field, PostgresDsn
from pydantic_settings import BaseSettings
from takk import Project, NetworkApp, ServiceUrl
from takk.secrets import PostgresHost, PostgresName, PostgresPassword, PostgresUsername, S3AccessKey, S3BucketName, S3Endpoint, S3RegionName, S3SecretKey

class OpenWebUISettings(BaseSettings):
    database_url: PostgresDsn
    s3_bucket_name: S3BucketName
    s3_region_name: S3RegionName
    s3_endpoint_url: S3Endpoint
    s3_secret_access_key: S3SecretKey
    s3_access_key: S3AccessKey
    # Currently using Scaleway for the heavy lifting, 
    # but this architecture makes it trivial to point to a local LLM later.
    openai_api_base_url: str = "https://api.scaleway.ai/v1"
    openai_api_key: str = "... "

project = Project(
    name="infra",
    chat=NetworkApp(
        docker_image="ghcr.io/open-webui/open-webui:v0.8.6",
        port=8080,
        settings=[OpenWebUISettings]
    )
)

The “Aha!” Moment: Connecting my Brain to my AI

With the infrastructure stable, I hit the most rewarding part of the project.

I had my files in OpenCloud and my AI in Open WebUI, but they were living in separate worlds. My chat interface could talk to me, but it couldn’t “see” my data. I couldn’t ask it, “What were my notes on the project from yesterday?”

I needed a bridge. Since there wasn’t a standard way to expose OpenCloud files to an LLM, I built one using FastAPI. I created a custom toolset that exposed my Obsidian paths, tags, and backlinks via an API.

Suddenly, the magic happened. I could ask my self-hosted AI:

“List out a few items with the ‘Papers’ tag.”

Because I was using takk, adding this new FastAPI “bridge” to my cluster was trivial. I just added it to my Project config, and the cluster handled the rest. I even extended this to Neovim, allowing me to get inline AI completions in my terminal that could actually “read” my local files.

project = Project(
    # ... existing config
    obsidian_tools=FastAPIApp(
        app="src.takk_k8s.obsidian_tools:app",
        settings=[ObsidianToolsSettings],
        compute=Compute(mvcpu_limit=250, mb_memory_limit=256)
    )
)

Scaling and Staying Sane

Once the “brain” was connected, I realized a cluster is only useful if it stays online. I didn’t want to find out my setup was broken only when I tried to use it.

I used the same logic to add MCP (Model Context Protocol) servers to monitor my cluster’s health and added Prometheus-based Alerting. Now, if a service enters a crash loop or a job fails, I get an email immediately. Everything—from the network to the alerts—is defined in code.

project = Project(
    # ... existing config
    alerts=[
        Alert(
            name="CrashLoop",
            query="increase(kube_pod_container_status_restarts_total[1h]) > 2",
            notify=[mats],
            severity="critical"
        ),
        Alert(
            name="FailingJobs",
            query="...",
            severity="critical",
            notify=[mats]
        ),
    ],
)

Network Architecture

graph TD
    subgraph "External"
        User[User Devices: Laptop/Mobile]
        DNS[Domeneshop DNS]
        LE[Let's Encrypt]
    end

    subgraph "Home Network"
        ISP[ISP / Open Port]
        Mac[MacBook Pro: k3s Cluster]
        PI[Pi-hole]
        OC[OpenCloud]
        OW[OpenWebUI]
        FA[FastAPI: Obsidian Tools]
    end

    User -- "Public URL" --> DNS
    DNS -- "Resolves to" --> ISP
    ISP -- "Routes to" --> Mac
    LE -- "Certificates" --> Mac
    Mac -- "Local DNS" --> PI
    Mac -- "Host Services" --> OC
    Mac -- "Host AI Interface" --> OW
    OW -- "API Calls" --> FA
    FA -- "Reads" --> OC
    OC -- "Syncs Files" --> User

Future Horizons

What started as a way to clear out a desk drawer has turned into a fully functioning, self-sovereign digital headquarters. I own my data, I own my AI, and I own the code that runs it all.

The next steps?

1. Heavy Lifting: Moving from Scaleway to a dedicated machine with a GPU for fully local, high-speed LLMs (Gemma 4, etc.).
2. Self-Healing: Integrating a coding agent that can monitor the logs and suggest (or apply) fixes to the cluster itself.
3. Hardened Backups: Moving from “syncing” to a professional backup strategy using tools like Velero.

What do you think? If you’ve ever tried to host your own AI or struggled with the “joy” of CG-NAT, I’d love to hear your thoughts in the comments!

They’ll be things like:

• A preprocessing step crashing because a column is missing.
• Data arriving in a slightly different format than your code expects.
• The input schema for the model doesn’t match the data provided at inference.

These problems aren’t glamorous, but they’re the ones that break production the fastest. The fix? Add smoke tests that run the pipeline end-to-end with tiny, synthetic datasets.

Why smoke tests work

The idea is simple: run your pipeline end-to-end using synthetic data that doesn’t need big hardware. The goal isn’t to prove the model is good — it’s to prove the pipeline still runs and still respects its expected input and output formats.

With this approach, you can catch schema changes, broken preprocessing logic, or missing dependencies before you commit to running a full training job.

A smoke test saves you from an 8-hour training run that crashes on a missing column. Rather it would catch it in seconds.

So how do we create these datasets?

Generating synthetic data

You can create synthetic test data in two main ways:

• Fully randomised data — great for checking that schemas match and code runs, without caring about meaning.
• Partially controlled data — lets you embed known patterns so you can confirm the model can still detect them.

If you’re working with a framework that defines your data contracts or feature views, you can often generate random data directly from those definitions. Here is an example using aligned and the wine quality dataset:

@data_contract()
class WineQuality:
    fixed_acidity = Float64()
    volatile_acidity = Float64()
    citric_acid = Float64()
    residual_sugar = Float64()
    chlorides = Float64()
    free_sulfur_dioxide = Float64()
    total_sulfur_dioxide = Float64()
    density = Float64()
    pH = Float64().bounded_between(0, 14)                  
    sulphates = Float64()
    alcohol = Float64().bounded_between(0, 20)
           
    quality = Int64().bounded_between(0, 10)
    
df = await WineQuality.n_examples(100).to_polars()

This contract enforces valid value ranges, or infer defaults based on the data types — which means we can safely auto-fill any column we don’t explicitly set.

Random data is good, but sometimes you need controlled patterns.

Defining a known pattern

Let’s make a known pattern with the following rules:

• Wines with alcohol > 12 get quality = 8
• Wines with alcohol <= 12 get quality = 4

We’ll specify just those two columns, and let the rest be random within their contract bounds:

from aligned.source import RandomDataSource

# Defining the pattern
patterned_source = RandomDataSource.with_values({
    "alcohol": [11, 13, 15, 9, 14],
    "quality": [4, 8, 8, 4, 8]
})

store = await ContractStore.from_dir(".")

# Switching to the new data source
store = store.update_source_for(
    WineQuality,
    patterned_source
)

Because the other columns in WineQuality aren’t defined here, they’ll be filled in automatically with valid random values. However, all rows will have the same random value, creating a strong deterministic relationship between the alcohol and quality columns. With that setup, we can now load the dataset and train a model.

from sklearn.ensemble import RandomForestClassifier

df = await store.contract(WineQuality).all().to_pandas()

X = df.drop("quality", axis=1)
y = df["quality"]

model = RandomForestClassifier().fit(X, y)

Next, let’s validate the pipeline by predicting on a controlled test-set.

Generating prediction samples

Now that we have a model, let’s make some predictions to test and validate that the model picked up on the known pattern.

Thankfully we can generate new prediction samples by defining only the features that matter for your pattern — here, alcohol — and letting everything else be random:

pred_df = await store.contract(WineQuality).select(X.columns).features_for({
    "alcohol": [11, 14]
}).to_polars()

preds = model.predict(pred_df)

assert preds.to_list() == [4, 8]

With this, we can finally check that our pipeline loads the correct data and trains a model that finds the known pattern. Furthermore, if you connect a proper model registry to this pipeline could you also test that those integrations work as expected.

Conclusion

Smoke tests won’t guarantee that your model is cutting-edge, but they will guarantee that your pipeline is alive and dependable. By mixing randomized data for schema validation with controlled patterns for behavioral checks, you catch the kinds of issues that cause real outages. Missing columns, broken preprocessing, or silent feature drops.

You can even extend these tests beyond data and training by wiring in a model registry. Verifying that a trained model can be stored, retrieved, and reloaded successfully gives you end-to-end confidence — from raw data all the way to deployable artifacts.

These checks are fast, cheap, and easy to automate. Making them perfect for CI / CD. Enabling us to check every code change in seconds, long before you commit resources to a full training run. The result is fewer surprises, faster iteration, and more confidence that your pipeline is doing what you expect. So when it’s time to improve accuracy, you can focus on the model, not firefighting infrastructure.

For more details on using aligned, check out the project on GitHub or its documentation.

The ML Problem

However, to implement such a project was an ML related problem needed. Therefore, I thought it would be fun to create an ML model that predicted my expenses for the next month, but grouped by categories like transportation, groceries, sports, etc.

Potentially not the most useful model, as I know fairly well how my expenses will be because of a fairly good banking app. However, since I know the data well, will it be easier to figure out when the model is way off, which makes it the perfect use-case for an end-to-end ML platform.

However I started looking into the API of my bank, and saw that I was lacking the data I needed… More precisely, I was lacking some of the data that told me if it was spent on transactions, groceries, etc. However, I thankfully had it for all my in-person VISA card transactions.

Therefore, this post will go info how I created a model that classified my bank transactions, while ensuring high quality datasets, how I made it easy to maintain models that depend on each other, and how I monitored the performance in production.

The Source Data

Before I can start doing anything really practical, would I need to get my hand on some real data. Or at least understand how the data would behave to some degree.

This lead to write a simple API integration with my bank that loaded my transaction data given a time period. E.g. from 1 January 2023 to 1 January 2024, or by providing a number of records to load.

I also defined the schema of the response I expected, which you can see bellow. We will come back to the package used to define the schema, as it is not your ordinary pydantic model, and for a very good reason.

class Transaction:
    transaction_id = String().as_entity()
    
    amount = Float()
    text = String()
    
    user_id = String().description(
        "Only a hard coded value, as it is only my own user"
    )
    account_id = String()
    
    accounting_date = Timestamp()
    interest_date = EventTimestamp()


    transaction_type = String()

    card_details = Json().is_optional().description(
        "Is only set where transaction_type = VISA VARE"
        "Contains a lot of interesting details tho"
        "E.g. merchant category, location, etc."
    )

However, rather then going for an ordinary SQL database, did I go for more of a data lake solution. Therefore, all of the data would be loaded from an API, but stored in a Parquet file locally. So how would I manage the data? This is where aligned comes into play. Aligned is a package used to manage data for ML applications, and is also the package used to define the schema for my transaction data.

However, in addition to defining schemas, can we add different types of sources. Like a load source, materialised sources, and stream sources. But also define how often we expect them to be updated. Which can be done with the following:

@feature_view(
    name="transactions",
    source=api_source,
    materialized_source=FileSource.parquet_at(
        "source_data/transactions.parquet"
    ),
    acceptable_freshness=timedelta(days=3),
    unacceptable_freshness=timedelta(days=6),
)
class Transaction:
    transaction_id = String().as_entity()
    amount = Float()
    ...

ML Data Management

Now that I finally had the source data loaded was it time to create the model.

This ment a few things:

• I needed to transform the raw data to extract the merchant category out from from the json data.
• I needed to create a dataset where the merchant category is the ground truth.
• I needed to define which inputs to use for the model.

Thankfully was the first step very simple. It was done by adding a new property to the schema, but that used the card_details as the source. Therefore, creating a “type-safe” computation, with data lineage and a lot of goodies.

@feature_view(...)
class Transaction:
    transaction_id = String().as_entity()
    ...
    
    card_details = Json().is_optional()
    merchant_name = card_details.field(
        "merchantName", 
        as_type=String()
    )

When it comes to the second part about defining the ground truth do aligned have an elegant method. aligned introduces the concept of model contracts which tell us what a model is predicting, their ground truth, and a lot of other related metadata. As a result can we use a similar syntax as the transformation to define that the merchant_name should be the ground truth and the predicted value.

transaction = Transaction()

@model_contract(
    name="transaction_category",
    input_features=[...],
)
class TransactionCategory:
    transaction_id = String().as_entity()
    predicted_at = EventTimestamp()
    
    predicted_category = (transaction.merchant_name
        .as_classification_label()
    )

By adding the as_classification_label will we tell aligned where to find the ground truth, what we are predicting, and what the model should name the prediction, as it is set to the predicted_category field.

Create a Training Dataset

In order to train a model would I need a dataset, and thanks to the defined relationship in the last section will it be easy to create. The only thing I need to define is who I want to create a training dataset for, and in my case do I want it for all transactions where we have a ground truth value. However, the ground truth check will be done automatically, so all we need to care about will be for who to load our data for. Therefore, this can be loaded with the following.

entities = Transactions.query().all_columns(limit=None)

And to create a dataset can we do the following.

store = await FeatureStore.from_dir(".")

datasets = (store.model("transaction_category")
    .with_labels()
    .features_for(entities)
    .train_test_validate(
        train_size=0.7,
        validate_size=0.15
    )
)

This will make sure of a few things.

• We have a ground truth in each row.
• We do not train on data that was created after we would have made the prediction. Also known as point-in-time correct data.
• Making sure we do not have data leakage in time, by splitting the dataset based on event timestamps. If there exist no will it do a random split.

From here we can load different datasets by accessing the train, test, or validate property.

train_data = await datasets.train.to_polars()

model = LogicticRegression()
model.train(
    X=train_data.input, 
    y=train_data.labels
)

for dataset in [datasets.test, datasets.validate]:
    eval_data = dataset.to_polars()
    preds = model.predict(eval_data.input)
    
    accuracy = (preds == eval_data.labels).mean()

And that’s it, I finally had my model which I then stored in a MLFlow registry. I skipped the MLFlow implementation, to keep it simple.

Use the model

There are a wide range of ways to deploy a model. A common way is use a MLFlow server, which can either be started locally or a data platform such as Databricks.

This is where aligned helps even more, as it makes it possible to describe where, and how to use a model. Therefore, we can define that we have an MLFlow model, served at a specific url. Or for simplicity sake, since the model has a low memory footprint can I use it in memory. Therefore, I modified the model contract to the following

@model_contract(
    name="transaction_category",
    input_features=[...],
    exposed_model=ExposedModel.in_memory_mlflow(
        model_name="transaction_category",
        model_alias="Champion",

        prediction_column="predicted_category",
        predicted_at_column="predicted_at",
    ),
    output_source=FileSource.parquet_at(
        "transaction_category_preds.parquet"
    )
)
class TransactionCategory:
    transaction_id = String().as_entity()
    predicted_at = EventTimestamp()
    
    predicted_category = (transaction.merchant_name
        .as_classification_label()
    )

Therefore, enabling me to just define who to predict for, and the feature loading, interacting with a model in memory or through an API will be done for you, metadata will be added automatically.

predictions = await store.model("transaction_category").predict_over({
    "transaction_id": [...]
}).to_polars()

Or if I want to store them in the output_source, I could do it with the following.

await store.model("transaction_category").predict_over({
    "transaction_id": [...]
}).upsert_into_output_source()

The model input

Until now have I skipped one crucial part. I have not talked about the input I used to the classification model.

Since I know the transaction text description would probably be the best feature, did I land on using an embedding model, and use the embedding vector as the inputs.

Thankfully, alinged have an easy integration to setup a complete model contract with Ollama integration.

TransactionEmbedding = ollama_embedding_contract(
    contract_name="transaction_embedding",
    text=transaction.text,
    endpoint="http://host.docker.internal:11434",
    model="mistral:latest",
    entities=transaction.transaction_id,
    output_source=FileSource.parquet_at(
        "transaction_embedding.parquet"
    )
)

From here the input can be sent into the original transaction model.

transaction_embedding = TransactionEmbedding()
@model_contract(
    name="transaction_category",
    input_features=[transaction_embedding.embedding],
    ...
)
class TransactionCategory:
    transaction_id = String().as_entity()
    predicted_at = EventTimestamp()
    
    predicted_category = (transaction.merchant_name
        .as_classification_label()
    )

This will make it possible for aligned to ensure that we never break the transaction category model, by making breaking changes to the embedding model.

Evaluate the model

Finally, I had a model to use, and I finally created an API and an iOS widget to send a request to predict over my new transactions each day. This ment I accumulated predictions all the time, and after some time I could finally evaluate it.

Thankfully, aligned knows where we store predictions, and where the ground truths are, and knows that it is a classification problem. Thereby making evaluation super easy.

Thereby, we can open the aligned UI and get confusion matrix, accuracy, precision, and recall metrics without doing anything.

Furthermore, we can add a model version variable to the model contract in order to compare different models against each other.

@model_contract(...)
class TransactionCategory:
    transaction_id = String().as_entity()
    predicted_at = EventTimestamp()
    model_version = String().as_model_version()
    
    predicted_category = (transaction.merchant_name
        .as_classification_label()
    )

Conclusion

Building an end-to-end ML project have made it clearer the wide range of problems that ML offers. Furthermore, aligned makes it possible to get more done with less work, by moving the focus away from model first. And focusing on how the ML data flow will behave, and then say how to integrate the model.

I missed such system-wide tooling, so I created aligned to make it easier to understand, develop, and evaluate AI and ML data products.

My Annoyances

Before we delve into my goals with aligned, it is important to understand what I felt was missing in the current AI stack.

Code Completion

One of the most surprising things about moving to the AI ecosystem was the lack of code completion and tooling to prevent bugs. This was especially clear when coming from a strictly typed language like Swift, where types need to be enforced, and the use of raw strings is seen as a code smell.

In contrast, the AI landscape was the clear opposite. Almost no types, and the types that were present were not enforced either. But also, almost all code was described using raw strings. For example, how pandas do transformations - df["new"] = df["a"] + df["b"].

Why is this raw string usage so bad? It makes it almost impossible for linters to find semantic errors, like mistyped attributes or invalid datatype operations. But it also makes it very hard to provide code completion, and help the developer understand what is possible to do. Therefore, reducing the dependence on package tutorials and documentation.

Implicit Schemas

Continuing on the usage of strings, I often found that Python programs heavily relied on implicit schemas.

One such example would be yaml config files, and how they often get accessed through dictionaries. Below is a fairly common training pipeline that I found in a Medium post.

config = load_config("my_config.yaml")

# load data
data = pd.read_csv(
    os.path.join(
        config["data_directory"], 
        config["data_name"]
    )
)

# drop id column
data = data.drop(config["drop_columns"], axis=1)

# Define X (independent variables) and y (target variable)
X = np.array(data.drop(config["target_name"], 1))
y = np.array(data[config["target_name"]])

# split data into train and test sets
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=config["test_size"], random_state=42
)

# call our classifier and fit it to our data
classifier = KNeighborsClassifier(
    n_neighbors=config["n_neighbors"],
    weights=config["weights"],
    n_jobs=config["n_jobs"],
)

If any of these keys are missing, or potentially in the incorrect data type, this pipeline can fail. For example, if an integer is loaded as a string. Rather, I want these implicit schemas to be defined explicitly, making it possible to fail faster and make it clearer which information is available. One example could be pydantic, and it shows how usefull explicit definition of schemas can be.

Implicit Logic

Another issue I found was that getting a Birds Eye view of our AI products was hard. We mostly documented our AI products manually, but it was often only showing the AI product, and not how it integrated into the system on a larger scale. Furthermore, the technical documentation was either missing or out of date.

Therefore, questions such as “what are the goals of the model,” “who owns the model,” “which data are needed to run the model,” “what are we predicting,” “where do we store predictions,” “where can I run the model,” “how often do we expect the model to run,” “where do we store training datasets,” and so much more were left unanswered.

Often I found that a lot of these questions were described implicitly in code, but outdated in the docs.

Default Implementation

Implicit logic and implicit schemas are somewhat similar. However, defining both of them can at first seem like more of a pain than anything. But defining the implicit schemas and logic explicitly can help derive default implementations. Such as data validation, setting up data freshness checks, row duplication checks, model evaluation, model performance monitoring, and so much more.

Therefore, explicit schemas and logic do not only help with failing faster, but it also helps with implementing reasonable functionality in less time.

Inflexible Data Sources

Lastly, I was surprised at how impractical it was to work with data sources in AI. We either created custom data wrappers, got locked into using one data warehouse, or we copied the data of interest to a local file.

However, we often wanted the possibility to combine our data warehouse with experimental sources, like local files or transactional databases, but this was either very convoluted or impossible to do.

The Goals

As a result, I wanted to improve all of this. Therefore, I wanted to create a tool that provides a good developer experience by offering code completion and catching errors earlier. I wanted you to explicitly define your expectations, so that we can fail faster but also go further with less work. I wanted

the technical documentation to be generated from code, rather than being dependent on us keeping it up to date. And of course, I wanted to make it possible to mix and match data sources, so you can run experiments in less time.

Aligned

This is how aligned was created. Now, if you are familiar with facts and dimensions, this should hopefully not be too unfamiliar.

At its core, aligned will introduce two concepts: feature_views - similar to a dimension, and model_contracts.

Model Contract

A model contract is crucial in aligned. This is where we define all the metadata for our model, such as intent, input features, and output.

To make this clearer, I will present a side project I created where I wanted to categorize my bank transactions.

@model_contract(
    name="transaction_category",
    features=[
        transaction.text_embedding
    ],
    prediction_source=FileSource.parquet_at("pred_data/transaction_category_test.parquet"),
    dataset_store=FileSource.json_at("datasets/transaction_category.json"),
    exposed_at_url="http://server:8000/openapi.json", # An internal docker compose URL
    acceptable_freshness=timedelta(days=2),
    unacceptable_freshness=timedelta(days=4),
)
class TransactionCategory:
    transaction_id = String().as_entity()
    
    predicted_at = EventTimestamp()
    
    model_version = String().as_model_version()

    predicted_category = (
        transaction.merchant_category
            .as_classification_label()
    )

So, what is happening here? First of all, we define a model that will take an embedding as the input (features) and that the model will produce a predicted_category together with some other metadata, like predicted_at the transaction_id it predicted for, and the model used in its model_version.

Furthermore, we also define where our predictions are stored (prediction_source), where the model is exposed (exposed_at_url), and how often we expect it to predict (acceptable_freshness, unacceptable_freshness), and where we store the train, test, validate sets (dataset_store).

However, we reference a transaction variable in both the input features and when defining what we predict, so what is this?

Feature View

This is where our feature views come into play. Very similar to a dimension, as we define different information associated with some kind of entity.

This is what our transaction variable is from the previous example.

@feature_view(
    name="transactions",
    source=api_source,
    materialized_source=FileSource.parquet_at(
        "source_data/transactions.parquet"
    ),
    description="The bank transaction made by a user",
    contacts=["MatsMoll"],
    acceptable_freshness=timedelta(days=3),
    unacceptable_freshness=timedelta(days=7),
)
class Transaction:
    transaction_id = String().as_entity()

    user_id = String()
    account_id = String()
    
    accounting_date = Timestamp()
    interest_date = EventTimestamp()

    amount = Float()
    abs_amount = abs(amount)

    text = String()
    transaction_type = String()

    card_details = Json().description(
        "Is only set where transaction_type = VISA VARE"
    ).is_optional()

    is_expense = amount < 0
    is_income = amount > 0
    
    merchant_category = card_details.field(
        "merchantCategoryDescription", String()
    ).is_optional()
    
    text_embedding = text.embedding(
        EmbeddingModel.huggingface("all-MiniLM-L6-v2")
    )

transaction = Transaction()

There are a lot of things happening here, so let’s go through it. First of all, we define all the information we expect a transaction to have, and its associated datatype.

Furthermore, we define where all this data will be fetched from source, but since this example will load from an API, we also write this data to a materialized_source working as a cache.

And similar to our model_contract, we define the expected freshness of this data. Making sure we are aware if our data is older than our expectations of 3 days.

Furthermore, by default, all columns will be required, which is why you see the card_details have a .is_optional(), as it can be missing in some scenarios.

Also, notice that we can add documentation directly on a feature with .description(...), in case there are some extra contexts needed to justify the setup.

But not only that, we can even provide transformations in the feature views (is_expense = amount < 0). And this without any strings. Therefore, providing both code completion, and linters can catch errors for you.

Data Lineage

All of this is nice to describe, but it gets better. You see, since we have all of

these transformations and references, aligned will automatically collect data lineage through features, and even more interestingly, through models.

Meaning, if we had created a downstream model that depends on the predicted_category, aligned would know and show it to you.

Therefore, making it easier to provide a valid overview of your ML products.

But it also makes it possible to prune unneeded transformations and debug transformations.

Just look at how we can ask for the question_embedding, and it knows that you need to provide a question and a description.

Conclusion

The AI landscape has evolved a lot in the past years, and we have a lot of good tooling to simplify the development of AI products. However, managing ML products on a system scale still needs a lot of implicit knowledge. Something that can also lead to extra work, longer onboarding times, and confusion.

As a result, I developed aligned, which tries to manage the data from AI products. Therefore, somewhat becoming the DBT of AI. As a result, aligned simplifies the development, management, and understanding of how our AI products actually behave. I have only touched the surface of what aligned can do, but hopefully it shows some of the potential.

So if this is interesting, try out aligned and let me know what can be done to improve it further.

While this evaluation process is quite valuable and usually helps determine if the model is good enough, it’s not without its drawbacks.

The primary issue here is that the evaluation might not provide accurate performance metrics. Moreover, there’s a wide array of data errors that could come into play, for instance, leaking information either through features or over time. Consequently, if such data errors occur, they could quickly lead to inaccurate performance metrics, making the evaluation reports misleading or invalid. As a result, the most precise way to gauge a model’s performance is through monitoring it in production.

However, evaluating the model performance in production can often be down-prioritized or an afterthought. Potentially ending up in a situation where we never add evaluation in production.

Yet, the concept of evaluating models in production is, in essence, straightforward.

What is needed?

There are two things needed to measure performance in production.

1. The predictions
2. The ground truths

We will already have the ground truth stored somewhere for most classification and regression models, as it is likely that we have trained a model ourselves using some version of a decision tree or regression model.

Furthermore, we hopefully have the predictions stored somewhere as we will need to serve them to the end user. However, the predictions and ground truths can be stored in different locations or data sources if using a microservice architecture, making it a bit more tricky.

One solution will be to ingest the predictions in a data warehouse or data lake. However, we must still set up data loaders and create processes that combine the data.

Another solution would be to use something like gantry.io that generates evaluation reports for you. However, this means you must send your data to a third-party service, which is not always possible or acceptable.

Therefore, I wanted to create a solution that enables easy evaluation of AI models in production using the business’s existing infrastructure.

Taxi ETA model

Let’s look at an example of a Taxi ETA model and how we could evaluate performance in production.

Imagine we have a PostgreSQL table that contains a list of trips taken, where they started, where they ended, and the duration of the trip. Therefore, we can use the duration as the ground truth.

Furthermore, we store the predictions in another table containing the predicted duration, a timestamp of when it was predicted, and the model used. Therefore, we now have the location of our predictions as well.

But how should we model this?

I will use aligned for this, so let’s start modeling the ground truth.

from aligned import PostgreSQLConfig, UUID, Int32

db = PostgreSQLConfig(env_var="PSQL_URL")

@feature_view(
    name="trip",
    batch_source=db.table("trips")
)
class Trip:
    trip_id = UUID().as_entity()

    duration = Int32()
	
    number_of_passengers = Int32()

The code above defines the schema we have stored in the Postgres database named trips. At the same time, we have specified that we will select the features using trip_id as the “identifier” or entity.

However, we have not yet defined our duration column as the ground truth. Mapping the ground truth is done in our model definition.

from aligned import model_contract, Float, UUID, EventTimestamp
from trips import Trip, db

trip = Trip()

@model_contract(
    name="trip_duration",
    features=[trip.number_of_passengers, ...],
    predictions_source=db.table("trip_predictions")
)
class PredictedTrips:
    trip_id = UUID().as_entity()
	
    predicted_duration = trip.duration.as_regression_label()

Again, the above code defines a model contract where we list the features that we use as input to the model, the table used to store the predictions in, that we will query the predictions using the trip_id, and that the trip.duration column is the label that we predict.

Evaluate the model

The above code has everything that we need. We have defined where we store our ground truths, predictions and that the duration column is the target for our regression model. However, now we need to evaluate the predictions.

from sklearn.metrics import mean_squared_error
from aligned import FeatureStore, PostgreSQLConfig

store = FeatureStore.from_dir(".")

db = PostgreSQLConfig("PSQL_URL")

entities = db.fetch("""
    SELECT trip_id
    FROM finished_trips_view
""")

eval = await store.model("trip_duration")\
    .with_labels()\
    .predictions_for(entities)\
    .to_pandas()

mse = mean_square_error(eval.predictions, eval.ground_truths)

In the above example, will we select the instances where a ground truth exists, join the predictions for them, and the ground truths themselves.

Our response from the database contains all of the data in one data frame but nicely separates the different information for us. They make it easy to compute metrics using the attributes of ground_truths and predictions.

Conclusion

Evaluation is an essential part of AI applications. However, evaluating how well our models perform in production can often be down-prioritised, even though this is the most accurate metric for our model, as data leakage is way less likely in such an environment.

Furthermore, using the aligned package makes it easy to define the relationship between our ground truths and predictions. While also defining where we to fetch the different information. Such a setup makes it easier to load the data needed to evaluate our models living in production.

Therefore, using more tried-and-tested principles can benefit this debate. However, I am not proposing to use principles like SOLID, DRY, or Clean Code; I’m instead proposing to use existing UX principles and apply them across different programming languages, frameworks, and paradigms.

But UX is for UI design, I hear you say.

And I would answer - no, it is not; it is much more flexible, so let me show you.

Example 1: Pydantic vs dataclasses

Let’s evaluate two codebases that fulfill the same need but use different implementations. Furthermore, we will determine why one solution is better using UX principles rather than the unclear “feel better.”

The following code makes it possible to encode and load a JSON object. Our first implementation will use dataclasses, and the second will use pydantic. Now, dataclasses’s use-case is not to make JSON easier to work with. However, dataclasses can implement our “happy path,” so it could have been used for such a use-case.

So, let’s look at the dataclasses implementation.

from dataclasses import data class, asdict
import json

@dataclass
class Size:
    width: int
    height: int
    
size = Size(width=10, height=20)
size_json = json.dumps(asdict(size))
reconstructed = Size(**json.loads(size_json))

This is pretty straightforward. Declare a data class and its properties, create an instance of the object, convert it to a dict, and then to a JSON string. Lastly, decode the JSON string back to a dictionary and pass the fields to the constructor.

So now is the pydantic solution.

from pydantic import BaseModel
import json

class Size(BaseModel):
    width: int
    height: int

size = Size(width=10, height=20)
size_json = size.model_dump_json()
reconstructed = Size(**json.loads(size_json))

So, both solutions are very similar, and if I had gone with my “feeling,” I would have implemented the dataclasses solution.

Why? Because I do not like that we need to subclass from BaseModel in pydantic. Furthermore, pydantic is an external dependency, while dataclasses is already included in Python.

However, would this be the correct choice, and if so, why?

Evaluate using UX

To evaluate our use case, I will mainly use Nilsens Heuristics. These are very generic, but it is also what makes them so flexible. Furthermore, it is still better than “it feels good,” so I think it is a step in the right direction.

However, using other UX principles like the Proximity principle - related information should be close to each other, which could be used to argue against C header files. Other concepts, e.g., from “the design of everyday things” would also work.

Error prevention

Good error messages are important, but the best designs carefully prevent problems from occurring in the first place.

NN Group

While the dataclasses and pydantic solutions appear similar, their approaches to error handling are notably different.

Our dataclasses solution allows the following invalid input.

size = Size(width="10m", height="20cm")

Here, we send in a string while we expect an integer to be passed, and we can use this and send it into functions using the Size object. However, pydantic will validate and catch such a case. Therefore, throwing an error if we do not match the expected data pattern prevents downstream problems.

Other examples that improve DX by preventing errors could be using something like Protocol Buffers rather than JSON for a similar reason.

Visibility of system status

The design should always keep users informed about what is going on through appropriate feedback within a reasonable amount of time.

NN Group

It can be harder to provide feedback in code as it is a static artifact. However, we can leverage tools like type systems, compilers, and linters, to fulfill this.

Furthermore, to continue our above example, let’s add a new method.

def area(size: Size) -> int:
    return size.width * size.height

Since pydantic throws when the data pattern is invalid, will pydantic, as a result, ensure our system state. Therefore, only pydantic will make our system state more visible, as we know that our Size object contains the expected state that we want.

Help users recognize, diagnose, and recover from errors

Error messages should be expressed in plain language (no error codes), precisely indicate the problem, and constructively suggest a solution.

NN Group

This is where the original UX heuristic may differ from “my” DX one. We sometimes need the technical jargon and details to fix the problem. However, the heuristic is still valuable, as precise error messages with guidance are super valuable.

Again, our dataclasses solution can fail if we call area(size), which can lead to the following error message:

TypeError: can't multiply sequence by non-int of type "str"

This tells us precisely what the problem is. However, we are told we can’t do what we want, which is multiplying. Furthermore, the compounding effect of the mismatching system state makes it harder to understand why we can’t multiply. The reason for this error is because of invalid input.

However, our pydantic solution provides an error when trying to create an object. But this leads to the following error.

ValidationError: 2 validation errors for Size
width
  Input should be a valid integer, unable to parse string as an integer [type=int_parsing, input_value='10 m', input_type=str]
    For further information visit ...
height
  Input should be a valid integer, unable to parse string as an integer [type=int_parsing, input_value='20 cm', input_type=str]
    For further information visit ...

This message mentions where the error is located, what is wrong, how to fix it, and a link for more information. It is more apparent that we have inputted invalid data, and most likely not due to poor code.

This guidance also makes Rust’s compiler helpful, as it helps you locate and guide you in the correct solution when something is wrong.

With the presented UX principles in mind, my decision to go with what felt best would have been a poor decision, potentially leading to more frustration in the long run.

However, this is one case; what about something that differs more?

Example 2: Aligned vs Pandas

Let’s look at another solution with data processing using an unconventional approach while presenting new heuristics.

So, let’s consider a scenario where we load geo data from a trip database, compute some features, and then use an ML model to predict the duration of the trip. We will use pandas and aligned to do the same thing. So, let’s look at the pandas code.

con = create_engine(os.environ["TAXI_DB_URL"])
df = pd.read_sql_query(
    sql="""
SELECT 
    id as trip_id, 
    
    pickup_latitude, 
    pickup_longitude,
    
    dropoff_latitude,  
    dropoff_longitude, 
    
    picked_up_at 
FROM trips;
""", 
    con=con
)

df["lat_diff"] = (df["pickup_latitude"] - df["dropoff_latitude"]) ** 2
df["long_diff"] = (df["pickup_longitude"] - df["dropoff_longitude"]) ** 2
df["travel_distance"] = (df["lat_diff"] + df["long_diff"]) ** 0.5

# The day in the month
df["day_of_week"] = df["picked_up_at"].dt.day

input = df[["day_of_week", "travel_distance"]]
duration = model.predict(input)

This is a common codebase for data applications. However, this solution has similar problems to our dataclasses solution earlier. We cannot validate our SQL code because it is written in a pure string. We do not make our code check against bad data, as we do not have data validation. This is intentionally not added, as presenting too much code in a blog post is uninteresting. Furthermore, experimental code can quickly skip data validation, as extra dependencies are needed, leading to odd errors. Therefore, making our df["picked_up_at"].dt.day potentially error-prone, as we forcefully cast our value to a date time type without converting the picked_up_at to a proper value.

Furthermore, we can not use our Python Language Server Protocol (LSP) to catch naming errors in our data frame, as we use strings to reference values. Leading to the potential error KeyError: 'day_of_weak' on runtime rather than an error on compile time. Similar to how we can not type check for values in dictionaries, but we can if described as a class object.

Therefore, we have little to no system status visibility, leading to less error prevention, and we do not help our users recognize, diagnose, and recover from errors.

A solution like aligned can significantly enhance the UX heuristics we’ve discussed. A quick introduction to aligned. Aligned is an ML tool used to make data more consistent at a system level. Therefore making it possible to standardize transformations in a technology-agnostic format. Making it possible to do ETL, even though it is not its primary purpose.

So, how do aligned compare?

taxi_db = PostgreSQLConfig(env_var="TAXI_DB_URL")

@feature_view(
    name= "trips,"
    description= "Features related to the departure of a taxi ride,"
    batch_source=taxi_db.table("trips", mapping_keys={"id": "trip_id"})
)
class Trips:
    trip_id = UUID().as_entity()

    picked_up_at = EventTimestamp()

    dropoff_latitude = Float().is_required()
    dropoff_longitude = Float().is_required()

    pickup_latitude = Float().is_required()
    pickup_longitude = Float().is_required()
    
    lat_diff = (pickup_latitude - dropoff_latitude) ** 2
    long_diff = (pickup_longitude - dropoff_longitude) ** 2
    travel_distance = (lat_diff + long_diff) ** 0.5
        
    day_of_week = picked_up_at.day.description("The day in the month")
    
    
input = await (Trips.query()
    .select(["travel_distance", "day_of_week"])
    .all()
    .to_pandas()
)
preds = model.predict(input)

Quickly, aligned makes it possible to declare data schemas and transformations using a class, enabling us to leverage the LSP and find errors earlier. Making it more straightforward which state our data is expected to be in and reducing errors by validating the anticipated state. Very similar to pydantic.

But let’s compare using some new UX principles.

Aesthetic and minimalist design

Interfaces should not contain information that is irrelevant or rarely needed. Every extra unit of information in an interface competes with the relevant units of information and diminishes their relative visibility.

NN Group

The pandas solution contains fewer characters, which makes it more minimal in that sense, as we type less. However, the transformations in our pandas solution keep repeating df["..."], which is irrelevant information. Therefore cluttering our logic and creating more noise. However, this is where aligned makes it way less noisy. Just look at how the two different lines differ below.

# Pandas
df["lat_diff"] = (df["pickup_latitude"] - df["dropoff_latitude"]) ** 2

# Aligned
lat_diff = (pickup_latitude - dropoff_latitude) ** 2

Both do the same thing, but aligned makes it possible to remove all the df references while also removing the usage of dict-like data access. Therefore, our business logic is described with less noise aligned, and I would take less noise over fewer characters.

Recognition rather than recall

Minimize the user’s memory load by making elements, actions, and options visible. The user should not have to remember information from one part of the interface to another.

NN Group

Furthermore, our pandas solution relies heavily on strings or a dict like access method. This has the unfortunate effect of hiding valuable information for our LSP. This results in less accurate code completions, leading to more cognitive load on the user, as a result, and as mentioned earlier. Our pandas code needs the user to remember that the picked_up_at is a date time and cast it to such a type df["picked_up_at"].dt.day.

However, aligned uses variables combined with defined types without casting. Therefore, helping our LSP provide valuable information about the types that each method returns and the potential paths our program can take. Enabling our users to search through our auto-completion and recognize possible programming paths. Therefore, recalling less than our pandas solution.

Again, this is why using the proximity principle can also help with recognizing more than recalling it.

Consistency and standards

Users should not have to wonder whether different words, situations, or actions mean the same thing.

NN Group

Following conventions and common standards is always good, making it easier to transition from different domains. One such standard could be the usage of SQL, using the standard operators as +, -, * etc., or using common names on data type as Float, Int64, and maybe not ‘long long int.`

However, this is where pandas and aligned provide consistency and standards at different abstraction levels. As pandas provides data transformation consistency across different machine architectures, while aligned provides consistency at an application service level, transformations, and data dependencies can be shared in a serialized format.

Therefore, they both provide consistency, depending on the intended use case. As a result, DX is not as easy as “this is better,” but it is more a question about “it is better for this use-case.”

For, If we want to run one script that does data analytics occasionally, then pandas provides the consistency we want, as we can be confident the transformations will run on multiple types of machine hardware. However, if we need data transformations to be shared across employees or applications, then pandas may not provide good enough consistency, but aligned will.

Furthermore, this is where we can argue for DRY, as the whole point is to make our codebase more consistent when changes get implemented. However, will repeating the same code in two places be bad for consistency? Probably not, so maybe following the “Rule of three” rather than a pure DRY approach would be better.

Flexibility and efficiency of use

Shortcuts — hidden from novice users — may speed up the interaction for the expert user so that the design can cater to both inexperienced and experienced users.

NN Group

Furthermore, both solutions enable our users to speed up their work for common use cases. The read_sql_query makes connecting to an SQL db easier in pandas, rather than reading them manually and converting them into the desired format. However, pandas is intended to be used as a generic data processing framework and not for ML use cases.

Therefore, fulfilling needs specific to ML can be clunky to do with pandas, such as creating train, test, validation data sets, or setting upstream processing for low latency features.

As a result, aligned can streamline such use cases and make them easier, enabling one to opt into more flexible solutions when needed. That’s why you see the .to_pandas() while also allowing you to set up a stream processing worker with the following code.

from aligned.worker import StreamWorker
from aligned import RedisConfig, FileSource

definitions = FileSource.json_at("data-defintions/v1.json")

worker = StreamWorker.from_reference(
    definitions,
    sink_source=RedisConfig.localhost(),
).expose_metrics_at(port=8000)

Therefore, finding tools specializing in your needs will make you move faster. However, making it possible to opt into the underlying details will be helpful to make our tools flexible enough.

Furthermore, this is why I like the fluent ORM created by the Vapor team. As they provide: A low-level raw SQL client. A SQL query builder based on existing data classes. A high-level ORM that fulfills common use cases. It provides shortcuts while also enabling flexibility when needed.

Latency

Lastly, even though this is not a heuristic, I would like to add performance as a criterion for DX, or how I would instead frame it, latency.

Furthermore, improved performance can affect cost. However, cost is more of a business side-effect, and I will not consider it affecting DX. I am only interested in how performance affects latency, as poor latency will degrade the developer experience, while good performance will be barely noticeable.

This happened to me at work, where we used pandas for most of our ETL. However, at some point, pandas lead to way too long run times. Therefore, switching to polar reduced the run time to 1 / 10, making the latency acceptable while avoiding complex spark clusters.

This is why aligned makes it possible to change the processing engine when needed. Therefore avoiding premature optimization, reducing costs, and making the solution performant.

In other words, high latency can ruin the developer experience, but only some will notice a more performant solution than an already good enough solution.

Delving Deeper into Heuristics

While I’ve touched on several heuristics in this piece, there’s a wealth more out there—like “Help and documentation”, “User control and freedom”, and “Match between system and the real world”. I’d love to dive deeper, but we’re already wading through quite a bit here. Perhaps in a future post!

Conclusion

Developer Experience (DX) has been somewhat diluted to represent tools developers find comfortable. Therefore, it often means “what I am used to.” But to drive valuable discussions and innovation, there’s a pressing need to view DX through a more objective and comprehensive lens. Incorporating UX principles to evaluate code can pave the way for a more inclusive and critical approach, helping developers describe their code in new ways and make more educated choices for why technologies are good or bad. Even when they look foreign and odd at first glance.

If you found the read interesting, please share or give Aligned a star. Thanks for reading.

However, the core concept of an ML model is simple. For some input, produce an output. Or, as math would describe it, \(f(X) \rightarrow y\). So how could this simple function lead to the chaotic landscape that exists?

We will look at some common components for AI models deployed behind an API to keep this post reasonably sized.

Categories of ML products

It helped to categorize our ML products into three high-level groups. These categories make understanding the core problem statement and the solutions’ constraints easier.

The model - \(f(X)\)

Let’s start with the most popular category - the model. Solving a problem from this category can be seen as a model-centric approach. Therefore, making it one of the most popular approaches as well. As we need a model to have an ML product.

Model creation

Before we can do anything, we need a model to use. Thankfully, tools such as sklearn, TensorFlow, PyTorch, etc., have made it possible to train a model with a few lines of code - model.fit(X, y).

This problem statement is what we see when being introduced to ML for the first time, and rightfully so.

However, this is only the start of even more questions. One natural question would be - How can we release a model, making it accessible to others?

Inference server

Meet one approach, an inference server. This approach often requires a separate server that handles model inference. Therefore, making it possible to access additional hardware like GPUs or TPUs if needed. As a result, an inference server is a standard solution.

However, typing up a new codebase only to make one model accessible can sound like a lot of work. Since an inference server is a common requirement, many solid solutions have popped up. For example, Huggingface offers a way to deploy transformer models within a few minutes, while OpenAI offers pre-trained models through an API. Then you have solutions like KServe, which makes it reasonably easy to deploy custom AI models on a Kubernetes cluster

But with our model deployed comes a new problem - What if I want to update my model to a new version?

Model Registry

Here is where the model registry comes in handy. In the same way we version code, will there be a need to version models at some point. Either because we found a new feature that could help or because the old pattern does not apply anymore - known as concept drift.

Therefore, we need to have some way of managing \(f_0(X), f_1(X), ..., f_n(X)\), and so on.

One simple way could be to store our models in a storage service like AWS S3. Such a solution can work if we have a few artifacts to handle. However, the complexity can quickly increase as we want the model to do more. For example, we could add pre-processing and post-processing, or we want to manage the runtime environment of the model. Using a more specialized model registry component for such a requirement would make sense. One such tool would be MLFlow.

But again, as we start to train new models regularly, new problems arise. For example, how will we know the differences between all our models?

Model Card

Here is where model cards come in. It summarizes what the model will do, how it was created, its limitations, some performance statistics, and more. Therefore, a model card summarizes \(f_n(X)\).

One tool that collects this information is model_card_toolkit. However, we will need to dive deeper into our model. So could we see how our model came to be?

Experiment tracking

Experiment tracking is one solution to such a problem. Providing a log of metrics as our model is trained, in addition, can all kinds of evaluation graphs be logged - as confusion matrices, and log which features contribute the most to our output. Therefore, answering how we arrived at \(f_n(X)\).

Some tools that answer this question are Weights & Biases, ClearML, and MLFlow.

Final thoughts on \(f(x)\)

The model category starts tackling problems from the model’s point of view. Answering a lot of the questions, such as how \(f(X)\) got created, when to use \(f(X)\), how to version \(f(X)\), and how we can expose \(f(X)\) to others.

However, we still have some huge questions to answer before we have a fully working ML system. Because, what is \(X\), and can \(X\) change how well our model performs? Such questions lead us to the next category.

Input data - \(X\)

Introducing our second category - input data.

Someone jokingly said that ML practitioners use 99% of their time debugging and 1% writing code. However, there is some truth to this. It can be hard to debug ML products, and silent bugs can be one reason.

An example of a silent bug

To bring home this point, let’s look at a simple example. Let’s say we have a ride-sharing company that wants to predict the duration of a ride. So the ML researcher finds some data in a database containing the ride’s distance. Therefore, thinking this can be a good feature. And the performance metrics were outstanding when evaluating using the train, test, and validation set. Thus, you are convinced to release a new model version. However, we notice that our model performs way worse in production. How could this be?

We start debugging our data and notice that we estimate the distance when the ride starts but update our database with the actual duration when the ride has ended. Therefore, we have trained on the actual distance, but we predict an estimate. Thus, our model performance is worse in production. Such a bug is also known as data leakage.

Overwriting an estimate with the actual value may seem like a poor choice. Still, overwriting old data - known as slowly changing dimension 1 is common practice because it simplifies the queries in an application database. However, this makes making mistakes in ML products easier if we mutate data, as we may train a model on data from the future.

Data warehouse

One solution and a common component in business analytics is a data warehouse. A data warehouse is a single truth source for all analytical workloads. Making it optimal to compute aggregates on millions of records, and ML products can leverage this component.

Furthermore, since some data warehouses store a historical record of our data - potentially using slowly changing dimension 2, will it be possible to travel back in time. Therefore, it is possible to ask the question - what is \(X_{t}\) when \(t =\) the prediction time?

As an ML practitioners, this means we can create higher-quality datasets. Let’s use the ride-sharing example again. We may store our estimated distance value and the overwritten actual distance in a data warehouse but with a different timestamp. By correctly crafting our queries, we can select the values available at prediction time, training on the estimated value, which is the same data we predict on. Therefore, the new training set increases the performance of our model in production, leading to a more significant value gain for our end-users.

However, crafting such queries can be complex, and it is easy to make mistakes. Therefore, an ML researcher would like to think about something other than this - so what can we do about this?

Feature store

Here is where our feature store starts providing value.

A feature store acts as a highly specialized database for machine learning applications. One such use case will be how to generate data sets that would be valid at the time of prediction. Therefore, fulfilling the same needs as described as a data warehouse. However, the feature store abstracts away the complex queries and logic needed to provide such data. Also known as a Point-in-time correct join.

Therefore, creating a dataset where each row has the features that were available at their prediction time \(D = \{X_{0,0}, X_{1,0}, ..., X_{t,e} \}\) where \(t\) is the prediction time, and \(e\) is the entity to predict for.

Furthermore, we often want to engineer our features a bit, either by generating embedding features or something simpler as computing a ratio between two columns.

We could add some pre-processing into an artifact and store them with our model in a model register. However, some features require more computing. For example, features like mean distance over 20 minutes will be impossible in a serverless pre-processing method. But at the same time, we need to ensure that we use the same features in our training run and our online inference server. Otherwise, our prediction could be meaningless. This problem is also known as the training-serving skew.

Again, our second reason for a feature store. Ensure training and prediction features are the same \(X_{train} = X_{predict}\).

Lastly, there is one more reason to set up a feature store. We have covered how a feature store leverages a data warehouse to generate higher-quality datasets. However, loading inference features from a data warehouse often lead to long waiting times. This latency is because a data warehouse is optimized to analyze large amounts. Therefore, a data warehouse optimizes its hardware and disk usage for such use cases. However, this often leads to slower reads when we ask for individual rows of data.

As a result, a feature store fixes this limitation by changing the data storage based on the query type. Therefore, using a data warehouse for generating large datasets, but then using a key-value store for low latency inference data.

Some alternatives for feature stores are Feast, Tecton, Hopsworks, or my own solution Aligned.

Final thoughts on \(X\)

Ensuring data quality can be hard, as logical errors can easily creep in. Thankfully, a feature store can be a very useful component. Therefore, leveraging well-established data engineering practices abstracts away a lot of the complexity needed for ML products.

But still, there is one category left.

Output data - \(y\)

Lastly, we have the output data category. The output category could be in a joint data category with the input data. However, there are a few reasons for separating it into a separate category.

First of all, the leading players in the input data category do not provide a structured system to handle the output of our models.

Secondly, the output is the most critical artifact of them all, as this is what provides value for our end-users. All other components only exist to facilitate the creation of our \(y\) value.

However, more tooling is needed to make it easy to validate our online predictions.

Monitoring

From my experience have, the most popular solution been to set up a Prometheus and Grafana server for such use cases.

The Prometheus stack can work for simple needs. In the same way, S3 works well as a model registry when the artifact complexity is low.

However, ML-specific use cases can be hard to fulfill with such a stack. One such use case could be to view performance metrics based on sub-groups. This is very doable if we know the sub-group in advance, but it can lead to a low feedback loop if we want to check new sub-groups.

Evaluation store

Here we have the evaluation store component. A component that specializes in monitoring online performance for ML products. However, this can sometimes be down prioritised to setup, because we may think training evaluation is good enough. But this will not always be the case, because of data drift, and silent data bugs. Therefore, using an evaluation store uses ground truth to evaluate our online predictions \(y = \hat y\).

An evaluation store also enables viewing performance in sub-groups of interest, leading to a better understanding of our model’s limitations.

One tool here is Gantry, but other players are also starting to pop up.

However, we have still on question left - why did the model return such an output?

Explanations

Finally, the last component we will cover, explainers. Knowing why a model return an output can be very important for businesses. So important that businesses sometimes will release a less performant model that is explainable, rather than a more performant blackbox model. Therefore, an explainer answers why \(\hat y\).

Here can a lot of approaches be used. Such as analysing the inner workings of a model - checking a decision-tree’s splits, but this will not work for complex models as neural nets. Such models are black boxes, resulting in a need for different tooling. Thankfully, examples of useful tooling can be alibi explain, SHAP, and LIME.

Final thoughts on \(y\)

The road to a fully functional ML product is long and complicated. However, online prediction evaluation can be an afterthought for ML products, even though online predictions are their most important artifact. Therefore, potentially leading to solutions with more boilerplate code.

However, solutions like an evaluation store can help with this. Furthermore, tooling that explains our output can be crucial to provide the confidence and trust our ML products needs.

Wrap up

The amount of existing MLOps solutions is enormous, and this post only touches the tip of the iceberg. Concepts like drift detection and improving label data with active learning, to name a few. However, we can broadly categorize MLOps solutions into three groups. Managing the input \(X\), the model \(f(X)\), or the output \(y\).

Furthermore, managing our models has become extremely easy, as noted by Chip Huyen

…, with models being increasingly commoditized, model development is often the easier part.

However, managing our input data takes a lot of work, as it is easy to create faulty datasets. Furthermore, finding these data faults is hard. Thankfully, we have designed tools such as a feature store to reduce the number of potential flaws.

Lastly, managing our output data can quickly become an afterthought. Especially for companies that need more experience with ML products. As we quickly focus on the tooling making ML possible in the first place. Furthermore, the need for tooling to manage our model outputs can point to MLOps being immature.

However, many exciting MLOps solutions are trying to evolve the field further. One such tool is Aligned, removing the separation between \(X\) and \(y\), and unifying it more as a data category. So follow me and the Aligned repo for some exciting upcoming announcements!