Glenn's Notepad

Navigating our internal development platform

Amedia was my first job focused on web development. Coming from a research and robotics background, HTTP, REST, and even databases were new to me. This made standing up API services both a challenge and a learning opportunity. At Amedia, we can move fast and spin up backends quickly to solve narrow problems. This also means that cohesion and uniformity have not always been the main priorities. The GitHub organization has close to a thousand repositories in than more than 13 languages(!).

As the sole developer on our data platform team with the time and motivation for developing API services (at least at the time), moving from task to deployment was a big learning opportunity. Especially since most of the concepts were new to me. When I started this work, much of the knowledge needed to deploy a service was distributed among the many teams, and a central guide or a "golden path" did not exist. I worked closely with the IDP team to debug unclear parts of the deployment pipeline and smooth over internal friction. Their responsiveness and willingness to collaborate were instrumental.

Flexible data modeling with EAV

A key design choice in Inquire is an entity-attribute-value data model. Data is stored in a table with columns for user ID, attribute ID which references an attribute catalog, metadata describing when data was updated and when it becomes stale, and the data value itself, stored as JSON. The attribute catalog contains a human-readable name and expected type information. This lets us avoid constant schema migrations and unblock teams who wanted to test new attributes quickly.

Consent integration

In addition to the main data table and the attribute catalog, we have a table describing data usage purposes, where we can map user consent to attributes. A user opts in to or out of separate purposes, so the same attribute for a given user may be available under the editorial purpose but will be null under the marketing purpose, if they opt out of personalized marketing.

Some attributes are only allowed to be used under certain purposes. For example, we allow the sports_affinity under the editorial purpose (given the user opts in to personalization for that purpose), while making in unavailable under the ads purpose. This simplifies downstream logic for data consumers who previously might have had to resolve this out of band.

Composed attributes

Some attributes are derived from multiple data sources. For instance, a user changing their newspaper subscription plan is a rare occurrence, therefore, the subscription_plan_batch attribute is based on a table in our data lake, which is updated and ingested nightly. However, we also receive events when this happens, which creates or updates the subscription_plan_stream attribute.

To streamline how the consumers use this data, they only see virtual user attributes, which are functions of the physical attributes we have talked about up until now. A virtual attribute (e.g., subscription_plan) has an ordered list of physical attributes (e.g., [subscription_plan_stream, subscription_plan_batch]). A virtual attribute's value resolves to one of the physical values depending on the chosen selection strategy for that virtual attribute:

Coalesce:

The first non-null value in the ordered list of physical attributes.

Newest:

The newest non-null value among the list of physical attributes. Order is ignored.

Special:

Value selection or composition is hard-coded in the application, i.e., not configurable at runtime.

Virtual attributes lets us present a clean and stable interface to consumers, even when underlying data sources change, vary in freshness, or in reliability.

Easy-to-use management dashboard

Our flexible data model gives us the power to add attributes, modify their composition or type, associate them with purposes and consent data. To make this easy, we have developed a management dashboard that streamlines common operations and gives us an overview of attributes and data freshness.

The management dashboard is a server-side rendered website with htmx for dynamic content.

The dashboard has list pages for the physical and virtual attribute catalogs, purposes, and ingest jobs. Each attribute entry has a detail page and an edit page. Each purpose can also be edited, and each job has a detail page. It's also possible to add new attributes via the dashboard. Every developer in our organization can view all data, but only data platform developers can POST to the edit and new endpoints.

The management dashboard allows us to swiftly carry out necessary modifications and lets us get fast overviews of the data in our service.

Type checking of attribute values

The EAV data model stores attribute values as JSON in the database. This is great for flexibility, but sacrifices type safety at rest. To simplify the batch ingestor pipeline, we do not have any type checking of the data imported from our data lake. (The data in our data lake is, however, typed.) Instead, we type check the JSON data on every query from consumers, and log fatal errors if a discrepancy from the expected type (as stored in the attribute catalog) should be found.

To date, after nearly one year of operation with approximately 250 requests per second on average over 30 attributes, we have not had a single instance of wrong typing. This has given us confidence in that the tradeoff for flexibility against rigidness has been worthwhile.

High-resolution metrics

We have metrics recording the use of each virtual attribute per consumer per purpose. Additionally, we record whether a request for an attribute returned a null value or actual data. This allows us to answer questions like these:

• Which attributes are currently in use by any consumer?
• Which consumers use a certain attribute?
• For what purpose does a certain consumer use data?
• What attribute does a certain consumer use?
• An attribute's hit rate suddenly dropped to 0%, why?

These metrics have been invaluable. We can now confidently deprecate attributes, catch stale data early, and prove compliance when needed.

Performance considerations

The EAV data model places multiple rows per user in our data table. This bars us from using a single ID (e.g., user ID) as lookup value when fetching data for a user. Instead, we have to fetch rows corresponding for all relevant attributes for that user. We also make the observation that multiple requests for the same user (but for different attributes) is likely to occur within a short span of time, since multiple backends querying Inquire are involved for every page load.

To make the EAV model viable at scale, I indexed user ID to speed up multi-row lookups.

To avoid hitting the database for each request for the same user in the same page load, we added an in-memory cache to the API application. When any user data is requested, we fetch all attribute values for that user and store them in the cache for a couple of seconds. Subsequent requests for that user use in-memory attribute values. We have approximately 30% cache hit rate. This is quite low, but is explained by us having multiple replicas, so subsequent requests might hit a different process. Each cache hit saves a database round-trip.