Stop the Sprawl: Centralize Your Pipeline Secrets with Tenzir and AWS

In any sophisticated SecOps environment, data pipelines are the critical arteries, moving telemetry from source to destination. But as these pipelines multiply, they often leave a trail of hardcoded credentials. This "credential sprawl" is more than just messy—it's a ticking time bomb for security.

Managing secrets is a foundational requirement for any serious data pipeline deployment. That's why we are excited to announce a new, robust approach to credential management. This isn't just an add-on; it's a core component for building secure, manageable, and scalable data pipelines, featuring a direct integration with AWS Secrets Manager.

Using Secrets in Tenzir: The secret() Function

We've made using secrets in your pipelines incredibly simple and explicit. The key is the secret() function in the Tenzir Query Language (TQL).

Let's say you have a pipeline that forwards data to a Splunk instance using its HTTP Event Collector (HEC). Instead of pasting the HEC token into your pipeline, you now pass it to the to_splunk operator by calling secret():

// A pipeline that sends filtered alerts to Splunk
subscribe "detection-tool"
where alert.signature == "Potential DGA"
to_splunk "https://splunk.example.com:8088", hec_token=secret("splunk-hec-token")

When this pipeline runs, the to_splunk operator retrieves its hec_token argument by resolving the secret named splunk-hec-token. The secret's value is never exposed in the pipeline definition itself.

The Secret Resolution Flow: A Clear Hierarchy

Tenzir resolves the value for secret("splunk-hec-token") by checking three locations in a specific order. This gives you a powerful combination of flexibility for development and central control for production.

1. Local Node Configuration

First, the node checks its own local configuration, either tenzir.yaml or environment variables for containers. This is the perfect place for node-specific credentials or for temporarily overriding a central secret during development and testing, without affecting other deployments.

2. The Platform's Built-in Store

If the secret is not found locally, the node automatically queries its connected platform. The platform comes with a secure, built-in secret store that acts as the central source of truth for your credentials. If you're not using an external secret store, this is the recommended place for most secrets.

You can easily manage secrets in the platform's store via the command line:

# Add a value to the Platform's secret store.
tenzir-platform secret add geheim --value

Alternatively, use the platform user interface to configure your secrets:

3. External Secret Stores: AWS Secrets Manager

Alternatively, if you configured an external secret store, the platform attempts to resolve secrets from there. We are launching this capability with first-class support for AWS Secrets Manager.

This federated architecture is uniquely powerful. A Tenzir Node, running deep inside your on-premise network, can resolve a secret stored in AWS Secrets Manager. The node only communicates with the platform; the platform handles the relay to AWS. This allows you to maintain your existing, audited secret management infrastructure as the ultimate source of truth, while giving your distributed pipelines secure access from anywhere.

Security by Design: More Than Just a Store

We obsessed over the security of this feature.

When a secret is resolved, its value exists in the node's memory for the briefest possible moment. Immediately after use, we explicitly erase the memory region where the secret was stored, dramatically reducing the attack surface.

We encrypt all transfers and only decrypt a secret at its actual use site. Moreover, we censor secrets in diagnostics, in case they would show up in a warning or error.

Secrets cannot be persisted. They are not a at-rest feature. In fact, a secret is just a unique ID that the node looks up when a pipeline uses the secret value.

The Road Ahead

AWS Secrets Manager is just the beginning. We are already planning to integrate with other popular providers, with Hashicorp Vault next on our list.

Ready to eliminate credential sprawl? Dive into the documentation to get started:

• Explanations: Secrets

• Guide: Configure the Secret Store

• Changelog: Tenzir Node v5.7.0

• Changelog: Tenzir Platform v1.14.0

Join our community on Discord to share your feedback and help us shape the future of security data pipelines.