Node
Platform
Release
Stop the Sprawl: Centralize Your Pipeline Secrets with Tenzir and AWS
In any sophisticated SecOps environment, data pipelines are the critical arteries, moving telemetry from source to destination. But as these pipelines multiply, they often leave a trail of hardcoded credentials. This "credential sprawl" is more than just messy—it's a ticking time bomb for security.
Managing secrets is a foundational requirement for any serious data pipeline deployment. That's why we are excited to announce a new, robust approach to credential management. This isn't just an add-on; it's a core component for building secure, manageable, and scalable data pipelines, featuring a direct integration with AWS Secrets Manager.
Using Secrets in Tenzir: The secret()
Function
We've made using secrets in your pipelines incredibly simple and explicit. The key is the secret()
function in the Tenzir Query Language (TQL).
Let's say you have a pipeline that forwards data to a Splunk instance using its HTTP Event Collector (HEC). Instead of pasting the HEC token into your pipeline, you now pass it to the to_splunk
operator by calling secret()
:
When this pipeline runs, the to_splunk
operator retrieves its hec_token
argument by resolving the secret named splunk-hec-token
. The secret's value is never exposed in the pipeline definition itself.
The Secret Resolution Flow: A Clear Hierarchy
Tenzir resolves the value for secret("splunk-hec-token")
by checking three locations in a specific order. This gives you a powerful combination of flexibility for development and central control for production.
1. Local Node Configuration
First, the node checks its own local configuration, either tenzir.yaml
or environment variables for containers. This is the perfect place for node-specific credentials or for temporarily overriding a central secret during development and testing, without affecting other deployments.
2. The Platform's Built-in Store
If the secret is not found locally, the node automatically queries its connected platform. The platform comes with a secure, built-in secret store that acts as the central source of truth for your credentials. If you're not using an external secret store, this is the recommended place for most secrets.
You can easily manage secrets in the platform's store via the command line:
Alternatively, use the platform user interface to configure your secrets:

3. External Secret Stores: AWS Secrets Manager
Alternatively, if you configured an external secret store, the platform attempts to resolve secrets from there. We are launching this capability with first-class support for AWS Secrets Manager.
This federated architecture is uniquely powerful. A Tenzir Node, running deep inside your on-premise network, can resolve a secret stored in AWS Secrets Manager. The node only communicates with the platform; the platform handles the relay to AWS. This allows you to maintain your existing, audited secret management infrastructure as the ultimate source of truth, while giving your distributed pipelines secure access from anywhere.
Security by Design: More Than Just a Store
We obsessed over the security of this feature.
When a secret is resolved, its value exists in the node's memory for the briefest possible moment. Immediately after use, we explicitly erase the memory region where the secret was stored, dramatically reducing the attack surface.
We encrypt all transfers and only decrypt a secret at its actual use site. Moreover, we censor secrets in diagnostics, in case they would show up in a warning or error.
Secrets cannot be persisted. They are not a at-rest feature. In fact, a secret is just a unique ID that the node looks up when a pipeline uses the secret value.
The Road Ahead
AWS Secrets Manager is just the beginning. We are already planning to integrate with other popular providers, with Hashicorp Vault next on our list.
Ready to eliminate credential sprawl? Dive into the documentation to get started:
Join our community on Discord to share your feedback and help us shape the future of security data pipelines.