Secure your data streams the right way

Secure your data by using Data Protection Policies, Data Masking and ACL Management

September 6, 2023

Sometimes it seems like we’re still living in the Dark Ages when it comes to securing data. This is especially true when it comes to streaming applications, where the potential to send large amounts of sensitive data through the network, non-stop, can be overwhelming for IT security administrators.

“U can’t touch this”

Unfortunately, I’ve seen far too many companies and agencies deal with streaming data security in MC Hammer style-by strictly locking down access to it. I’ve seen it locked down so tightly that developers cannot access the data for development and troubleshooting issues. Analysts cannot access it either, which reduces the ROI on real-time data.

When access to data streams is blocked or limited, it means that not only the developers but the business in general is flying blind. Applications take longer to build and are infinitely less robust and useful. And real-time data takes longer to be put to use in new ways or on an ad hoc basis within the organization, which lowers the company’s agility.

Ironically, with necessity being the mother of invention, developers sometimes work around the security barriers by simply logging the data in order to gain insight for troubleshooting. This results in the data ending up in Elasticsearch, which is accessible to all. Ouch!

The better way to do it: Data Masking with Kadeck Teams

To ensure this does not happen in your organization, Kadeck Teams (which you can run for free) provides a comprehensive privacy policy module that covers data masking and auditing. Data masking involves partially or completely obscuring certain fields of a primitive or structured data object before it is displayed to the user. This is done based on policies that can be defined either for a single stream or cluster (e.g., production) or for multiple specific streams and clusters.

A data policy comprises four parts, namely:

Classification (e.g., PII, Social Security Number, GDPR)
Resources (All, certain servers, all + certain topics, or regular expressions)
Impact (Low, Medium, High - for auditing purposes)
Fields (e.g., "creditcard", "creditno", "cardno" or regular expressions)
Masking Type (either random chars, or specific redaction methods)

Data Protection Policy overview

These policies are applied on key, value and headers (converted to string) directly after decoding the data. In the case of a decoding failure, no raw data is sent for troubleshooting.

Fine-granular access rights and team spaces

You can assign rights and roles to your data teams and individual engineers that give fine-grain access to data and functions in Kadeck for individual data streams or entire environments such as production and development. These can also be assigned automatically via groups in Directory Services.

Kadeck follows the philosophy of self-service teams, which means that all access rights required by application development teams can be restricted so that they can operate independently in their own team space.

This gives teams access only to the data streams and consumer groups they need for their project, and allows them to create and manage ACLs for their own application without seeing the ACLs of other teams.

Data Security Policy ACL Management

Audit

Every time a user accesses data or executes functions in Kadeck, the activity is stored in Kadeck's audit log. This way, you won't miss anything and you'll always be on the safe side.

For a more in depth look on how to set up your Data Protection Policies visit our Support center: Data Protection Policies and Data Masking‍

Benjamin Buick

CEO

Software architect and engineer with a passion for data streaming and distributed systems.

Secure your data streams the right way

Benjamin Buick

Latest posts

Kadeck 5.1 - Alerts, improved AI-based Health Assistant, and more!

Kadeck 5.0 - A quantum leap in Apache Kafka infrastructure monitoring

A first look at the WarpStream data streaming platform