TDM Release 4.8 delivers a number of powerful new features, including Horizontally Scalable Portal Masking, which helps to efficiently mask high volumes of data without affecting TDM Portal performance.
What is Data Masking and Why Do We Need It?
Data masking is basically hiding the original data with new content so that the original data is unrecognizable. For example, let’s say you need credit card numbers for an application test. Using production data with actual credit cards is not acceptable and in many cases violates many regulations regarding the use of personal information.
One solution to this problem is to copy the data from a production environment and then modify the credit card data so that it’s unrecognizable. Masking maintains the original structure of the data without using the real data.
What is Horizontally Scalable Portal Masking?
In TDM Release 4.6, we introduced an end-to-end workflow to identify, tag and mask sensitive data stored in various tables across disparate but connected data sources – all of that in an intuitive and easy to use TDM Portal UI.
Masking is typically based on resource- intensive operations that deal with huge volumes of data. This often leads to the presence of long-running jobs in the job queue, which can cause other jobs to stall. It also requires administrators to allocate significant hardware resources for the single Portal machine, because all these resource-intensive processes are running on the TDM Portal machine itself.
In TDM 4.8 we are introducing Horizontally Scalable Portal Masking architecture that prevents that from happening – and in fact much more.
Horizontally Scalable Portal Masking is based on a network of connected masking engines that can run on different machines independent of the machine where Portal is deployed. These engines are available to take up masking jobs triggered from the TDM Portal, which ensures that resource-intensive tasks can run on machines other than the one running Portal.
In addition to that, a big masking job could be split into smaller sub-jobs (based on schema or table level) that will be distributed between available masking engines and run in parallel to achieve horizontal scalability and increased performance.
Once all sub-jobs complete their work, the logs and audit files are consolidated and available in TDM Portal to provide a smooth and transparent user experience.
New TDM Portal Homepage and Insights Dashboard
The first thing a user sees in the TDM Portal after successful login is the Home Page. In TDM 4.8 we focused on re-design of this prominent and frequently visited page.
First of all, the TDM Portal now includes the “Getting Started” banner, for easier on-boarding of new TDM users – the “Getting Started” banner guides users through the logically connected steps, which must be completed in order to effectively use TDM Portal, within a few minutes. These are:
- Create new project
- Define connections to data sources
- Create new data model
Besides the “Getting Started” banner, the primary function of which is to help users during on-boarding, there is also a re-designed “Quick Links” panel displayed on the TDM Portal Home Page. The “Quick Links” panel provides a convenient and easy way to trigger common TDM functions like Data Masking or Find & Reserve.
Administrators and Project administrator can toggle between the Home Page view and the Insights Dashboard view, which helps to track TDM activities and utilization via a set of metrics presented as Dashboard Tiles. Using the Insights Dashboard, Administrator users can easily see patterns in the usage of TDM, and use this for internal reporting and operations tuning.
TDM 4.8 provides the following metrics:
- Running Jobs
- Job Queue Time
- Job Type Chart
- Recently Created Projects
- Connection Profiles Generators
- Find & Reserve statistics
All these metrics are available as Dashboard Tiles which can be rearranged or hidden.
Find & Reserve: Show More Properties than Model Keys
The “Model Data” table in the “My Reservations” section is no longer limited to displaying model key columns only – in TDM 4.8 the “Model data” table displays the other columns from the root entity in the Find & Reserve model. This change allows testers to see all data in the reserved entity and easily reserve data for their test needs. This enhancement applies also to the reserved data exported as a CSV file.
Find & Reserve: DB2/AS400, LUW and AIX support
Find & Reserve functionality now supports more DB2 platform flavours than DB2 on zOS support introduced in TDM 4.6. TDM 4.8 adds support for DB2/AS400, DB2 LUW and DB2 AIX. This enables the use of the TDM Find & Reserve feature (vital for tester self-service scenarios) across different variants of DB2, and also empowers testers to use the right data they need for their tests.
License Management in TDM Portal
The TDM Portal is now the central element from which to manage product activation, using either a license key or via Portfolio License Agreement terms. If the TDM product is not activated by license yet, TDM Portal displays the Product Activation Wizard, which helps with product activation, and includes the ability to upload the license key with a redesigned UI.
TDM Repository Upgrade via Portal
TDM’s internal repository structure is now automatically upgraded during the first start-up of TDM Portal 4.8 service. This process is completely transparent and requires no manual steps. A dedicated schema-management utility is available, to control the process of repository upgrade manually and also to retrieve a detailed log of repository upgrade progress.
TDM Usage Telemetry
TDM is able to collect usage data across its different components – including Portal, Datamaker, GT Subset and Fast Data Masker. This usage data can be exported via TDM Portal so you can access this and create various reports about TDM usage. For more information on this feature, please refer to TDM Documentation.
Active Directory & LDAP Configuration Improvements
Active Directory or LDAP connection settings are no longer limited to the mandatory setting of User or Group Container attributes – these fields are no longer defined as required. This change allows users to setup connection to Active Directory and LDAP in a more flexible way that is no longer limited by the requirement to link it with a single User or Group Container.
Data Model & PII Discovery Improvements
We have listened to the feedback we received from our customers regarding the Data Modal and PII Discovery flows. Based on that, TDM 4.8 introduces the following enhancements in the respective TDM Portal screens.
Data Discovery scan level – as well as discovering relations based on keys defined at the database level, the TDM Portal Data Discovery process can now discover relationships that are defined on the application level. Data Discovery uses heuristics-based data content and column matching, however in some cases the key-based relationships may be sufficient. TDM 4.8 now provides the ability to pick the Data Discovery scan level, to either use key relationships only (which is the default) or to perform an advanced scan based on content.
Column name based PII classifiers – the PII Discovery process in Portal now uses classifiers that can determine the nature of the data based on the column name. This, together with the already available PII Discovery based on the data content provides more accurate PII Discovery results, and also provides an option to run a quick PII Discovery based on column names only.
PII Discovery scan level – PII Discovery settings allow users to define the scan level based on the percentage of data to be scanned. However, a scan level of 10% could result in millions of rows for very large datasets, or just hundreds for smaller datasets. To be able to control the level of the scan on a more granular level, TDM 4.8 provides a way to define the absolute limit of rows to be scanned for a table. This can significantly reduce the time required to perform PII scan, while still keeping the ability to control scan level.
TDM Masking with Brightside
The TDM documentation contains a step-by-step guide to performing TDM mainframe masking operations using the Brightside project, which allows users to perform mainframe operations via a command-line interface available for distributed platforms like Windows, Linux or MacOS. This enables a unique method for users to easily trigger masking jobs, and also to integrate mainframe masking into industry standard CI/CD platforms like Jenkins.
This new version can be downloaded directly from the Support Site. Once logged in you will be able to select the latest version from Download Management. To find the files more quickly, search for ‘Test Data Manager’ in the Search field.
For more information, watch our recent webinar on TDM 4.8 here.
Petr works as Product Owner at Broadcom with focus on Continuous Testing products and solutions. In past he worked in architect and product roles in areas of workload automation, application security and blockchain research. His current passion is finding of innovative ways how to take Continuous Testing solutions to the next level and make them developer-friendly.