 |
|
84 More Data Sources and a 6th NextForm Edition
IRI has partnered with CONNX Solutions to acquire and work directly with a wider range of data formats in the IRI Workbench, including:
84
new formats are supported through special JDBC (viewing) and ODBC (movement) drivers available for users of: IRI CoSort for big data transformation, IRI FieldShield for data masking, and IRI NextForm for data migration. Each also supports data replication, remapping, and reporting.
For the full list of supported sources and targets for data integration, masking, migration, replication, federation, and reporting, see this page.
To provide a simple and affordable way to move data from these legacy formats into more modern applications, IRI will offer a sixth, Legacy edition of NextForm to migrate more data, joining these prior five editions:
1. Lite - a free edition for flat-file (e.g., LDFIF>CSV), data-type (e.g., Packed > Numeric), and endian (e.g., big>little) conversions
2. COBOL - for managing Micro Focus ISAM and Vision index files
3. DBMS - to facilitate platform migrations like SQL Server to Oracle
4. Unstructured - to find, flatten, and forensically examine data in MS Office, PDF, XML files, email, etc.
5. Premium - a combination of the above
The cost of NextForm Legacy edition willl include the use of the driver(s) you need. Email nextform@iri.com if you need to move, mask, or otherwise make use of the data in those sources.
TAP3 Call Detail Record (CDR) Support
Unique among big data management and protection software, IRI CoSort can now directly process CDR data in the TAP3 roaming standard of ASN.1. Without the need for separate mediation software and steps, CoSort users can transform, convert, protect, and report from raw CDR files, as well as integrate them with other data sources for mash-ups, hand-offs, and analytic tools. If your CDRs are in another ASN.1 format, please let us know. See this blog article for a sample CDR
job in CoSort.
8X Faster Data Prep for Tableau
An independent BI expert recently demonstrated the performance benefit of using IRI CoSort as a data preparation tool for Tableau. Pre-visualization tests were conducted using both Tableau and CoSort to integrate and query large data sources necessary for visualizations, and they showed an 8x speed difference at the 20-million row level. See this blog article for a summary of the tests.
Using ODBC? A Loaded Comparison
Have you ever wondered about the relative data movement performance of ODBC? When it comes to pulling data from SQL Server tables, Microsoft has maintained that ODBC drivers will deliver data as quickly as native ones. But what about loading, and particularly in the case of Oracle, which is more often used in large data warehouse ETL environments? Our tests show ODBC is a viable alternative below a million rows, but using a native driver for extraction, and bulk loads of pre-sorted flat files, is much faster. See the results of a basic test here, and tell us what you think.
Tech Tip: Use Format Templates to Improve Data Quality
A template is a structure used by the SortCL program in IRI CoSort to describe a particular format of a source and target data field. CoSort users can create templates with simple or composite values, and use them to check their output during data processing to find data errors. See an example here.
|
|
|
Copyright © 2014, Innovative Routines International (IRI), Inc. All Rights Reserved. CoSort, FieldShield, and NextForm are registered trademarks, and RowGen is a trademark of IRI, Inc. FACT is a trademark of DataStreams Corp. (CoSort Korea). All other product, brand, or company names are, or may be, (registered) trademarks of their respective holders.
|
|
|
|
|
More Dark Data Insights
Much of the data corporations keep to comply with retention policies can also be useful. The challenge is that a lot of data sits in unstructured repositories, which are hard to access. In the previous newsletter, we discussed the
new ability IRI Workbench users have to search and structure values in unstructured sources, and then use the flattened extract data in data mash-ups, analytics, etc. What's new this quarter is IRI's ability to conveniently discover some forensic information along with that data. The Data Restructuring wizard in IRI NextForm can now display metadata about each file, including: its author, creation and modification dates, hidden attributes, linkages, etc. This information can then be used in
compliance audits and data stewardship efforts.
Web Log Data Masking
IRI CoSort users have long been able to manipulate CLF and ELF web log data. Those file formats and a processing example are shown here. Now, CoSort and FieldShield users can also mask sensitive information found in those logs, such as IP addresses. Read about the methodology here.
Use IRI Software to Speed and Secure Pentaho
We recently demonstrated a few beneficial ways to use IRI software in Pentaho's 'Spoon' workflows. Calling IRI CoSort to speed up the sort process in Pentaho with a 1GB source showed a near 15X improvement. Calling IRI FieldShield jobs from the shell step offers more than a dozen data masking functions to protect table and file sources at the column level.
Calling pre-built IRI RowGen flows from Pentaho will produce safe, intelligent test data for Pentaho without having to access or mask its production data. See this section of the IRI blog site for sample calls.
Master Data Management with EGit
Because the IRI
Workbench is built on Eclipse, its users can leverage the plug-ins and applications also designed for Eclipse. EGit is the Eclipse provider for Git, a popular, secure, and cloud-ready repository used for source code control, and the management of other project assets like IRI metadata. In the same way, IRI Workbench users can enable and enforce master data centralization and consistency across IRI data management applications. EGit integrates seamlessly with master data and master metadata in the Workbench for team-sharing, change management, and security. Read more about using EGit for MDM here.
|
