Decision-makers in federal government agencies are always looking for ways to improve efficiency, optimize workflows, and streamline government processes. Automated Data Pipelines are the ideal marriage of data delivery and efficient processing for federal agencies seeking savings in time and resources while still maintaining the accuracy of their information. This technology automates the process of collecting, cleansing, transforming, and delivering data and is a cost-effective and efficient way for federal agencies to manage their information.
Data Security & Accuracy
The Office of Information Security for a US Executive Department was faced with consolidating over 20 different cybersecurity data streams into their data warehouse. The department was using a semi-automated process to ingest nightly data dumps through a series of scripted processes with manual handoffs. Even though the legacy process provided some time-savings over a purely manual process, it still required manual data pulls, was not extensible, and was quickly nearing end-of-life due to the age of the technology on which it was originally developed.
In response, IntelliDyne implemented Alteryx, a low-/no-code data science platform, and has developed robust, extensible automated data workflows that reliably extract, transform, and load over 20 data feeds nightly, populating the department’s cyber data warehouse for reporting and analytics. IntelliDyne also embedded API calls in these workflows to pull FISMA, POAM, A&A, Asset, and other critically-important information from eMASS – the U.S. Department of Defense’s cyber information system, and currently receives and processes millions of security records nightly.
Another benefit of having created the data pipelines is that compliance with security policy can now be inspected while data is ‘in flight’ and on its way to the data warehouse. Nightly, the automated pipelines are looking for information systems that may be out of compliance with specific security controls such as Multi-Factor Authentication and Data Encryption. Upon finding instances of non-compliance, system owners are alerted and directed to a dashboard that provides information regarding the systems’ specific security issues and provides guidelines for taking corrective action.
Benefits of Automated Data Pipelines
Automated pipelines help federal agencies ensure data accuracy and integrity, protect data security, and improve operational scalability, allowing for:
- Management of large data sets, systems, and process
- Automatic correlation and delivery of relevant data
- Automation of required next steps and action items
- Audits of the quality of data quickly and easily
- Audits of the quality of data quickly and easily
- Detection of security issues throughout the pipeline
Additionally, federal agencies can customize automated pipelines to meet their specific data requirements, enabling them to quickly update systems when new regulations or laws take effect. This technology can help federal offices optimize their data management processes, reduce costs and errors associated with manual labor, and improve the accuracy of their datasets.