The state Department of Transportation (DOT) has data regarding a large number of business areas, including but not limited to plates, titles, drivers’ licenses, court judgments, local authorities, incidents, emissions, secured parties, dealers, agents, payment places, and salvage.
It had many separate systems that it used to manage its responsibilities, including systems for titles and registrations, drivers’ licenses, dealerships, vehicle inspections, and more. DOT wanted to migrate all of these—in total, approximately 16 million driver and vehicle records—to use a single, customer-centric relational database system. DOT would build the destination system and application itself to ensure that the new system met all of DOT's operational needs.
There were a significant number of challenges to this project.
The two largest systems to be migrated used vastly different technology: one used relational tables, while the other used a custom database type that had been built in-house. Additional information resided in many other systems, which used a variety of commercial databases.
These source systems did not have reliable keys for cross-system customer identification, and their formats were highly variable and not well understood.
To compound this problem, the data itself was complex. As the source systems accumulated data, the systems themselves did not change. The laws and regulations surrounding the DOT did change, often. As a result, the source systems contained hundreds of special cases, encoded in different ways, as DOT personnel invented strategies to accommodate regulatory changes using inflexible technology.
The data also suffered from typical data quality issues, such as duplicate data caused by accident or fraud, and data errors such as misspellings and outdated information.
The client began an analysis and cleanup project using its own staff, but the task was so difficult that the internal team would not be able to complete the project on the needed timeline.
We used our expertise and tools to help explore and describe the client's data and then perform the actual migration. This enabled the client's internal team to focus on building the destination system and application; they had already created a data model based on their understanding of their data.
Upon joining the project, we used our trailblazer approach to explore the data so that we could verify this model. Doing this, we made a significant finding: there were an extremely large number of special cases hidden in the DOT's historical data. These special cases were not documented in the existing model, and the DOT’s application-building efforts were not accounting for them.
We identified all special cases and worked with the client’s subject matter experts to create a specification for each of them, bringing the data model to parity with reality.
To bring the data from the source systems together, we built a solution that could match and merge the data automatically based on the information contained within the records. Incoming data was to enhance the quality of the matching.
A key aspect of this required utilizing address data from various system records, regardless of the address format, then using a MIO-developed technique to distill the address into a comparable identifier. Fuzzy matching further supported the identification of related records in the absence of keys.
Our solution also automatically identified relationships between the entities represented in the data, including individual-vehicle relationships, judgment-driver relationships, and many more.
The rules governing the match, merge, and relationship discovery process were created by our team in conjunction with the DOT subject matter experts, ensuring an auditable solution that reflected the depth of the client's domain knowledge and historical operations understanding.
The consolidated data was automatically standardized, cleansed, and otherwise prepared to the specifications of the new system.
Finally, after we had created the consolidated customer data from all systems and the new system was ready, we used our tools to quickly and safely execute the migration of the data from the original source systems to the new system.
With MIOsoft’s help, the DOT gained golden customer data, derived from its many historical systems, that was prepared for the new destination system according to business standards.
Because of our trailblazer strategy, the DOT was alerted to a significant gap in its specification while the new system was still under development. We completed the specification and, by finding this relatively early in the process, enabled our client to develop the system with the updated model from the get-go.
The DOT could plan for the new system to go operational with confidence that the new application could handle all the nuances of the data, including historical special cases, and that the data in the new system would be up to business standards.
Using our staggered migration approach, the new application went live with all historical and current data up-to-date. During go-live, completing the migration needed less than an hour during cutover.
