What I Learned From Migrating E-Commerce Platform Data

When Do You Need to Migrate Data?

At work, I have helped vendors migrate from one e-commerce platform to another. This kind of need usually comes up after a vendor has been running their store on the old platform for a while, then decides to move because they want more features or because of various business decisions. Since they have already been operating on the old platform for some time, they will usually have accumulated quite a bit of data. From the vendor’s point of view, of course they want to bring that data with them. And because they usually do not want to keep paying the old platform, they will eventually need to close the old platform’s admin panel. That is when the data originally stored on the old platform needs to be moved to the new platform.

This kind of data usually falls into three groups: products, customers, and orders. Below are some notes from my experience with each of them.

Products

Products are basically fairly straightforward. Compared with the other types of data, there are usually fewer records, and even after migration, the vendor often still needs to rearrange and clean up the product pages themselves. The issues that tend to come up here are usually around how each platform defines categories, or how backend categories map to what gets displayed on the frontend. Every platform has its own way of doing this, but categories are usually something the vendor handles on their side.

Customers

Customer data can also be migrated, but after the migration, customers will not necessarily move with you 😹. From the engineering side, all we can really do is help the vendor save the data to the new platform and preserve contact information and membership levels. At most, we can check whether the platform supports generating account activation links and sending notification emails to customers. The vendor still needs to plan campaigns or activities to actually encourage customers to activate their accounts on the new platform.

Orders

Orders are the trickier part. In my first few attempts, I would call the source platform’s API, fetch a dozen or so records, and compare them against the fields on the target platform. Once I felt the conversion and mapping were mostly done, I would start calling the target platform’s API to feed the data in. But as the data was being sent, some records would fail to convert or fail to be created. When I went in to check why, I would realize they were special-case orders: maybe returns, maybe cancellations, and so on. So I would adjust the field mapping rules, send the data again, and then more failures would appear. After checking the details, I would find, oh, the cashback rules were not mapped correctly… so I would adjust the rules again, send the data again… and go through this loop several times until all the data was finally imported.

I ended up doing several data migrations like this, and what was funny is that I never once encountered a case where the source platform and target platform were the same combination. Taiwan’s e-commerce platform market really is full of variety. I have run into everything from traditional platforms with no API, where the only option was downloading order Excel files from the store admin panel (and the files were so huge they were hard to even open), to target platforms where I kept running into issues while calling the API to save data, only for the platform contact to say this was also their first time seeing someone use the API to import that much data 🤨.

From Random Samples to Intentional Test Cases

After doing this a few times, I tried a different approach: instead of passively receiving data ⭢ parsing it ⭢ experimenting with it, I started “actively choosing the data I wanted to inspect.” Based on earlier experience, orders can basically be divided into “normal orders” and “special orders.” A normal order is one where the customer places an order, pays, the store ships it, the customer receives it smoothly, and everyone is happy. A “special order” might involve a return, partial return, exchange, cancellation, unpaid checkout, discount, cashback, and so on. For these orders, there is absolutely no way to understand the fields just by looking at the JSON returned from the API. You have to pair it with the source platform’s e-commerce admin interface, directly find the order numbers for those cases, compare them against the actual screens, and understand what the fields you received actually mean. In other words, instead of blindly calling the API and getting a few random order records, I intentionally went into the admin panel to pick out “the shape of the orders I wanted to study,” then called the API to fetch the data and map the fields. After switching to this approach, importing data into the target platform through the API went much more smoothly. There were still some failed records, but once I looked into the details, sure enough, almost all of them were test orders (usually obvious from the email, notes, and so on).

This experience stayed with me. I used to think that ETL / data conversion was about considering the data source and destination, then building a connector in the middle, where the main engineering work was mapping fields and converting formats. But after changing my approach this time, I realized what really matters is first understanding what kinds of data I am dealing with, getting a broader view, and focusing on the important majority first: handle the 70% of normal cases, then deal with the remaining special cases separately. This way of thinking made me stop looking only at the data itself. Instead, I started looking at the data together with the real-world scenarios behind it. Only when those real scenarios are taken into account does the data itself truly make sense.