Data Import & Field Mapping

Overview

Import Mapping lets administrators upload a CSV file to update the profiles of LINE friends who already exist in the system. It is designed for CMS administrators who hold customer data in an external system and need to sync those values onto friend profiles or custom attributes.

Key principles to understand before using the feature:

• The import is update-only — no new users are ever created from the CSV.
• A match key is always required: a pairing between a profile field and a CSV column that tells the system which friend each row belongs to.
• Rows with no matching friend are counted as "no match" and skipped; they do not fail the whole file.
• The import runs as an asynchronous background job. After submitting, the user returns to the list page to track progress.

The feature spans two screens: the import history list (/import-mapping) and a four-step wizard for creating a new import (/import-mapping/import).

An import job can be in one of five states:

Status	How it appears on screen
pending	Queued for processing
processing	Currently running
success	Completed
partial	Completed with errors
failed	Failed

The selectable fields come from the backend and fall into two groups: fields usable as a match key, and fields that can receive imported values. Target fields prefixed with custom. are grouped separately as "Custom attributes" — these are the custom attributes declared in Attribute Setup.

Business Flow

Import history list (/import-mapping)

1. Opening the page checks the user's access rights, then sets the breadcrumb and highlights the side-menu entry.
2. The system loads the import history for the current page. There is no search or filtering here — only pagination.
3. The system checks whether any job on the visible page is pending or processing. If so, the data refreshes automatically every 10 seconds, and the refresh stops once no jobs remain in progress.
4. The table shows the file name (truncated, with a tooltip for the full value), a colour-coded status tag, the record counts, the creation date, and a log download link.
5. The records column adapts to status: finished jobs show how many rows were updated and how many were skipped, while running jobs show the total number of rows awaiting processing.
6. If a job produced a log file, the Import log column becomes a download link that opens in a new tab; otherwise a dash is shown.
7. When no history exists, the table renders an empty state with an explanation and an "Import CSV" button so the user can start immediately.
8. Clicking "Import CSV" opens the wizard for creating a new import.

Import wizard (/import-mapping/import)

Step 1 — Upload

1. Drag a CSV file onto the drop zone or browse for it; one file at a time is supported.
2. The file size is checked first. Anything over 20 MB is rejected with an error and never uploaded.
3. Once accepted, the file is uploaded and the system receives the file name, its storage path, the column names, sample rows, and the total record count.
4. Uploading a new file resets every subsequent step — the match key, all column mappings, and any validation results.
5. A summary line confirms the file name and record count. The Next button stays disabled until a file has been uploaded.

Step 2 — Preview

1. A summary line states how many sample rows are shown out of the total.
2. The preview table renders real data from the file. Each cell is kept to a single line with a tooltip carrying the full value, and empty cells display a dash.
3. This step exists to confirm the file was parsed as expected before mapping is configured.

Step 3 — Mapping

1. Match key card — pick a profile field on the left and a CSV column on the right to define how rows are matched to friends. The profile options are grouped into profile fields and custom attributes.
2. The column chosen as the match key is automatically removed from the set of columns that write values, since it now serves as the lookup key.
3. Column mapping card — every CSV column is listed as a row showing the column name with its first non-empty sample value, an arrow, and a target field selector.
4. The match key column has no selector; it displays a "match key" tag instead.
5. The first option in every selector is "Don't import", for columns that should be ignored, and the selectors support type-ahead search over field names.
6. A target field already claimed by another column is disabled, enforcing a strict one-column-to-one-field relationship.
7. Colour is used to distinguish data origin: columns from the user's file use neutral tones, while system fields use the brand blue.
8. The Next button becomes available once both halves of the match key are set and at least one column has been mapped to a target.
9. Clicking Next sends the mapping configuration to the backend for a pre-flight validation, and the result feeds the final step.

Step 4 — Review and start

1. Four summary cards show the total row count, how many rows will be updated, how many found no matching friend, and how many contain invalid data.
2. If any rows will be skipped, a warning banner reports the combined total so the user can decide whether to fix the file first or proceed.
3. If the backend returns skip reasons, they are shown in a table pairing each reason with its row count.
4. Clicking "Start import" creates the job, submitting the file details, the match key, the column mappings, and the validation result from the previous step so the backend retains a snapshot of what was checked.
5. On success the system confirms the job was queued and returns the user to the list page to follow its progress.

What the validation numbers mean

Metric	Meaning
Total rows	Every row in the file that was checked
Will update	Rows successfully matched to a friend that will actually be updated
No match	Rows where the match key found no friend; these are skipped
Invalid	Rows whose data is malformed; these are skipped
Skip reasons	A breakdown of why rows were skipped, with a count for each reason

The same figures are submitted along with the job and drive the pending row count shown in the list, until the real results replace them once the job finishes.

Key Screens & Components

List page (/import-mapping)

• Page header and import button — the standard CMS section header with the Import CSV action.
• History table — file name, status, record counts, creation date, and log download link, with pagination.
• Empty state — shown when no history exists, with an explanation and a button to start the first import.
• Automatic status tracking — refreshes every 10 seconds while any visible job is still running.

Primary files: src/app/import-mapping/page.tsx, src/components/import-mapping/import-mapping-list.container.tsx, src/components/import-mapping/import-mapping-table.tsx

Wizard page (/import-mapping/import)

• Step indicator — tracks the four stages, with Back and Next buttons that adapt to the current step.
• Upload step — a drop zone with size validation and an upload summary line.
• Preview step — a sample-data table drawn from the file.
• Mapping step — the match key card and the per-column mapping card.
• Review step — the summary statistic cards, the skipped-rows warning, and the skip reasons table.

Primary files: src/app/import-mapping/import/page.tsx, src/components/import-mapping/import-wizard.container.tsx, src/components/import-mapping/steps/upload-step.tsx, src/components/import-mapping/steps/preview-step.tsx, src/components/import-mapping/steps/mapping-step.tsx, src/components/import-mapping/steps/review-step.tsx

API service

All calls live in src/services/import-mapping.service.ts under the import-mapping base path.

Capability	Endpoint	Used in
Upload a CSV file	POST /import-mapping/upload	Step 1
Load target fields and match key options	GET /import-mapping/fields	Loaded when the wizard opens
Validate before importing	POST /import-mapping/validate	Step 3, before advancing to step 4
Create an import job	POST /import-mapping	Step 4
List import history	GET /import-mapping	List page and status polling

Known limitations

• There is no job detail page and no way to cancel or delete a job — once submitted, it must run to completion.
• The list page offers no search, filtering, or sorting; only pagination.
• Automatic status tracking covers only the jobs on the page currently displayed; running jobs on other pages are not refreshed.
• The log download link is a backend-generated URL that must be self-accessible; it does not pass through CMS authentication.
• Returning from the review step to change the mapping requires clicking Next again to re-run validation.

Dependencies

• Permissions — both the list and the wizard share the view permission on the import-mapping module, so anyone who can open the list can also run an import.
• Attribute Setup — target fields prefixed with custom. are the custom attributes declared in Attribute Setup, so adding a field there makes a new import target available here.
• Friend profiles — the destination of every update. Imported values then surface in the All Friends report and can be used as Audience conditions, subject to each attribute's configuration.
• Background job system — the actual processing happens server-side; the UI tracks progress only through the job status, the summary counts, and the log file.
• Member Database — a separate feature that stores CSVs as standalone reference datasets rather than overwriting friend profiles.
• Shared infrastructure — the CMS authentication and HTTP client, the breadcrumb and side-menu system, and the standard section header and table scroll area used across other list pages.

Backend Details (CMS API)

The backend lives in internal/modules/importmapping/. This module only prepares the work and accepts the instruction — the actual profile updates are performed by line-management-worker-go, a separate service.

Required permissions

• Every route is wrapped in the module gate for the import-mapping module.
• The declared policies differ per endpoint: readAll for the job history, read for the target field list, while uploading a file, running pre-flight validation, and creating an import job all require create-level policy.
• Note — the front end gates both the list page and the wizard behind the same view permission, but the backend draws a clearer line between reading history and actually launching an import.

Where the target fields come from

• The selectable field list (GET /api/import-mapping/fields) is assembled from two sources combined: the standard fields of the line_user table and the custom attributes declared in the attribute_master table.
• Adding a new attribute in Attribute Setup therefore makes a new import target available here immediately, with no change needed in the import module.

Validation and business rules enforced by the backend

• The upload step (POST /api/import-mapping/upload) stores the file on object storage first and returns only the detected column headers. The file itself is not re-sent in later steps.
• The pre-flight validation step (POST /api/import-mapping/validate) actually compares the data against the line_user table for the current LINE OA, not merely against a format specification. The "will update" and "no match" figures on the review step are the result of real lookups at that moment.
• Every query is scoped to the LINE OA and organisation currently in use, so the process never touches another OA's friends.
• The import is update-only: the backend never creates users from the file, and unmatched rows are counted and skipped rather than failing the job.

What gets stored, and the side effects

• Clicking "Start import" does not update anything immediately. It inserts a job row into the import_mapping_job table with a pending status and then publishes the job id onto a message queue for the worker to pick up.
• The worker is what writes values into the line_user table and what advances the job status. The status the UI polls therefore comes from the worker, not from cms-api.
• The pre-flight validation result is submitted along with the job so the backend retains a snapshot of what was checked, until the real results replace it.
• The original CSV file remains on object storage after the job completes.

Edge cases worth knowing

• The validation result is a snapshot, not a reservation. If friends are added or removed while the job waits in the queue, the number of rows actually updated can differ from the figures shown on the review step.
• If the worker is down or the queue is backed up, a job stays pending indefinitely with no error signal on the CMS side, and there is no endpoint to cancel it.
• The downloadable log is a file the worker wrote to storage, so the link is direct file access and does not pass through a second CMS permission check — treat anyone holding the link as able to read it.
• This module has no endpoint for a single job's detail, so everything the UI can show comes from the paginated list and the log file alone.