Implementation Plan — Deduplicate media on product import (v1.6.1)

1. Symptom

A product CSV in which the variable parent and all its variations share one Obrázky URL currently produces image.jpg, image-1.jpg, image-2.jpg, … — one fresh download and one distinct attachment per row. The parent and every variation end up pointing at a different physical copy of what is logically the same image.

2. Root cause (condensed)

Full analysis is in fix-multi-image-plan-00.md. In short:

upload_image_from_url() (includes/class-studiou-wc-product-manage-product-import.php:464-511) unconditionally runs download_url() + media_handle_sideload() with no dedup, and it is called once per row — by process_variable_product() (:288) for the parent and by process_variation() (:412) for each variation. On every call after the first, media_handle_sideload() → wp_unique_filename() sees uploads/YYYY/MM/image.jpg already exists and renames the new upload to image-N.jpg, minting a fresh attachment + file each time. That is the <name>-<ordinal>.jpg clutter, and the reason each variation references its own attachment ID instead of sharing the parent's.

3. Goal

When several rows reference the same Obrázky URL, the importer must:

Download that URL once per unique (trimmed) source-URL string.
Make every later row reuse the existing attachment, so the variable parent and all its variations converge on one attachment ID.
Hold across the AJAX batch boundary (5 rows per request) and across entirely separate imports — even weeks apart.
Touch nothing that prior imports already created — additive only, no migration.

4. Non-goals

No retroactive merge of duplicates left by past imports (see follow-ups).
No content hashing and no aggressive URL normalization. Same URL string ⇒ same image; two different strings stay independent even if the bytes are identical. Normalization is limited to trimming surrounding whitespace.
No change to failure semantics. A failed download/sideload still returns null and the row proceeds with no image.

5. Design — two-layer cache keyed by the exact trimmed source URL

The key for every step — in-request cache, DB lookup, download, stored postmeta — is the same trimmed URL string the validator already approved. If those strings ever diverge, dedup silently misses (see §5.3).

5.1 Layer 1 — persistent (attachment postmeta)

Each sideloaded attachment is tagged:

meta_key = _studiou_wcpcm_source_url
meta_value = <the exact trimmed URL from the Obrázky column>

Before downloading, look up an attachment carrying that meta value. Hit ⇒ reuse its ID. This survives across batches, across imports, and across server restarts.

5.2 Layer 2 — in-request static cache

private static $url_cache = array() on Studiou_WC_Product_Manage_Product_Import, checked before Layer 1 and written on every resolution (DB hit or fresh sideload). It saves a DB round-trip when the same URL recurs inside one AJAX request (one batch, up to 5 rows) and resets naturally when the request ends — Layer 1 carries the cross-request case. (A fresh Product_Import object is built per request, so static vs. instance scope are equivalent in lifetime here; static is used to state the request-scoped intent and to share across any second instance in the same process.)

5.3 Key normalization — also a latent correctness fix

The two callers currently pass the raw cell while the validator approved the trimmed value:

process_product_row() validates trim($data['Obrázky']) (:203).
process_variable_product() (:288) and process_variation() (:412) call upload_image_from_url($data['Obrázky']) — untrimmed.

So " https://x/img.jpg" is validated trimmed but downloaded padded, and two rows differing only by surrounding whitespace would key and download separately.

Fix: trim() once at the top of upload_image_from_url() and use that single value everywhere. No case-folding, no trailing-slash or query-string stripping for the key (see Non-goals). The query string is stripped only from the on-disk filename (§6.1c) so wp_unique_filename() doesn't fold ?v=2 into the basename.

5.4 Why postmeta, not a transient

Postmeta already delivers cross-batch persistence and is durable beyond the import session — a re-import a week later still hits it. A transient would duplicate that, need explicit teardown, and expire at an arbitrary moment mid-import. Postmeta is both simpler and stronger.

5.5 Why a plugin-namespaced meta key

We use our own _studiou_wcpcm_source_url rather than WooCommerce's internal _wc_attachment_source convention. Rationale: the namespaced key is predictable, owned by this plugin, and immune to changes in WooCommerce importer internals. The trade-off — we won't dedup against images that WooCommerce's native CSV importer brought in — is acceptable; this importer is the only writer of these images in practice. (A future enhancement could additionally probe _wc_attachment_source on a miss; deferred to keep this change self-contained — see follow-ups.)

6. Changes

6.1 `includes/class-studiou-wc-product-manage-product-import.php`

(a) Static cache property — add immediately after the $db property (:22):

/**
 * In-request cache of trimmed source URL => attachment ID, so the same URL
 * appearing multiple times within one AJAX batch costs at most one DB lookup.
 *
 * @var array<string,int>
 */
private static $url_cache = array();

(b) Finder method — add next to upload_image_from_url():

/**
 * Find an attachment previously imported from this exact source URL.
 *
 * Uses get_posts (not a raw $wpdb query) so a row whose attachment post was
 * deleted but whose meta was somehow orphaned can't resolve to a missing post.
 * Orders by ID ascending so the earliest (canonical) attachment wins
 * deterministically if more than one ever carries the same source URL.
 *
 * @param string $url Trimmed source URL.
 * @return int Attachment ID, or 0 if none.
 */
private function find_attachment_by_source_url($url) {
    $ids = get_posts(array(
        'post_type'              => 'attachment',
        'post_status'            => 'inherit',
        'posts_per_page'         => 1,
        'orderby'                => 'ID',
        'order'                  => 'ASC',
        'fields'                 => 'ids',
        'no_found_rows'          => true,
        'update_post_meta_cache' => false,
        'update_post_term_cache' => false,
        'meta_query'             => array(
            array(
                'key'     => '_studiou_wcpcm_source_url',
                'value'   => $url,
                'compare' => '=',
            ),
        ),
    ));

    return !empty($ids) ? (int) $ids[0] : 0;
}

(c) Replace upload_image_from_url() (:464-511) entirely with:

/**
 * Upload an image from a URL, deduplicating by source URL.
 *
 * A given source URL is downloaded at most once: an in-request static cache
 * absorbs repeats within a batch, and a `_studiou_wcpcm_source_url` postmeta
 * lookup reuses attachments across batches and across imports.
 *
 * @param string $url Image URL (raw value from the Obrázky column).
 * @return int|null Attachment ID, or null on empty input / download failure.
 */
private function upload_image_from_url($url) {
    // Normalize the key. process_product_row() validates a *trimmed* URL but
    // the callers pass the raw cell, so trim here to keep the cache key, the
    // DB lookup, the download, and the stored postmeta all identical to the
    // value that was validated (and to collapse whitespace-only variants).
    $url = is_string($url) ? trim($url) : '';
    if ($url === '') {
        return null;
    }

    // Layer 2 — in-request static cache.
    if (isset(self::$url_cache[$url])) {
        if (defined('WP_DEBUG') && WP_DEBUG) {
            error_log('STUDIOU WC IMPORT: image cache HIT (request) ' . $url . ' -> ' . self::$url_cache[$url]);
        }
        return self::$url_cache[$url];
    }

    // Layer 1 — persistent postmeta lookup (across batches and re-imports).
    $existing = $this->find_attachment_by_source_url($url);
    if ($existing) {
        self::$url_cache[$url] = $existing;
        if (defined('WP_DEBUG') && WP_DEBUG) {
            error_log('STUDIOU WC IMPORT: image cache HIT (db) ' . $url . ' -> ' . $existing);
        }
        return $existing;
    }

    // Layer 0 — fresh sideload.
    require_once(ABSPATH . 'wp-admin/includes/media.php');
    require_once(ABSPATH . 'wp-admin/includes/file.php');
    require_once(ABSPATH . 'wp-admin/includes/image.php');

    // Suppress intermediate sizes for import speed. Use the stable
    // '__return_empty_array' callable (not a fresh closure) so remove_filter()
    // can actually match it, and detach it in finally so it can never leak
    // into later media operations in this request.
    add_filter('intermediate_image_sizes_advanced', '__return_empty_array');

    try {
        // download_url()'s second arg sets the request timeout (30s); no
        // http_request_timeout filter is needed.
        $tmp = download_url($url, 30);
        if (is_wp_error($tmp)) {
            error_log('STUDIOU WC IMPORT: image download failed for ' . $url . ' - ' . $tmp->get_error_message());
            return null;
        }

        // Derive a clean on-disk filename. parse_url() strips any query string
        // (img.jpg?v=2 -> img.jpg) so wp_unique_filename() doesn't fold the
        // query into the basename. Null-safe: parse_url() may return null/false
        // and basename(null) is deprecated on PHP 8.1+ (we require 8.2).
        $path = parse_url($url, PHP_URL_PATH);
        $name = (is_string($path) && $path !== '') ? basename($path) : '';
        if ($name === '') {
            $name = 'image-' . md5($url) . '.jpg';
        }

        $file_array = array(
            'name'     => $name,
            'tmp_name' => $tmp,
        );

        $id = media_handle_sideload($file_array, 0);
        if (is_wp_error($id)) {
            @unlink($tmp);
            error_log('STUDIOU WC IMPORT: image sideload failed for ' . $url . ' - ' . $id->get_error_message());
            return null;
        }

        // Tag the attachment so future rows / batches / imports dedup on it.
        update_post_meta($id, '_studiou_wcpcm_source_url', $url);

        self::$url_cache[$url] = (int) $id;
        if (defined('WP_DEBUG') && WP_DEBUG) {
            error_log('STUDIOU WC IMPORT: image sideloaded ' . $url . ' -> ' . $id);
        }
        return (int) $id;

    } catch (Exception $e) {
        error_log('STUDIOU WC IMPORT: image upload exception for ' . $url . ' - ' . $e->getMessage());
        return null;
    } finally {
        remove_filter('intermediate_image_sizes_advanced', '__return_empty_array');
    }
}

Design notes on the rewrite:

Trim at the top aligns the key with the already-validated URL — the precondition for dedup to fire at all (§5.3).
Cache-hit paths return before any filter is touched, so filter management only wraps the actual sideload.
Clean filename via parse_url(..., PHP_URL_PATH) + basename(), null-safe, with an image-<md5>.jpg fallback. The full trimmed URL still drives the key, so distinct query strings remain distinct attachments; only the on-disk name is cleaned.
Filter handling uses the stable '__return_empty_array' callable added once and removed in finally, so it detaches on every exit (success, is_wp_error, exception). The old http_request_timeout closure is deleted — redundant with download_url($url, 30) and never removed (§7).
Logging: hard failures (download/sideload/exception) log always-on under the documented STUDIOU WC IMPORT: prefix, per CLAUDE.md's "always enabled" rule — replacing the current WP_DEBUG-gated STUDIOU WC: lines. High-volume per-row diagnostics (cache hits, successful sideloads) stay behind WP_DEBUG so a 30-row single-image import doesn't emit 30 lines every run. No new user-facing strings ⇒ no translation work.
Return contract unchanged: fresh and DB paths return int; empty/failed paths return null. Both callers already gate on if ($image_id), so nothing downstream changes.

6.2 `studiou-wc-product-cat-manage.php`

Header Version: 1.6.0 → 1.6.1 (:6).
STUDIOU_WCPCM_VERSION '1.6.0' → '1.6.1' (:26).

The version-stamped asset mechanism regenerates admin-1.6.1.js / admin-1.6.1.css automatically on the next admin page load from the bumped constant — no manual renaming; the admin-1.6.0.* copies are gitignored build artifacts.

6.3 `assets/js/admin.js`

Banner console.log('STUDIOU WC: Admin script loaded v1.6.0') → v1.6.1 (:22).

No JS logic change — the dedup is entirely server-side; the AJAX flow, batch size, and handlers are untouched. This keeps the banner in sync per the release process. Because the fix is server-side, the studiou.cz ?ver=-stripping cache issue is irrelevant here — there is no behavioral JS/CSS change for a browser to miss.

6.4 `CLAUDE.md`

Update the Current Version line to 1.6.1.
Add a bullet under the product-import class description: > Source-URL deduplication (NEW in v1.6.1): each unique Obrázky URL is downloaded > once and reused via a _studiou_wcpcm_source_url attachment postmeta plus a > per-request static cache, so all rows sharing an image (variable parent + > variations) point at one attachment.

6.5 `readme.md`

Bump **Version: 1.6.0** → **Version: 1.6.1** (:3).
Add a ### Version 1.6.1 changelog entry directly above ### Version 1.6.0 (:355) summarizing the dedup fix and the two latent-bug fixes.

6.6 Translations

No new translatable strings ⇒ no .po/.mo recompile. Optionally bump the Project-Id-Version header in both .po files for consistency.

7. Latent bugs fixed as a side effect

Both live in the method being rewritten, so this is the natural moment to fix them.

intermediate_image_sizes_advanced is never actually removed. :488 and :493 add and "remove" the filter with two separate closures. remove_filter() matches by callable identity, so the fresh closure never matches the added one; the filter stays attached for the rest of the request and suppresses intermediate sizes on any later media operation. The rewrite uses the stable '__return_empty_array' callable and removes it in finally.
http_request_timeout closure added every call, never removed (:470). Across a 5-row batch this stacks up to five closures clamping every later HTTP request to 30s, and it is redundant with download_url($url, 30). The rewrite deletes it.

8. Performance / query cost

find_attachment_by_source_url() filters wp_postmeta by meta_key = '_studiou_wcpcm_source_url' with an equality compare on the full URL in meta_value. meta_value (LONGTEXT) isn't usefully indexed for a long-string =, so this scans the rows carrying that key. Acceptable because:

Only attachments imported by this plugin carry the key, so the candidate set is small relative to the whole table.
Layer 2 absorbs repeats within a batch, so at most one such query runs per distinct URL per request.
no_found_rows + update_post_meta_cache => false + update_post_term_cache => false keep the query lean.

A direct $wpdb->get_var(... ORDER BY post_id ASC LIMIT 1) would shave the WP_Query overhead, but get_posts guarantees the returned ID belongs to an existing attachment post, which is worth the small cost. If a library ever grows large enough for this to matter, the follow-up is a dedicated lookup table or an indexed hash column — out of scope.

9. Edge cases handled

Empty Obrázky → early null, no cache write (matches current behavior).
Same URL within one batch → first row sideloads + writes postmeta; later rows in the same request hit Layer 2.
Same URL across batches of one import → batch 1 sideloads; later batches hit Layer 1 (the static cache starts empty in each new request).
Same URL in a re-import months later → Layer 1 still hits.
Whitespace-only variants ("…img.jpg" vs " …img.jpg") → collapse to one attachment via the trim.
Two different URLs with the same basename → two attachments (distinct keys); the second file lands as name-1.jpg on disk, which is correct.
Query-string URL (…/img.jpg?v=2) → on-disk file is img.jpg; the attachment is keyed by the full URL including ?v=2.
Failed download → returns null, does not cache the failure; the next row retries. No worse than today.
Sideload fails after a good download → temp file unlinked, cache not poisoned, returns null.
Pre-1.6.1 attachments (no _studiou_wcpcm_source_url) → cache miss → fresh upload. We don't retroactively tag old media.
Attachment manually deleted (post + its meta gone) → cache miss → re-upload. Correct.
Concurrent batches → not applicable; the JS issues batches sequentially.

10. Edge cases NOT handled (deliberately)

URL differs by trailing slash / case → treated as distinct (predictable exact match).
Bytes change at the same URL (image rotated upstream) → reuses the stale attachment. Acceptable for an import tool; delete the attachment to force a refresh.
Same image served from two different URLs (CDN vs. origin) → two attachments; can't dedup without hashing bytes (a Non-goal).
Attachment post survives but its file was deleted from disk → the dangling attachment ID is reused, yielding a broken image reference. Rare; get_posts only guards post existence, not file existence. Out of scope — re-deleting the attachment forces a clean re-import.

11. Execution order

Edit class-studiou-wc-product-manage-product-import.php: add the static property, add find_attachment_by_source_url(), replace upload_image_from_url().
Bump version in studiou-wc-product-cat-manage.php (header + constant).
Bump banner in assets/js/admin.js.
Update CLAUDE.md (Current Version + class bullet).
Update readme.md (Version line + ### Version 1.6.1 changelog entry).
Commit as a single change.

No view edits, no JS logic edits, no CSS edits, no .po/.mo recompile, no AJAX handler changes.

12. Test plan

Happy path — CSV with 1 variable product + 5 variations, all the same image URL. Expect exactly 1 attachment row, 1 file in uploads/YYYY/MM/, and the parent + all 5 variations sharing that one attachment ID. SELECT meta_value FROM wp_postmeta WHERE meta_key='_studiou_wcpcm_source_url' → one row, value = the exact trimmed URL.
Cross-batch — 30 rows, same URL (6 batches of 5). Batch 1: one Layer 0 sideload, postmeta written. Batches 2–6: Layer 1 hit on every row, zero new downloads. With WP_DEBUG: one "sideloaded" line, then "cache HIT" lines.
Re-import — run the same CSV twice. Second run: every row hits Layer 1; zero new uploads.
Distinct URLs — 5 rows, 5 different URLs → 5 sideloads, 5 attachments, 5 postmeta rows.
Mixed — 3 rows URL A, 2 rows URL B, 1 empty Obrázky → 2 sideloads, 4 cache hits, 1 row with no image.
Whitespace normalization — row 1 "https://x/img.jpg", row 2 " https://x/img.jpg" → 1 sideload, 1 attachment, row 2 is a cache hit. (Without the trim: two attachments.)
Query-string filename — URL https://x/img.jpg?v=2 → on-disk file is img.jpg, not img.jpg?v=2; attachment keyed by the full URL including ?v=2.
Failure resilience — 3 rows at a 404 URL → each returns null, no cache write, products created without images, one always-on failure line per row.
Backwards-compat — a pre-1.6.1 attachment (no meta) is not picked up; re-importing its URL re-sideloads. Documented limitation; verify no crash.
Filter-leak regression — run a product import, then immediately upload an image through the normal WP Media Library in the same admin session; confirm intermediate sizes are generated (the filter is now removed in finally).

13. Rollback

Single commit; revert to restore current behavior. No schema changes — the _studiou_wcpcm_source_url postmeta is additive, and leftover rows after a rollback are harmless orphan metadata.

14. Optional follow-ups (not in this plan)

Probe _wc_attachment_source on a miss so images brought in by WooCommerce's native CSV importer also dedup. Cheap, but couples to a WC-internal key; deferred.
Don't set a variation image when it equals the parent's. With dedup the variation and parent already converge on one attachment ID, and a variation with no image set falls back to the parent's featured image, so set_image_id() on variations is mostly redundant. Suppressing it changes variation behavior — defer until requested.
Merge existing duplicates — an admin action that finds attachments from past imports (same base filename minus the -N ordinal), picks a canonical ID, repoints every product/variation _thumbnail_id, and deletes the orphans. Needs dry-run + confirmation UX.
Retroactively tag old attachments — a one-time action that infers original URLs for existing product images and writes _studiou_wcpcm_source_url, extending dedup back to pre-1.6.1 imports.

fix-multi-image-plan-01.md 21 KB History Raw