Export, Merge and Reimport multiple libraries

[ZarK]

Okay, I have finally sifted through 10 terabytes of photos and videos using any tool imaginary, and am left with three photo libraries.

For the final step of cleanup I will need file-based access to them, so my plan is to:

• Export the three Apple Photos libraries to disk using osxphotos
  -- As is, including any edits, duplicates etc.
  -- With as much metadata as is useful for reimporting (ideally keeping any analysis metadata from Apple, face tags, album info etc.)
  -- Including live photos, spatial photo / videos
• I will do some scripted operations on the files including changing some names
• Lastly, merge and re-import (in any order) all photos to a new, primary library

I know I can reimport with --exportdb, but I guess that won't work if I rename files in the mean time. Would a sidecar file include all data that apple photos can make use of when reimporting?

Are there any useful way for merging, or would I have three separate imports to the same library?

[ferid333]

Hello @ZarK

Step 1: Export the Libraries

Using osxphotos to export the three Apple Photos libraries is an excellent choice, as it supports exporting with metadata, edits, live photos, and albums. Here's how you can achieve your goals:

1. Export with Metadata and Edits:
   Use the --export command with osxphotos to export photos as is, including edited versions and metadata:

   osxphotos export /path/to/export --update --edited --sidecar json

   • --edited: Exports both original and edited versions.
   • --sidecar json: Creates sidecar files with metadata in JSON format, which you can use for reference later.

2. Include Live Photos:
   osxphotos will automatically handle live photos and export them as paired files (.jpg or .heic and .mov).

3. Spatial Photos/Videos:
   If your libraries contain spatial photos/videos (e.g., 360-degree media), osxphotos will export them with their metadata intact.

4. Album and Face Tags:
   The metadata in the sidecar files includes album info, face tags, and other useful data, ensuring these can be referenced later during re-import.

---

Step 2: Perform Scripted Operations

After exporting, you can script operations on the files, such as renaming them. However, note that renaming will break direct reimport with the --exportdb option, as Apple Photos tracks files by filename and path.

Workaround for Renamed Files:

• Use the JSON sidecars to maintain metadata. The sidecars contain a rich set of data, including filename, album, person, and location info.
• If renaming files, ensure you keep track of the original names and map them back when needed using a script.

---

Step 3: Reimport and Merge Libraries

There are two approaches to merging the libraries:

Option 1: Use Apple Photos Directly

• Re-import the photos manually into a new, primary library. This will involve three separate imports for each exported library.
• Metadata like albums and face tags might not fully carry over unless Photos recognizes them natively (which can vary).

Option 2: Use osxphotos for Advanced Reimport

• If you've kept the sidecar files intact, you can use osxphotos for importing while associating metadata from the JSON sidecars. Unfortunately, Apple Photos does not natively support importing custom metadata for albums, faces, etc., from sidecar files.
• Use osxphotos to generate a CSV or script that recreates albums or tags within Apple Photos after reimporting.

---

Merging Libraries

Apple Photos does not provide native tools for merging libraries, so importing into one library is your best bet. Here's what you should expect:

• Duplicates might appear during reimport. Apple Photos has a feature to detect and manage duplicates, but it's not perfect.
• Album structure and face tags might not carry over seamlessly.

If you want a more automated solution for merging libraries:

• Third-party Tools: Consider apps like PowerPhotos by Fat Cat Software, which are designed specifically for merging Apple Photos libraries while preserving metadata.

---

[RhetTbull]

The best way to do this is probably to use something like PowerPhotos.

If you want to use OSXPhotos to merge (using the osxphotos import command) as @ferid333 suggested, then I would do the following:

• Use --export-aae with the osxphotos export command. This exports the Apple adjustments data (non-destructive edits to your Photos) needed to reconstruct the edits upon import. If you drag a photo + it's AAE file into Photos, Photos recreates the edit non-destructively (that is so you can still "revert to original"). osxphotos import will also handle the AAE files appropriately.
• I would use --sidecar xmp instead of --sidecar json as the XMP files are more portable and usable by third-party apps like Lightroom or Digikam.

I will do some scripted operations on the files including changing some names

If possible, I would do any filename manipulation upon export using the very flexible osxphotos template language. E.g. use --directory and --filename to do the name changes. This allows OSXPhotos to maintain state of the files in the export database so that you can use the --exportdb option of the osxphotos import utility to match metadata to the exported files. Alternatively, if you do rename the files after export, you must ensure you rename any associated files such as the live video component, the AAE file if exported, and the edited file (which OSXPhotos will name IMG_1234_edited.jpeg where IMG_1234 is the name of the original file). If you do this then osxphotos import can still associate the metadata albeit not with the --exportdb option.

If using osxphotos import to re-import, you can use the --sidecar option to import metadata from your JSON or XMP sidecars. This will preserve keywords, title, caption/description, and favorite (if used with --favorite-rating). To preserve favorites when using sidecars you should also use --favorite-rating with osxphotos export. See osxphotos help export favorite.

The only metadata that cannot be restored from sidecars is albums and folders because these are not stored in the sidecar. That is the biggest advantage of using --exportdb (see osxphotos help import exportdb) to restore metadata but this requires the imported filenames be exactly the same as those exported. It is also possible to restore album structure from the exported directory structure if you used --directory "{folder_album}" with osxphotos export and using --album upon import. For example:

osxphotos import /Volumes/Photos/* --walk --album "{filepath.parent}" --relative-to "/Volumes/Photos" --split-folder "/"

Will import all photos in /Volumes/Photos using the parent folder as the folder/album path but stripping /Volumes/Photos from the path first and this, if used with --directory "{folder_album}", upon export will reconstruct the folder/album structure.

Using this combined with sidecar files will allow you to reimport everything into Photos while preserving metadata.

My goal is to have a set of OSXPhotos tools that allow you to merge libraries, copy or move assets between libraries, re-build a library etc. osxphotos import is close to being able to do part of this but still has a few issues. One of the big limitations (and I can't work around this in Photos) is that this doesn't work well with referenced libraries. If using osxphotos import you should do so only when "copy files to the Photos library" is enabled (this is the default).

See also #1659 for a discussion on some of these topics.

[ferid333]

That is great, Also please accept my response as an answer. Thanks

[RhetTbull]

@ferid333 no. You are using an LLM for engagement farming. Please don't do that on this repo.

[ferid333]

Also can we follow each other ?

[RhetTbull]

The best way to do this is probably to use something like PowerPhotos.

If you want to use OSXPhotos to merge (using the osxphotos import command) as @ferid333 suggested, then I would do the following:

• Use --export-aae with the osxphotos export command. This exports the Apple adjustments data (non-destructive edits to your Photos) needed to reconstruct the edits upon import. If you drag a photo + it's AAE file into Photos, Photos recreates the edit non-destructively (that is so you can still "revert to original"). osxphotos import will also handle the AAE files appropriately.
• I would use --sidecar xmp instead of --sidecar json as the XMP files are more portable and usable by third-party apps like Lightroom or Digikam.

I will do some scripted operations on the files including changing some names

If possible, I would do any filename manipulation upon export using the very flexible osxphotos template language. E.g. use --directory and --filename to do the name changes. This allows OSXPhotos to maintain state of the files in the export database so that you can use the --exportdb option of the osxphotos import utility to match metadata to the exported files. Alternatively, if you do rename the files after export, you must ensure you rename any associated files such as the live video component, the AAE file if exported, and the edited file (which OSXPhotos will name IMG_1234_edited.jpeg where IMG_1234 is the name of the original file). If you do this then osxphotos import can still associate the metadata albeit not with the --exportdb option.

If using osxphotos import to re-import, you can use the --sidecar option to import metadata from your JSON or XMP sidecars. This will preserve keywords, title, caption/description, and favorite (if used with --favorite-rating). To preserve favorites when using sidecars you should also use --favorite-rating with osxphotos export. See osxphotos help export favorite.

The only metadata that cannot be restored from sidecars is albums and folders because these are not stored in the sidecar. That is the biggest advantage of using --exportdb (see osxphotos help import exportdb) to restore metadata but this requires the imported filenames be exactly the same as those exported. It is also possible to restore album structure from the exported directory structure if you used --directory "{folder_album}" with osxphotos export and using --album upon import. For example:

osxphotos import /Volumes/Photos/* --walk --album "{filepath.parent}" --relative-to "/Volumes/Photos" --split-folder "/"

Will import all photos in /Volumes/Photos using the parent folder as the folder/album path but stripping /Volumes/Photos from the path first and this, if used with --directory "{folder_album}", upon export will reconstruct the folder/album structure.

Using this combined with sidecar files will allow you to reimport everything into Photos while preserving metadata.

My goal is to have a set of OSXPhotos tools that allow you to merge libraries, copy or move assets between libraries, re-build a library etc. osxphotos import is close to being able to do part of this but still has a few issues. One of the big limitations (and I can't work around this in Photos) is that this doesn't work well with referenced libraries. If using osxphotos import you should do so only when "copy files to the Photos library" is enabled (this is the default).

See also #1659 for a discussion on some of these topics.

[ZarK]

Thanks!

I’ll have to use regex to achieve my wanted rename to undo a weird naming scheme I implemented on an import years ago, so I think that would be outside what’s supported by the template system.

I do have PowerPhotos, so I’ll look at that as an option as well.

I currently have a hierarchy of both folders and albums - I think there is not any way to preserve which is which when importing from a file structure.

I will use your tips on the expert in any case to get as complete an export of metadata as possible.

I might attempt to manipulate the sqlite3 database directly myself based on a log of the renames. I can identify them by uuid through the sidecars, and should be able to handle the Apple NFD Unicode normalization as well.

I’ll have a look at the db implementation in osxphotos, but if it’s just a few tables like RKMaster and RKVersion I have to update then it seems possible.

[RhetTbull]

I currently have a hierarchy of both folders and albums - I think there is not any way to preserve which is which when importing from a file structure.

osxphotos import can preserve the folder/album structure. See my previous reply. I think Photos has a way to preserve these as well by dragging in the folders but I've not experimented with this.

I’ll have a look at the db implementation in osxphotos, but if it’s just a few tables like RKMaster and RKVersion I have to update then it seems possible.

RKMaster, etc. are the names used by Photos 4.0 (pre-Catalina) and older in the Photos database. These are not used in modern versions of Photos. The export database created by OSXPhotos is fairly simple with a handful of tables. You would need to modify export_data.filepath_normalized to match your new naming config (the path is normalized (all lower case)). For internal consistency I would also modify export_data.filepath which is the non-normalized path (mixed case).