Family Media Migration 2019
I documented a few months ago how we’d been keeping our family photos and short videos in Apple iPhotos and then Photos for years. My issue was that when I actually went to look in the library bundle at the folder structure it was all over the place - different folder structures and naming conventions, whole libraries and their sub directories imported into a single date directory and so on. Basically it seems like they stored photos based on import to library date, rather than the date the photo was taken, and that might be fine for some, but it wasn’t really what I was looking for. There was also that phase where folders were named by event.
TL;DR :
I moved all my photos from the Apple Photos ecosystem/‘mess’ to a simple open directory structure (on a Windows machine, but that’s not really relevant).
What I did
I’m not saying the following is the best way, in fact in retrospect, I know it isn’t, so I’ll attempt to note along the way which ways occurred to me to improve it.
I went into my Photos bundle and found the Masters folder, this is where the current application keeps all the originals. It has other folders which may have versions of the files, but this is where the untouched originals go. I think in the iPhoto days this folder was called Originals. I then made a complete copy of it. You can do this via a GUI copy, or a command line cp -a or rsync, you know what I mean, to make a complete copy of it (and check that copy with diff if you feel the need). Yes, this is the old mantra of having a backup in case things go Bad (TM).
Next I made a spreadsheet to give me a quick look at how much work was ahead by using the excellent EXIFtool. You don’t have to do this at all, but I found it really gave me a view on how much work was ahead. If you have a huge library (tens of thousands to hundreds of thousands of files) you might want to go one directory layer down to ‘years’ for this, or just…wait longer. It will finish.
It was doing this which highlighted the issue I had, when I found a library import I’d done a few years before and which looked seamless in the Photos browser but which was actually several hundred photos from across multiple years, all dumped into one sub-directory of the date I imported it. It also reminded me how awful the kids LEGO cameras were for EXIF data.
There are a few ways to automate all this, but I went the hard way - image by image, mainly as I knew from the spreadsheet I had a pile of potential duplicates, and I wanted to see what if any differences there were. My main tool on the Mac then was GraphicConverter because it’s a file browser, and allows decent EXIF editing control as needed within the app. On Windows, I’d use IrfanView or digiKam.
My plan then, was to store all of our photos in a ‘Year > Month > Day’ structure, and allow myself the odd luxury or a ‘Year > Month > Special Event’ system if really needed to, all based on the day the photo was taken.
So off I went to create a whole load of directories for this. Next tip then - and so obvious - is that you can script this directory creation, please do it! I worked around and added to the one I had - it likely didn’t save me any time.
I started going through the photos, with a lot of delays for reminiscing whilst checking them out instead of just looking at metadata. Most of the work was simple and easy - moving photos from wherever they were, to the correct creation date folder. Some photos needed a little EXIF work, such as for where the date was wrong or corrupted, losslessly rotating images which were the wrong orientation which was more common on older images, and adding some comments.
You could script some of this too - it would likely have been quicker to get a script to dump files by EXIF data into folders and then review, but I wasn’t in a hurry and for me it was a chance to look at some memories and cull a bunch of near-identical images and clean things out.
Getting to Normal
This left me with my pristine directory structure on the Mac. I then rsync copied it to my local archive server, and then the same down to my Windows machine, before deleting the Mac versions. I also made sure Spideroak was seeing it correctly for encrypted internet backup in its new home! Just to side track on backups then, this ‘family media’ folder is now on a Windows SSD, a local rsync copy to another drive, then a weekly Robocopy to the archive server, which itself has a second internal storage drive which gets rsync’d every week. That’s four local copies and one internet copy.
The Future
As I’m not using a management tool by default, importing photos is different now. I’m dumping into an import folder, then assigning them out from there. It’s manual, but at least I know it’s correct. I’m testing digiKam as a management system like Lightroom and such, as it’s happy for me to manage my own structure. I think Photos also allows this, but the structure was already too broken and I had to move away from the Mac also because its CPU was struggling with 1440p video and couldn’t touch HEVC video. Along with needing to fix the file structure, it seemed better to bite the bullet.
This new structure also means I can now merge in some old camcorder footage - in .dv format and an .mkv copy - as well as a lot of GoPro footage which was just too big or HEVC to shove in to Photos, as Photos or the underlying Mac just didn’t like the format. I should say then that this wasn’t part of wanting to move to Windows, it really was just for the folder structure, and just my Windows machine is the one with the current horsepower for the job.
Possible Risks and Issues
The main risk of an open folder idea is that if someone makes a change to a photo in say GIMP or Photoshop without saving a copy, then the ‘original’ is not protected. My plan then is to use something like DigiKam for management since it’s opensource and cross-platform as a management tool. I’ve started playing with it already, and it seems easily powerful enough for my needs, and plenty more besides. I’ve already made headway into it, enough to find some more duplicates using it’s fingerprinting technology, and for editing various metadata.