User:Quantumgardener/my IMatch

From ptWiki

How I moved to IMatch] has been by far one of the most popular pages on my website. It details how I categorised my images and managed both photos and video within the one system. The original article was written on March 5, 2003 and this article reflects what I have learnt and implemented since then.

The Journey

My use of IMatch has been through three distinct phases. The first phase, lasting some four years, was swamped by a desire to categorise every detail of every photo. There were several iterations, the last culminating with an attempt to place everything in a category structure reflecting the Thesaurus for Graphic Materials. This brave attempt suffered because I was categorising photos slower than I was accumulating them.

iView MediaPro was a complete departure from IMatch and was my second phase of digital asset management. I gave up on IMatch for a time and had an affair with another piece of software. iView is a simpler tool to use than IMatch and simplicity in software can often come at a price — you find you can't do all that you want to. In my case this was a blessing in disguise. Trying to bend iView to my will made me realise that I didn't need every photo categorised every which way. After all, I never searched that way. My new mantra became, "only tag by what you search for". iView has its plusses. There is a nice interface which makes it very easy to add images to categories.

In journeying with iView I met some fellow travellers. One became very important to my understanding of how to manage digital assets. Peter Krough is the author of "The DAM Book: Digital Asset Management for Photographers". The associated forum is a learning tool which supports the book very well. I came to understand that cataloging and management of image files were separate things. This article details my learning and the application of my iView experience to IMatch. From this point onwards I'll use the term asset to refer to generic digital assets such as photos or video, and only use the specific terms of photo or video when it makes sense to do so.

When I returned to IMatch it was with a vastly superior workflow. I firstly deleted every category that I wasn't going to use again. That was quite a few. Things are simpler now. My IMatch categories mimic those I used in iView. I have Date Finder, Place Finder, People, Source Device, File Type, Creator and Catalog Sets. To those I've added Physical Location and ZZ_Quality. The use and structure of each category, including the tools within IMatch that I use to manage them is described below. They are the way I find things and I believe they will be common to most people.

My workflow

My workflow is based on the process outlined by Peter Krough in his book mentioned above. I'll outline it here and expand further where I think it will be helpful. It's not possible to describe everything in a purely linear form so you may have to read back and forth to understand fully.

Digital camera workflow

Take photo(s) with a digital camera (I currently have a choice of a Canon EOS 400D (Canon RAW) or Canon Powershot A510 (jpeg)). It does not matter if I span multiple dates.
Remove card from camera, insert into card reader.
Download from card using Downloader Pro into my directory structure. This completes several IPTC fields as it downloads including copyright.
View image using Adobe Bridge. This generates thumbnails directly from the image and not the built-in thumbnail. It is a good check of image integrity. IMatch does the same on import, however it is easier to make adjustments to the RAW file in Bridge than IMatch. If all ok, return card to camera and format in-camera. I should back up before I do this but I usually don't.
Adjust RAW images using Adobe Camera Raw.
Convert adjusted RAW images to DNG.
Import new images into IMatch
Categorise (part by script, part by hand)
Move from my working directory to my current archive directory. If archive directory is now full (~4.4GB) then burn to DVD and create next archive directory.
Export for flickr
1. Batch convert to JPEG
2. Populate IPTC from catalog into JPEG file
3. Add GPS information to JPEG file
4. Upload to flickr
5. Assign to sets in flickr if I couldn't do that on the upload

Digital video workflow

Take digital video
Download from tape using scenalyzer, splitting each file by date/time into my working directory.
Import new video into IMatch
Categorise (part by script, part by hand)
Move from my working directory to my current archive directory. If archive directory is now full (~4.4GB) then burn to DVD and create next archive directory.

Scanned print workflow

Scan images into my working directory
Import new images into IMatch
Categorise (part by script, part by hand)
Move from my working directory to my current archive directory. If archive directory is now full (~4.4GB) then burn to DVD and create next archive directory.
Export for flickr (see Digital camera workflow above)

Managing the digital files

Managing files is as important as cataloging. I work with a bucket system. Each bucket is approx. 4.4GB, convenient to burn to a DVD.

Hard drives

I have internal and external SATA drives. Each night Acronis backup does a backup of my catalogue file and when I've been working I used Genie Backup Pro to duplicates (not archive backup) the internal drive to the external in case of fire grab this on the way out drive.

Hard drive directory structure

A simple structure. For more detail refer to the DAM referneces mentioned above.

Mediabank (internal drive)
- archive — archive directories, each 1xDVD in size. When full, create and move to the next. These are my buckets.
  - DRV_001
  - RAW_001
  - RAW_002
  - RAW_003
  - ...
- catalog — storage for IMatch's catalog file
- working — asset directories, by date or reel (video). Anything here has not yet been fully categorised. One it has been fully categorised it is moved to the latest open archive directory.
  - 2007-06-16
  - 2007-06-17
  - ...
  - Reel 2007-001
  - ...

When I do a backup following work, the external drive becomes an exact copy of the first. My IMatch catalog files are backed up each night as part of my normal file system backup as well.

Offsite media

Once I can afford another external sata drive I'll have 2 external drives for backup and they will be rotated. Until then I'll rely on DVD (which will continue).

As each archive directory fills to 4.4GB I burn two copies to DVD. One stays at home, the other goes to my Dad's house. The DVD's are titled RAW_###_YYYYMMDD where ### is the sequence I'm up to (always three digits so I have space for 999), YYYYMMDD are year, month and day respectively. The same date is entered into IMatch's description field for the original folder.

DVDs are burnt to two separate drives in parallel. After burning they are run through Adobe Bridge which generates an image from the file itself (not any embedded thumbnails). If it can do that, the integrity of the file on DVD is fine.

Categories

IMatch has a very powerful category engine which is supported throughout the application. I use categories over searching as my primary means of finding photos and video that I've cataloged. I have few top level categories. The total of assets in each category is displayed on my system as I find it helpful to manage the categories when I know if there are assets assigned to them or not.

@All

Catalog Sets

Catalog Sets are convenient groupings of assets. They cross multiple of the categories below. Take Christmas for a simple example. It typically spans multiple days and a Christmas catalog set gives me a way to bring all Christmas 2006 photos together without tricks. Assets are manually assigned to catalog sets but as they are usually groups of photos it's not too bad.

I'll show a generic structure to explain how they work for me. Words after the '—' are commentary.

@All
- Catalog Sets
  - Events — all events such as birthdays, weddings, engagements, holidays
    - birthdays
      - family — non-immediate family i.e., neither myself, my wife, nor my children.
        family member A
        year - age'th — e.g., 1998 - 28th
        year - age'th — e.g., 1999 - 29th
        ...
        
        friends
        friend A — birthdays of friend, structure same as for family.
        friend B
        ...
        
        immediate family A — myself, my wife or my children are immediate family members
        year - age'th — e.g., 2002 - birth
        year - age'th — e.g., 2003 - 1st
        ...
    - engagements
      - year - engaged couple — e.g., 2002 - Jamie and Janet Dawson
      - ...
    - holidays
      - year - holiday name — e.g., 2001 - Murray River houseboat
      - year - holiday name — e.g., 2002 - London
      - christmas
        year — e.g., 2005
        year — e.g., 2006
        ...
      - easter
        year — e.g., 2005
        year — e.g., 2006
        ...
      - picnics & daytrips — effectively short holidays
        year - picnic name — e.g., 2001 - Werribee Zoo with the Clarks
        ...
    - school
      - year - event — e.g, 2006 - Caitlin's graduation
      - ...
    - weddings
      - year - wedded couple — e.g., 1996 - David and Wendy Buchan
      - year - wedded couple — e.g., 2003 - Jamie and Janet Dawson
      - ...
  - Projects — contains one sub-category for each project. This lets me find all assets I've used on a home video DVD together in one place.
    - DVD Projects
      - ...
    - flickr — special category for photos uploaded to flickr. Each sub-cateogory matches a flickr set. and gives me some redundancy if flickr goes down.
      - flickr set A
      - ...

Creator

This is the person who took the photo. It is a single level list with the photographer's name. Photos from friends and family are typically identified by both names as I can't be sure which person actually took the photo. There is one special entry called Unknown'. If I look at an asset, and I can't truly identify who took it, then the creator is unknown. This approach is superior to leaving them uncategorised because it designates that I have thought about this particular asset.

@All
- Creator
  - David & Wendy Buchan
  - Nick and Mary Thompson
  - ...
  - Unknown
- ...

An asset can only belong to one category in the Creator tree.

Date Finder

The date I believe the asset was imaged (photo taken, video shot). It is usually the EXIF date but not always as sometimes that may be wrong or unavailable (note: if the EXIF date is wrong and stored in a modifiable format I will correct it but I can't always do so).

Dates are recognised by the year, month and day. I can get all assets from a given year, a given month or a given day.

@All
- ...
- Date Finder
  - 1958
  - ...
  - 2006
  - 2007
    - 01 January
    - 02 February
      - 10 Saturday
      - 15 Thursday
      - ...
    - 03 March
    - ...
- ...

This field is populated by a script that finds the best date, creates the categories and finds the right day for each date in the month. Assets can only belong to one day.

If I know the month, but not the day the asset is categorised as Month.00 Unknown. If I know the year, but not the month the asset is categories as Year.00 Unknown.00 Unknown. There are other combinations but all such instances are rare.

File Type

I no longer track file type. I found it was something I rarely referred to and if needed I could use the built-in search functionality.

The file type of the asset.

@All
- ...
- File Type
  - Audio Video Interleave File
  - Canon RAW
  - Digital Negative RAW
  - JPEG
  - Mpeg Video File
  - MPEG-4
  - PDF
  - Photoshop
  - TIFF
- ...

This field is populated by a script that determines the file type from the file extension.

People

The people in the image, by name. Included are characters such as Mickey Mouse or Santa. People is a single level list and this contrasts to the complex tree structure I had outlined in How I Moved to IMatch.

Assets can belong to any number of People categories.

Physical Location

Relating primarily to scans or video, physical location is a way for me to mark the original photo album or tape.

Assets can only belong to one Physical Location category.

Place Finder

Place Finder is either the location where an image was captured, or the location of the thing that is captured ie. a photo of a sporting event at the Melbourne Cricket Ground (MCG) would be categorised MCG, as would an image of the MCG taken from the Rialto Towers observation deck.

Place Finder is structured by Country.Province (or State).City.Location.Sublocation.SubSubLocation... as far down as I think I need to go. It can be quite detailed to give me the preciseness of location that I require.

@All
- ...
- Place Finder
  - Australia
    - Queensland
    - Victoria
      - Melbourne
        Melbourne Cricket Ground
        Rialto Towers
      - Port Fairy
  - Great Britain
    - England
      - London
        Greater London
        Camden
        City of London
        Tower Bridge
        ...
- ...

GPS population

I use GeoSetter to set GPS locations in my photos. There is nothing I have found as fast, reliable or easy to use.

Source Device

Not much to describe with Source Device. It is simply the original equipment which took the image.

Assets can only belong to one Source Device category.

ZZ_Quality

ZZ_Quality is a special category created to assist with the assignment of assets to the other categories. It has several sub-categories, each with a formula of the form:

("@All" NOT "Creator") AND "ZZ_Quality.Current folder"

ZZ_Quality looks like:

@All
- ...
- ZZ_Quality
  - Current folder — assigned to the folder on the hard drive I'm working on
  - NOT creator — equals Current folder and NOT Creator to give assets not assigned a creator (all must have one, even if it is unknown)
  - NOT date finder — equals Current folder and NOT Date Finder to give assets not assigned a date (all must have one)
  - NOT file type — equals Current folder and NOT File Type to give assets not assigned a file type(all must have one)
  - NOT people — equals Current folder and NOT Peopler to give assets not assigned people (optional, but a useful check)
  - NOT physical location — equals Current folder and NOT Physical Location to give assets not assigned a physical location (optional, but a useful check)
  - NOT place finder — equals Current folder and NOT Place Finder to give assets not assigned a place (all must have one)
  - NOT source device — equals Current folder and NOT Source Device to give assets not assigned a source device (all must have one)

For each directory I'm processing I reset Current folder and the rest of the categories update automatically.

Security and privacy

A quick word of caution for those of you who upload images to the web, especially if you populate them with IPTC or GPS data as I do. If you want to keep something secret (i.e., you don't want it public on the web where you children go to school or live) then don't put this information into your photos before you upload them.

Take advantage also of the security features of the site you have your photos on. I use flickr's friends and family permissions extensively and apart from one photo the public cannot see any photos of my children.