Using a WD elements 3tb drive to backup my photos on UBUNTU

Tools

1. md5deep
2. cut
3. sdiff

Actions

format the WD drive

Using Gparted format to EXT4

Mount the Western Digital Drive.

Edit fstab and add this line for the UUID

UUID=186ddeed-f1da-47cd-9b78-78336237517b /mnt/WDBOOK     ext4    rw 0       1

sgoggin@sgoggin-G31M-ES2L:~$ blkid

/dev/sda1: UUID="6c4b86a7-8336-47ef-a0a2-68634de48875" TYPE="ext4" 

/dev/sdc1: UUID="057713e2-f8eb-4fb7-a424-4be170d5672d" TYPE="ext4" 

/dev/sdc5: UUID="1dff8337-1ec2-4e10-b0bb-19b2f31ff9de" TYPE="swap" 

/dev/sdb1: LABEL="RawPhotos" UUID="120920e7-1780-47d5-b320-21d5d51e1865" TYPE="ext4" 

sgoggin@sgoggin-G31M-ES2L:~$ sudo blkid

/dev/sda1: UUID="6c4b86a7-8336-47ef-a0a2-68634de48875" TYPE="ext4" 

/dev/sdc1: UUID="057713e2-f8eb-4fb7-a424-4be170d5672d" TYPE="ext4" 

/dev/sdc5: UUID="1dff8337-1ec2-4e10-b0bb-19b2f31ff9de" TYPE="swap" 

/dev/sdb1: LABEL="RawPhotos" UUID="120920e7-1780-47d5-b320-21d5d51e1865" TYPE="ext4" 

/dev/sdh1: UUID="186ddeed-f1da-47cd-9b78-78336237517b" TYPE="ext4" 

Now use rsync to backup the photos

Now generate a md5 checksum using md5deep

•    Navigate to the root directory 
• Run MD5DEEP
  md5deep -rel

Compare

• sdiff -s A.log B.log
• the -s suppresses common lines

Need to massage the files a little because I forgot the -rel

• cat md5deepRawPhotos.log | cut -c -34,35- > md5deepRawPhotosTrim.log
• Removes character in position 34 to 35 from every line in the log.

Canon HF200 Linux transfer script

#!/bin/bash
#Startup settings
date=`date "+%Y-%m-%dT%H:%M:%S"`
ddir=`date "+%Y-%m-%d"`
destdir="/home/sgoggin/TransferAREA"
logfile='/tmp/TransferPhotosCardReader'

echo "Started TransferPhotosCardReader on $date" | tee $logfile
echo " Dest Dir is $destdir" | tee $logfile

mkdir -p $destdir

# Step 1a - Move the existing Transfer Area into a dated copy
# mv $destdir $destdir_$ddir

# Step 1 - Transfer files from card to Transfer AREA
cd $destdir;

# Set path for Video Camera
if [ -d /media/disk/PRIVATE/AVCHD/BDMV ]; then sourcedir="/media/disk/PRIVATE/AVCHD/BDMV";

echo " Copying HD Video from source Dir $sourcedir " | tee $logfile
echo " Transferring file ..."
find $sourcedir -type f -exec rsync -av {} $destdir \; | tee $logfile
# Step 1 - Rename imapges that are still in canon format
cd $destdir;
for f in $( ls | grep -i MTS);
do
echo "Processing $f" | tee $logfile
echo " Checking to see if "${f/.MTS}.CPI" exists" | tee $logfile
FileName=`ls -l --time-style=+%Y%m%d-%H%M%S $f |cut -d' ' -f6`
echo " Timestamp is $FileName"
mv -v $f $FileName.MTS

if [ -e "${f/.MTS}.CPI" ]; then
echo " moving CPI file $f.CPI"
mv -v ${f/.MTS}.CPI $FileName.CPI
fi;
done;

fi;

# Retrieve photos from the plugged in camera
# gphoto2 -P
if [ -d /media/CANON_DC ]; then sourcedir="/media/CANON_DC/DCIM/"; fi;
# Set path for Video Camera
if [ -d /media/disk ]; then sourcedir="/media/disk/DCIM/"; fi;


if [ -d /media/EOS_DIGITAL ]; then sourcedir="/media/EOS_DIGITAL"; fi;
echo " Source Dir is $sourcedir " | tee $logfile

echo " Transferring file ..."
find $sourcedir -type f -exec rsync -av {} $destdir \; | tee $logfile

# Step 1 - Rename imapges that are still in canon format
cd $destdir;
for f in $( ls | grep -i jpg);
do
echo "Processing $f" | tee $logfile
echo " Checking to see if "img${f/.jpg}.cr2" exists" | tee $logfile
if [ -e "img${f/_r.jpg}.cr2" ]; then
echo " [Originated from a CR2 therefore skipping]" | tee $logfile
else
echo " No it does not exist, renaming" | tee $logfile
# Rename the jpegs
jhead -ft -nf%Y%m%d-%H%M%S $f | tee $logfile
fi;
done;

# Ensure all files are in lower case
rename 'y/A-Z/a-z/' *

# Step 2 - Rename the CR2's
exiv2 -Ft rename img*2 | tee $logfile
exiv2 -Ft rename crw*crw | tee $logfile

# Rename the AVI files
/usr/local/bin/RenameAVI *avi | tee $logfile

# Step3 - Now sort the files into a tree based in the file name.
for ftm in 200*; do
echo ${ftm:0:4}/${ftm:4:2}/${ftm:6:2}/$ftm
echo "Processing $ftm into ${ftm:0:4}/${ftm:4:2}/${ftm:6:2}/$ftm" | tee $logfile
if [ ! -d ${ftm:0:4}/${ftm:4:2}/${ftm:6:2}/$ftm ]; then
echo "Creating ${ftm:0:4}/${ftm:4:2}/${ftm:6:2}/$ftm" | tee $logfile
mkdir -v -p ${ftm:0:4}/${ftm:4:2}/${ftm:6:2}
fi
mv -i -b $ftm ${ftm:0:4}/${ftm:4:2}/${ftm:6:2}/$ftm
done



tree -D

Time Lapse Video - Command line

mencoder "mf://*.JPG" -vf scale=640:480 -o timelapse.avi -of lavf -ovc lavc -lavcopts vcodec=mjpeg -lavfopts format=avi -fps xx

sgoggin@BlueBox:~/TransferAREA/2009/08/02$ mencoder "mf://*.jpg" -vf scale=1920:1080 -o timelapse.avi -of lavf -ovc lavc -lavcopts vcodec=mjpeg -lavfopts format=avi -fps 10
MEncoder 2:1.0~rc2-0ubuntu19 (C) 2000-2007 MPlayer Team
CPU: Intel(R) Celeron(R) CPU 3.06GHz (Family: 15, Model: 4, Stepping: 9)
CPUflags: Type: 15 MMX: 1 MMX2: 1 3DNow: 0 3DNow2: 0 SSE: 1 SSE2: 1
Compiled with runtime CPU detection.
success: format: 16 data: 0x0 - 0x0
MF file format detected.
[mf] search expr: *.jpg
[mf] number of files: 107 (428)
[demux_mf] file type was not set! trying 'type=jpg'...
VIDEO: [IJPG] 0x0 24bpp 25.000 fps 0.0 kbps ( 0.0 kbyte/s)
[V] filefmt:16 fourcc:0x47504A49 size:0x0 fps:25.00 ftime:=0.0400
Input fps will be interpreted as 10.00 instead.
** MUXER_LAVF *****************************************************************
REMEMBER: MEncoder's libavformat muxing is presently broken and can generate
INCORRECT files in the presence of B frames. Moreover, due to bugs MPlayer
will play these INCORRECT files as if nothing were wrong!
*******************************************************************************
OK, exit
Opening video filter: [expand osd=1]
Expand: -1 x -1, -1 ; -1, osd: 1, aspect: 0.000000, round: 1
Opening video filter: [scale w=1920 h=1080]
==========================================================================
Opening video decoder: [ffmpeg] FFmpeg's libavcodec codec family
Selected video codec: [ffmjpeg] vfm: ffmpeg (FFmpeg MJPEG decoder)
==========================================================================
VDec: vo config request - 2592 x 1456 (preferred colorspace: Planar 422P)
VDec: using Planar 422P as output csp (no 1)
Movie-Aspect is undefined - no prescaling applied.
SwScaler: reducing / aligning filtersize 7 -> 8
SwScaler: reducing / aligning filtersize 7 -> 8
SwScaler: reducing / aligning filtersize 7 -> 6
SwScaler: reducing / aligning filtersize 12 -> 11
[swscaler @ 0x883d3d0]SwScaler: BICUBIC scaler, from yuv422p to yuv420p using MMX2
[swscaler @ 0x883d3d0]SwScaler: using 8-tap MMX scaler for horizontal luminance scaling
[swscaler @ 0x883d3d0]SwScaler: using 8-tap MMX scaler for horizontal chrominance scaling
[swscaler @ 0x883d3d0]SwScaler: using n-tap MMX scaler for vertical scaling (YV12 like)
[swscaler @ 0x883d3d0]SwScaler: 2592x1456 -> 1920x1080
videocodec: libavcodec (1920x1080 fourcc=47504a4d [MJPG])
New_Face failed. Maybe the font path is wrong.
Please supply the text font file (~/.mplayer/subfont.ttf).
subtitle font: load_sub_face failed.
VIDEO CODEC ID: 8
Writing header...
Pos: 10.7s 107f (100%) 4.43fps Trem: 0min 17mb A-V:0.000 [13969:0]
Flushing video frames.
Writing index...

Video stream: 13969.006 kbit/s (1746125 B/s) size: 18683546 bytes 10.700 secs 107 frames
sgoggin@BlueBox:~/TransferAREA/2009/08/02$

Why Western Digitals New 750GB green drive is really brown.

I just bought one of the new 750GB WD drives with the new "Green" technology. Basically that means that the drive spins down within a few seconds of access. That's all OK - but the problem I have is that under UBUNTU - I am getting a SATA fault indicating a frozen interface when it spins up.

these are appearing in my syslog

[ 3541.483329] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x2 frozen
[ 3541.483422] ata4.00: cmd b0/da:00:00:4f:c2/00:00:00:00:00/00 tag 0 cdb 0x0 data 0
[ 3541.483425] res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 3542.182473] ata4: soft resetting port


I also notice that the hard drives load cycle count (there is an upper limit of something like 300k load cycles before the drive is rooted) is increase dramatically, for example after 10 mins in my server its at


193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 57

I'm running this kernel
2.6.20-15-server
Which I think was ubuntu feisty.


-- Further notes - The error was actually raised by smartctl when it failed to wake the sleepy drive.   Looked into it a bit further - the frozen SATa link appears to be caused by smarton tools not recognising that the drive has spun down. (http://www.mail-archive.com/linux-id.../msg11092.html) .



However the large increase in Load cycles is cause by the green power saving in the drive - and this cannot be disabled. Depending on the way the server uses the drive then this number can increase very quickly - and at a largish number (perhaps 300k) the drive will be considered end of life and it's health status will show failed.


-- So I think I can risk using the 750GB to replace the failed 500GB SATA and perhaps one of the older IDE drives - there's one in there with nearly 20k hours on it.

Configuring an Elisabuntu Client

Networking

/etc/fstab

//10.1.1.100/ActiveData /mnt/ActiveData smbfs credentials=/home/sgoggin/.smbcredentials,dmask=775,gid=1001 0 0
//10.1.1.100/nzb /mnt/NZB smbfs credentials=/home/sgoggin/.smbcredentials,dmask=775,gid=1001 0 0

/etc/network/interfaces /interfaces

auto lo
iface lo inet loopback


iface wlan0 inet static
address 10.1.1.67
netmask 255.0.0.0
gateway 10.1.1.1
wireless-essid GOGGS

Script to automagically sort files.

Sources
http://ubuntuforums.org/showthread.php?t=526437

Original Script

gphoto2 -P # The -P option gets all your files from your
# camera, both pictures and videos

JPGS="*.JPG"
AVIS="*.AVI"

# Creating folders and moving JPG files

for file in $JPGS
do
mkdir /home///"`date -r $file +%Y%m`"
mv $file /home///"`date -r $file +%Y%m`"
done

# Creating folders and moving AVI files

for file in $AVIS
do
mkdir /home///"`date -r $file +%Y%m`"
mv $file /home///"`date -r $file +%Y%m`"
done

echo "Done, Boss!"

Ok

CFV...

CFV Links
md5deep
cfv

CFV Sample command lines

cfv -rr -t md5 -f test.cfv *
cfv -r -t md5 -C -f md5Checksum.md5 -p /srv/ActiveData/ *
Create Individual md5's

sgoggin@Orangutang:/srv/ActiveData$ sudo nohup cfv -v -r -C -f Checksum.md5 *

Create Master Md5

sgoggin@Orangutang:/srv/ActiveData$ sudo nohup cfv -v -rr -C -f

SummaryChecksum.md5 *

TBC

Objectives
Photos - Create checksum files once
- Only regenerate checksum if there are new files in a directory and no failed CRC
- If failed CRC then send report to sysop and do nothing.
- Store the CRC files in each directory and let these be copied to other backups.
- Backups - As part of the backups execute checksums on each directory and send results

Plan - For Photo Tree.
1 - Once a week run the master checksum and send failures and new file in a report
2 -After that has run check the output for any failed CRC - If any then DO NOTHING
3 - Check report and if there are new files then regenerate checksums using -r option
4 - Once a month - check report and if there are new files and no CRC then regenerate -rr
5 - Send report