Mirror Job backups up files at random

[MichaelA]

Hi.

The below problem already existed in Backup, now also confirmed in Reflector.

When running a mirror job from a NAS (Linux Samba) to a windows client for local mirror copy, the job selects folders -seemingly at random- which it deems needed to copy despite source and target file being the same with same name and datestamps. Even if the files were not copied in the previous backup run and there has been no change or access at all to the source and destination files.

I am using a read only account on the NAS side and explicitly unchecked using attributes, so source and dest filename/date/time stamps should be the criteria for fresh copy.

This does not work and it is unpredictably random. I have to note that this is a large file set; 670K+ files in 15TB+ of data. If effectively backups up TB's and TB's and more TB's every time it runs without the need to mirror it.

Any direction in this is appreciated and i can be contacted for any details required, testing or debugging.
Greetings from Deep, Dark, Africa and keep up the great work.

[cobian]

Archive attributes will never work with a NAS source so this is good that this is disabled. The way it works is:
* The program checks the existence of the file on the destination. If it doesn't exist, the the file will be copied.
* If the file exists, then the timestamps will be compared. If the timestamp on the source is newer than the one on the destination, the copy will be done.
* The timestamp of the source will be copied then to the destination.

It's quite an easy operation, so there is no much that could go wrong here. If Options-Engine-Copy timestamps selected?

[MichaelA]

Thanks for the swift feedback.
Agreed. Very straightforward, yet it keeps on copying same files over and over.

Attached the job info, engine settings and file comparison of one of the files.
I get the feeling I am missing something very basic here...

[MichaelA]

Second half of the files.

[cobian]

But your task is Full and not incremental (?)!

[MichaelA]

Ah, so the meaning in mirror job is thus. Understood. Actually makes 100% sense. I knew I was doing something stupid wrong.

After changing this the 'smaller' 350 GB job works now as intended. No more wrong copies.

The 14 TB job however still has significant problems. It is now in incremental mode with archive bit use off and archive bit set off; it now correctly skips large parts of the backup set. To test I ran it twice now for about 10% of the files, but it still selects files at random for recopy (in those 10%). Usually on a per folder basis?

Attached a file that the mirror job (incremental, no archive bit set, no archive bit use) walked over 3 hours ago and when rerunning the job went and did the file again. I do not know if it copied on the first run, but it should have definitively skipped it on the second run.

The properties of the file(s) were taken with job paused a few files before, so they are from just before the file was passed and copied when unpaused. The actual copy process (and file count) also included in screenshot.

This is not the first file. It skips many before it gets to this one, but also copies many others in the same set. Files i have explicitly seen it copy in the runs before.

Any further thoughts? Is it possible the size / number of files, perhaps path name length has an influence?

[cobian]

Hmm, no. I don't thing the length of the file name is the problem. Nor the number of files either. I would try to create a task with only that file as a source to try to isolate the problem and see if the problem persists. If it does, then there is something about that file....

[MichaelA]

Update:

(you said the file may be cause; it were many, many files where it was doing this, the one I captured was an example)

After another test confirmed same fault I let the entire 14 TB set run. When I returned several hours later the app had crashed. It was no longer there/visible from the engine and could not start it either. I did not reboot the PC but using the UI tools I reinstalled as service, then application again and then it started again.

The job now immediately runs correctly in the first run. Straight away skipped over 80-90% of the files, which it should have done from day one and all seems now ok.

Looks like something was not updating settings from the UI to Engine (the change from full to differential perhaps), or something got confused enough to do things differently. Either way, it looks now ok.

I will complete a few large jobs but seems the problem is resolved by the application restart.

[rotbus197142]

Just for the record - I'm seeing the same problem.

With 'Full backup' selected, a small percentage of the 80GB I'm backing up is backed up every time I run the task. The majority is not. Engine > Copy timestamps checked.

I can't see anything unique or unusual about the files that are being repeatedly backup up, and they are not being updated between task runs.

Switching to 'Incremental' solves the problem.


EDIT - the archive attr is being cleared on the 'problem' files.

[cobian]

Perhaps your source contains Reparse points.