Veeam and Tape IV - Cataloging a tape with a huge number of files
In the first two parts of this series, we have seen how to move a tape library with data from one VBR server to another.
(See them either here on this site or at Veeam Community)
and how to move a tape with data into another media pool.
In this part we will see how to catalog a tape with a huge number of sessions and files on it.
In this case the catalog process takes a huge amount of time and the amount of used RAM by this process increases more and more.
We had to stop the process because the catalog process used more than 80% of the available 160 GB RAM of the server – combined with the VBR Server, the Veeam database and the OS 98% RAM were used… It is not the biggest server you can imagine, but it is more than enough under normal circumstances.
Now, what can we do about this? All other processes are stopped. There is nothing we can stop anymore to get more RAM.
One try was to upgrade the VBR server from version 11 to 11a. But this did not change anything regarding this situation.
So, now it was time to involve Veeam support.
After some log analysis it became clear that the existing catalog process has a problem with many sessions and files on one tape.
Veeam created a hotfix for VBR V11a, unfortunately this fix did not solve the problem. The same behavior occurred.
After some more analysis a second hotfix for VBR V11a P20211211 was provided. This hotfix changed the behavior of the catalog process to not use the memory for the catalog data but a file in the Windows Temp Folder.
And this caused a disk full situation after around 6 hours runtime of the catalog process….
We configured the environment variables for the temp directories from C: to E:. Drive E: is an Veeam Repository drive which is unused now and has around 20 TB free disk space. I hope 20 TB will be enough space for this…
Follow the process described in the next picture to configure the environment variables for the Temp folders.
Please change both TMP and TEMP locations.
A reboot after these changes is strongly recommended.
Let’s try again….
This time the data is written into the TMP directory on drive E. OK, so we should have enough diskspace this time.
After several hours the memory usage is not significant and the diskspace usage is moderate. Looks good this time. Let’s see how long it takes to process all the catalog files on this tape…
After one and a half days (!) runtime of the catalog process another situation has developed. The catalog process has written two files to TMP directory with 144 GB in size. And the RAM usage is at 99% again. This time it is not the Veeam Agent process, but the Veeam Backup Manager process consuming the RAM…
So… this version of the hotfix does not solve our problem, too….
Next round for Veeam Support to investigate. 😊 Just uploaded the logs....
Question for me is now: How big can a catalog of a tape get? I know that on theses tapes is data from many sessions. But every single catalog file should be very small. And... the catalog of all these tapes were in the database of the old VBR server. This database was less than 10GB in size. Now the catalog process for ONE tape has more than 288GB temp data and 145 GB RAM used. What is all this data????
To be continued....
Last updated