MVS server backups

Last updated 11/29/2023

Overview:

MVS server is backed daily up to tape.

Neil-PC-7ro external usb drive I: holds the simple file copy backups from  sabrina, neil-pc-7pro, mobiecast, phone.

All other backups are kept on Google Drive or OneDrive via neil-pc-7pro or the Hosted client 2008 server. 

We even backup OneDrive and Google Drive to a USB drive monthly to protect against ransom-ware attacks.

 

Routine backup procedures

Our routine backup procedures show who does what to which servers and how often. 

MVS

SCO Cron  /crontab/root launches /u/mvs/sco/bin/night_sco.ksh  at 11:30PM Mon, Tue, Wed, Thu, Fri

/u/mvs/sco/bin/backtp script runs cpio to write backup to tape device /dev/rStp0.

Friday tapes are saved for 1 month, daily tapes are saved for 1 week.

Hosted 2008 internal server for MVS (Fogbugz server)

10/18/2016.  Visual Basic 2008 Express Projects folder has a manual backup on neil-pc-7pro : \OneDrive\SoftwareTools\ProfitsPlus\vbprojects.NET\VB_2008_Express

08/23/2017.  If you ever need to rebuild the Fogbugz server (again) go to neil-pc: OneDrive \archive\Fogbugz\buildserver and read the "Build the Fogbugz server from scratch.docx" document.  You'll thank me later.   You'll find backups for Fogbugz, Maxene_reporter and VB projects and full instructions on how to re-install all the software. 

I try to backup the Fogbugz server files to neilpc Onedrive once a month.  Use this .bat file on the Fogbugz server to run the Fogbugz backup and read the comments to see where to put the backup .exe file after it is made.

C:\MVS\Fogbugz\backups\backup_all.bat

This creates a self-extracting 7-zip .exe file named fogbugz_bkup.exe that I then copy to neilpc: OneDrive\archive\Fogbugz\backups 

I've learned not to copy the fogbugz_bkup.exe file from the Fogbugz server directly to the neilpc OneDrive folder.  It takes so long over RDP that OneDrive creates errors trying to sync it while the copy is still going on and the copy fails.   The solution is to

•  go to the Fogbugz C:\MVS\Fogbugz\backups folder and click the "Office 365 Login - Microsoft Office" shortcut
• login as neilinmich@outlook.com   password: AlwaysOpen!2021
• navigate to the web OneDrive folder archive.Fogbugz.buildserver.Fogbugz.backups and delete the old fogbugz-bkup.exe file
• drag the new C:\MVS\Fogbugz\backups\fogbugz_bkup.exe 7-zip file to the web  OneDrive:archive>Fogbugz>backups>buildserver>Fogbug>backups  folder.  It will take several minutes to sync.
• After it syncs to web OneDrive it won't automatically sync to neilpc (the file is too big) so you have to go to neilpc File Explorer and navigate to the Onedrive: archive/Fogbugz/buildserver/Fogbugz/backups  and right click on the new  fogbugz_bkup.exe file and select "Always keep on this device" from the drop down menu.  It should begin to sync to neilpc.   It takes several minutes.  
• Finally, to protect against ransomware, connect the "Seagate USB Drive I:" drive to neilpc (we normally keep it disconnected) 
• copy the new neilpc: OneDrive:archive\Fogbugz\backups\buildserver\Fogbug\backups\fogbugz_bkup.exe file to the neilpc Seagate USB Drive I:  (or M:) to This PC: Seagate Expansion Drive (M:)\OneDrive\archive\Fogbugz\buildserver\Fogbugz\backups folder.  
• Disconnect the Seagate USB drive from neilpc after the copy is complete.

Maxene Reporter can be backed up directly to OneDrive without using 7-zip.   Use the RDP connection to 

Copy paste Fogbugz: C:\maxene_reporter\MaxeneReporter\MaxeneReporter.mdb 

to neilpc: OneDrive\archive\Fogbugz\buildserver\Maxene_reporter\install

Copy paste Fogbugz: C:\maxene_reporter\MaxeneReporter\ReportGen folder 

to neilpc: OneDrive\archive\Fogbugz\buildserver\Maxene_reporter\install

Copy paste Fogbugz: C:\Revalue\SiProData folder 

to neilpc: OneDrive\archive\Fogbugz\buildserver\Maxene_reporter\install

Visual Basic 2008 express.   The project files don't get backed up regularly.  Whenever I make a change to the fogbugz project files I manually copy them to neil-pc-7pro : \OneDrive\SoftwareTools\ProfitsPlus\vbprojects.NET\VB_2008_Express  

I change the date for the most recent backup on this  trello card https://trello.com/c/TUr9RNdy

Hosted 2008 server for clients

The C: drive is for backups.   They are kept in the C:\Backups folder.  The files in these folders are not compressed. They are just copies.  You can drill down through the sub folders and copy any file back to the C:\ drive to restore that file. 

Daily backups

Each day of the week has its own folder.  For example, this is what the Monday folder would look like.   Each company has its own folder in the Day's folder.   The 3.MON.exe file is a 7zip.exe file that will be put on Google Drive.

A DOS batch file (C:\Backups\backup_all.bat) runs out of a Windows scheduled task at 9:00 pm every night.  Our 9:00 PM backup must be complete before the CSD backup starts at 11:00 pm.  Our night processing tasks start at 2:15 am. 

The C:\Backups folder will contain 1 copy for each day of the week plus the last 4 fridays. 

C:\Backups\backup_all.bat is the main batch file that drives the backup.  Each customer has it's own section in this batch file.  Here is the section for Revalue, it is one of the simplest ones:

• To add a new customer to the backup copy the section for an existing customer to make a new section and modify the lines as needed.  You must be sure to change the line for "SET COMP=xxx" to the abbreviation for the new customer.  This controls which \MVS\xxx folders get backed up for the section.  COMP stands for Company.   
• To remove a customer, simply delete or comment out their section.
• To modify a customer, add or change lines to their section, matching the pattern of the other lines.   

C:\Backups\backup_all.bat calls these other batch files (in the same folder) to do specific tasks. 

• backup_dsk0.bat
• backup_dsk0_sm.bat
• backup_dsk1.bat
• backup_dsk2.bat
• backup_misc.bat
• make_zip.bat
• mvtouch.exe (changes date/time stamp on backup folders)

This is NOT a full backup of ALL files.  The intention is to backup all essential Profits Plus folders and files so a working restore can be accomplished. Only certain file extensions in certain folders are backed up.  These files are backed up:

• Live company dsk0 module data folders for files with .dat, .mmo, .tbl, .ini, .sav, and .log  extensions.
• dsk0\100xxx, 210xxx, 230xxx folders for all G/L archive companies. 
•  \MVS\xxx\backup.out night processing log
•  \MVS\xxx\dsk0\100001\trash\blat_log.txt email log.
• dsk1, dsk2, win32, mysql folders and files.
• dsk0\002002, 007006, xxx000 folders and files.
• C:\maxene_reporter .mdb database and custom report .xlsx templates
• C:\Program files\MYSQL  my.ini 
• C:\Program files (x86)\stunnel stunnel.conf 

Weekly backups

\MVS\xxx\dsk2\???000\*.run files ARE NOT INCLUDED in the daily backup folders in C:\Backups\1.SAT, 2.SUN, 3.MON, 4.TUE, 5.WED, 6.THU, 7.FRI.x 

Run files are in a separate backup folder C:\Backups\runs 

To save time during the daily backup we do the .run files in a separate scheduled task on Sunday. 

The scheduled task backupPPruns is launched at 7:00 PM every Sunday to run  C:\Backups\backup_runs_all.bat which backs up the .run files to a separate set of  sub folders in C:\Backups\runs

 

This is an example of a section in backup_runs_all.bat for a single company. 

This is an example of a section in the backup_runs_one.bat that is called from the backup_runs_all.bat above.

• To add a new customer to the backup copy the section for an existing customer to make a new section in backup_runs_all.bat and modify the lines as needed for the new customer.   You shouldn't have to make any changes to backup_runs_one.bat (it is the same for all customers).
• To remove a customer, simply delete or comment out their section.
• To modify a customer, add or change lines to their section, matching the pattern of the other lines.   

 

Google Drive offsite storage

The backup compresses all backups for a day into a single self-extracting 7zip.exe file and puts it on our Google Drive folder HostedBackp_clients.  With names like "4.TUE.EXE", "5.WED.EXE", "6.THU.EXE".  This takes about 2.3 GB per day. To protect them from ransomware attacks we delete them on the Web site from the cloud HostedBackup_clients folder the next day so they will go into the cloud trash folder which is safe from ransomware.  We only keep the last 3 days in the cloud trash folder of google Drive, deleting older copies as needed. Daily backups for Saturday or Sunday are created locally but never put on Google Drive. 

The Weekly Sunday run files backup has the date in the filename and looks something like runs20180225.exe.  We keep just the most recent runs*.exe  backup on Google Drive in the trash folder.  

Warning: We have to regularly (usually daily but at least once a week) log into the Google Drive web site for the ProfitsPlus account and delete the oldest files in the trash folder, or our 15GB of free storage will run out.  Files in the cloud trash folder can only be deleted from the Web site application.    

Warning: We no longer have to leave the MVSoft user always logged in at night so the Google Drive will sync the backup files.  Google drive doesn't sync anymore on Hosted 1 server  because Google doesn't support Windows 2008 anymore. 

Misc notes:

• Backup the backup scripts: The C:\Backups\*.bat backup scripts also have a backup on C:\MVS\z_backup_scripts folder.  This isn't done automatically so you have to copy them to there yourself after you make changes to the live ones on the C: drive.   The C:\MVS\z_backup_scripts folder is added to the daily *.exe backup file.
• Backups are forever: To make the backup run as fast as possible, the backup scripts never delete the old version of the files.  They just copy new versions, replacing old versions.  So if for some reason you delete the file C:\MVS\rev\dsk0\007006\menu23.ctl  it will not get deleted in the C:\Backups folders.  It will just stop getting backed up, but the last backup of it will still exist forever (or until you delete it from there as well). 
• mvtouch.exe is a Visual Basic program written just for doing these backups.  The backup scripts call it to change the date/time stamp of a backup folder.  The source code is in Visual Basic 2008 Express on the other hosted 2008 server for fogbugz.       

 

Restore Hosted 2008 server from Google Drive copy.

This is the worst case scenario.  It is assuming the C:\Backups hard drive is lost and you must use the off-site Google Drive copy to do a full restore.

Open a web browser on the machine that will be receiving the Google Drive backup.  This doesn't have to be the same hosted server that created the backup originally. 

Go to the profitsplusmv@gmail.com google account web page.  

• www.google.com/drive
• Login: profitsplusmv@gmail.com    Password: mvsoftware

Restore from daily backup 1st

Find the latest daily backup copy.  

• If there is one or more backup .exe files still in the HostedBackup_clients folder use the latest one.  
• If not, go to the trash folder and right click on most recent backup .exe file.  Choose "Restore" from the pop-up menu.   That will put the backup file back into the HostedBackup_clients folder.  

Go to the "MY Drive"  folder and then open the HostedBackup_clients folder.  You should see the .exe file you just restored from the trash folder. 

 

Right click on the .exe file and choose "download" from the pop-up menu.

 

You might get a warning that the file is too big for a virus scan.  Choose "Download anyway".

 

The .exe file will get downloaded into your normal web download folder.  It is a big file so it will take several minutes. 

Create a new empty folder on the same drive that will become the Profits Plus folder.   I created C:\u\backup  and the live Profits Plus folder will be C:\MVS\rev (for the Revalue company).

If you are only restoring one company then move the backup .exe file from your Web download folder to some  new (temp) folder on the hard drive.  If you are in the unfortunate situation where you are restoring all companies at once then you might as well put the backup .exe file into an empty %MVS_ROOT% folder (in this case C:\MVS) and let the file unzip directly into the live folder.   

The backup .exe file will always unzip to create the 3 character company sub folders in the same folder that has the backup .exe file.     So if you put the backup .exe in C:\MVS  it will create sub folders C:\MVS\aba, C:\MVS\ber, ....   Or if you put the backup .exe file in C:\u\backup it will create sub folders C:\u\backup\aba, C:\u\backup\ber, ...

Double click the .exe backup file so it will self-extract into the current folder.  

Windows might not trust the  .exe file.  My PC popped up a window saying Windows Smartscreen could not be reached and warned me that the .exe file was from an unknown source.  It offered me two buttons "Run", "Don't Run".   I chose "Run".

The 7-zip dos window will popup and display the extract progress.  It runs slow so it will take several minutes to extract and un-compress the files and folders.   There is no way to select just certain files to extract so you have to extract ALL files. 

After the extract is finished the backup folder will contain the original contents of the hosted server's C:\backups folder for that day.

For my restore test I dragged the rev folder of the backup to the %MVS_ROOT% (C:\MVS) folder on the new machine.  This moved the rev company backup to the same path that Revalue had on the original hosted machine. 

Restore from weekly backup 2nd 

You're not done yet.  We don't backup the compiled .run files in the daily backup (to save time and space) so you'll have to decide how to restore them.  You have 2 choices:

• Do a separate mass compile for each customer that is being restored.  If you are restoring 3 customers, you'll have to do 3 mass compiles. Each mass compile will take about 1 hour.
• Or restore the backed up .run files using the Sunday runs*.exe file on Google Drive.   This will restore all .run files for all customers at once.   It takes about 1 minute.  You have to restore ALL customers this way, you can't select which customers to unzip.
  • Copy the runs*.exe file to the C:\MVS folder or your temp version of that.  For example, unzipping the runs*.exe from C:\backup will create sub folders C:\backup\aba\dsk2\???000,     C:\backup\ber\dsk2\???000, etc. 
  • Double click on the runs*.exe file to unzip it and say you want to always overwrite any existing files.  
     

Warning:  These .run files might be the wrong version.  These Sunday  .run files were not backed up on the same day as the .dat data files were backed up.   So if you know that a customer had a full software update since the Sunday backup and it changed .fdm files which converted .dat files to new versions then the backed up .run files are not good anymore.  You should do a mass compile for that specific customer instead.   That will guarantee the .run files are the right version for the .dat data files.   

There is an %MVS_ROOT%\au\logs\fdm_cvt_???.prt text file that lists the .fdm files that got converted in the most recent full software update.   This might help you decide if a mass compile is needed. 

If you aren't restoring all customers then you'll probably be unzipping the daily and weekly backup files into a temp folder anyway and restoring files from the temp folder.  In that case you can just copy the .run files from the backup temp folder for the customers that haven't had a recent software update, and do a mass compile for the customers who did. 

 

When you have the .run files restored or recompiled you should be able to run Profits Plus again.   Go to the %MVS_ROOT%\win32\startup folder and double click the ProfitsPlus.bat file to start Profits Plus and verify you can run programs normally. 

You will probably have to set the DOS Window properties for %MVS_ROOT%\win32\startup\Profits Plus.bat  shortcut to be Font Tab: Lucida Console size 20 and Layout tab: Height 25.

Since this is a worst case scenario there might be other issues to resolve before the customer is fully restored.  If they used any of these 3rd party software packages then they would have to be re-installed.

• Mysql 5.5.
  • Installation instructions are here
  • Replace the my.ini int the MySQL installation folder with the one in the root folder in this Google drive backup.

• MaxeneReporter. 
  • Installation instructions are here
  • There is a MaxeneReporter folder in this Google drive backup.
  • Replace the MaxeneReporter.mdb and customer .xlsx templates with ones from this Google drive backup. 

• MVSWEB.  Currently we don't have MVSWEB on the hosted 2008 server.
• Stunnel version 17.  The stunnel.conf file is in this backup root folder.
  • Installation instructions are here
  • Replace the stunnel.conf file with the one from the root folder of this Google Drive backup. 

Full restore complete!

Mocha (defunct)

As of 07/15/2015 we are moving away from mocha because it has become unreliable.   As of 8/26/2016 mocha is turned off an not used at all.

Mocha E: drive.    There is a simple file copy backup of mocha E: drive in Neil-PC-7Pro I:\mocha\copy_archive_drive as of 7/16/2015.  I hope we never need it.

4/28/2016. The contents of old mocha e: drive (for archives and downloads) has been moved to \neil-pc-7pro\Users\Neil\OneDrive\archive.  Please consider this as the live working folder.  

Note: Many of the important download folders are also archived to OneDrive "SoftwareTools" folder but they are not considered the live working copies.

================================
Maxene Reporter folders
================================
This is probably confusing (it was to me).

Both mocha e: and c: drives contained a folder named maxene_reporter but with different files.
neil-pc-7pro merges mocha's c: and e: drives into a folder/subfolder arrangement.

  Old location
  -------------
  mocha e:
      maxene_reporter     (contains many subfolders)
  mocha c:
      maxene_reporter     (contains just one subfolder)  
          MaxeneReporter  (contains many folders as new as 2014 different than e: drive)

  New location
  --------------
  neil-pc-7pro c:\Users\Neil\OneDrive\archive (merges mocha's c: and e: drives  into a folder/subfolder arrangement)
      MaxeneReporter_classic     (from mocha e:)
         MaxeneReporter  (from mocha c:)

     MaxeneReporter_DotNet  (.net software)

neil-pc-7pro

We use neil-pc: OneDrive for backup of most files.  

To guard against ransomware attacks we make periodic (monthly) simple copies to the USB I: drive  which is kept unplugged most of the time.  These folders are copied to the I: drive:

• OneDrive
• GoogleDrive
• Oracle Virtual  Box
• ftproot MVSWEB for unix MVS

OneDrive contents

neil-pc-7pro c:\Users\Neil\OneDrive\ contains 99% of MVS Alliance backed up files (except for the Unix MVS server).

• the archive folder that used to be on mocha e: drive.
  • And the following new folders
    • email_documents
    • Trello
    • VB_2008_Express (from hosted 2008 fogbugz server)
    • VB_2010_Pro (from neil-pc)
• BusinessData folder that used to be on the jre server.
  • u\user\ contains xfer, dh_accting, irina, mv_accting folders.
  • Dependable_archive for backing up Dependable's server.

Google Drive contents

neil-pc-7pro c:\Users\Neil\Google Drive\ contains the connections-rdp  and "Applications and tools" folders.

Oracle Virtual Box

• A virtual XP machine that used to be Vivek's VB6 MVSWEB development machine is installed on neilpc.  

• The VM file is in C:\VirtualBox VMs\Windows XP Mode\Windows XP Mode.vdi.     I backed it up to a USB drive I:\OneDrive\archive\Vivek's desktop\MV Software Development\VirtualBox VMs  on 5/15/2020.

• We also use it to maintain old VB6 Profits Plus programs (probably never need it again).  The live project files are on  Neil-pc-7pro\c$\Users\Neil\OneDrive\SoftwareTools\ProfitsPlus\vbprojects.vb6  and can be accessed from within the virtual machine via a shared drive. 

• A one time copy (7/15/2015) of VB6 vbprojects is kept on Neil's personal neilinmich@gmail.com Google drive WORK:vbprojects\sabrina_vb6\vbprojects

Sabrina (defunct)

We keep it powered down now.  Most usable files have been moved to OneDrive.   

vb6 source code is in neil-pc: OneDrive\SoftwareTools\ProfitsPlus\vbprojects.vb6.   Visual Basic 6.0 compiler is on the virtual XP machine on neil-pc.   Hopefully we will never need it.

A one time DOS copy backup  8/23/2017 was put on neil-pc USB drive I: of the following folders just in case we ever need them.

Sabrina

   Pavan

      Download

      mail

      pavan_mvs

      Revalue

   U    - profits plus 2014

 

 

Mobiecast (defunct)

These files are no longer used or backed up.  Most files moved to neil-pc: OneDrive

 

Stella (defunct)

8/23/2017. Most files are on neil-pc: OneDrive

 

Phone (defunct)

Simple file copy.  No backup software needed.

D: drive is backed up to Neil-PC-7Pro "I:\PhoneBackup-D"

This is a one time file copy created 9/3/2013 with no scheduled repeats.

I also found old backups on the "backup" server \Phone folder.