68304: Retention rules: how and when they work

    Last update: Fri, 2021-03-12 12:41

    Introduction

    For clarity of the article, we will define two terms: “backup type” and “backup sets”.

    Backup type determines the way data is stored inside the archive. It can be “full”, “differential” and “incremental” . Full backups store all information present at the backup creation time; differential backups store only the difference between the last full backup and the current state of data; incremental backups store the difference between the current state of data and state of data from previous successful backup. For more information, see Difference between Full, Incremental and Differential Backup.

    Backup sets determine the exact schedule and conditions under which the backup had been initiated. Backup type can be disregarded when working with backup sets because the backup set has no connection to the methods of storing backup contents.

    For the purpose of retention calculation, only the backup sets are used. If you want to learn how retention affects different backup types and backed-up data, please refer to the description of archive12 format instead.

    Retention rules

    Relation to backup scheme and backup types

    The schedule of the backup plan defines the backup scheme used to operate with a specific archive as well as various types of backup sets.

    Glossary:

    A backup set is a group of backups to which an individual retention rule can be applied. For the Custom backup scheme, the backup sets correspond to the backup methods (Full, Differential, and Incremental). In all other cases, the backup sets are Monthly, Daily, Weekly, and Hourly. 

    A monthly backup is the first backup created after a month starts. 

    A weekly backup is the first backup created on the day of the week selected in the Weekly backup option (click the gear icon, then Backup options > Weekly backup). If a weekly backup is the first backup created after a month starts, this backup is considered monthly. In this case, a weekly backup will be created on the selected day of the next week. 

    A daily backup is the first backup created after a day starts unless this backup falls within the definition of a monthly or weekly backup. 

    An hourly backup is the first backup created after an hour starts unless this backup falls within the definition of a monthly, weekly, or daily backup.

    Under default settings, retention rules are applied based on the backup sets and are tied directly to them. Alternate retention rules allow changing this approach based on customer requirements.

    Retention rules will only contain rules for the sets that are applicable to the current backup scheme and schedule. If you are missing sets that you otherwise expect to see in the retention settings make sure to check and verify the schedule settings first. Keep in mind that certain retention rules are not applicable to certain archive types or destinations – refer to User Guide for details

    Retention rules types

    • By backup age: define a separate rule for each backup set or use a single rule for all sets
    • By number of backups: define the exact number of backups to keep
    • By total size of backups: define the size of the archive to trigger retention
    • Keep backups indefinitely: retention rules are disabled

    The last three rules are generally simple, thus do not require detailed explanation. The focus of the remaining article will fall on the primary rule “By backup age”.

    By number of backups: when the number of backups stored inside the archive exceeds the set threshold (maximum amount), the retention rules are applied. Users can choose to apply the retention before or after the backup and can also set the maximum amount of backups to the threshold of 1 (retention after backup) or 0 (retention before backup) to keep only one backup in the archive at all times.

    By total size of backups: when the backup archive exceeds the set threshold (maximum size), the retention rules are applied. The software will attempt to remove the oldest backups from the archive until the archive no longer exceeds the threshold or until only one backup is remaining in the archive. If the size of a single backup exceeds the value provided, the archive will exceed the size as well, but will only contain that single backup until the values are modified.

    By backup age: detailed

    Single rule for all backup sets

    A simplified method is to use a single rule for all backup sets. The user defines the exact duration by selecting the number and the measurement unit: weeks, days, months, hours, etc. This rule will be applied to all backup sets, removing ambiguity but also reducing flexibility.

    Individual rule for each backup set

    The most flexible and complex method is to set a separate rule for each backup set. With a custom backup scheme, the rules will be set per backup type: Full, Differential, and Incremental. With any other backup scheme, the rules will be set per backup set: Monthly, Weekly, Daily, Hourly. Only backups sets applicable for the current backup scheme will be available in the retention rules settings, e.g. for “Weekly Full, Daily Incremental” scheme the retention rules will not contain monthly settings because the monthly set is not present in that backup scheme, thus the retention can only be set for “Weekly” and “Daily” sets.

    Whenever software creates a backup, it defines the current backup set of this particular backup for further retention purposes, this value cannot change over time. Refer to Glossary to find out how exactly the backup set of the current backup is defined.

    Before software starts the “applying retention rules” procedure, whether it is before or after the backup, it goes through existing backups in the current archive and calculates which ones are expired based on defined retention rules. These backups get removed from the backup chain, consolidating or deleting any unreferenced data remaining in the archive.

    You need to define the backup set of each backup that the archive currently stores for a better understanding of the process. The examples below aim to help with that.

    General calculation rules

    To calculate if the retention applies to certain backups follow this pattern:

    1. Define the set of the target backup:
      1. For custom schemes, define if the backup is Full/Differential/Incremental based on schedule
      2. For other schemes define if the backup is Monthly/Weekly/Daily/Hourly based on the definition of the corresponding set – refer to Glossary
      3. If any backup is skipped or failed when it runs per schedule, the next successful one will take its place in this logic.
    2. Check when the retention process starts. The exact time matters because the software checks backup age exactly when the retention process starts (and not when the associated backup process starts).
    3. Define the age of the target backup in full years/months, weeks, days, and hours, based on the retention process starting time.
    4. Compare the age of the backup to the retention rule for its set (if a single rule is used for all sets – compare the age to that rule):
      1. If the age value is larger than the set rule even by the smallest margin, the backup is subject to retention. Consider 1 month and 1 minute larger than 1 month.
      2. Otherwise, the backup is not subject to retention.

    Important notes:

    • Outside of retention logic, users can manually delete any backup in the chain regardless of their type or set. Since the sets are static, using manual deletion does not change anything for retention logic. Deleting a monthly backup that is not a subject for retention will not change the set of any subsequent daily or weekly backups to become a monthly one; thus retention may still remove the remaining backups of the same month, resulting in the complete absence of a backup in a period.
    • In the current Acronis Cyber Cloud implementation, retention is always a part of a backup plan. If the backup plan is not running due to any circumstances, fails, or is interrupted before retention can start, the retention will not work. Disabled and revoked backup plans also do not apply retention. Separate retention plans are planned to be included in future versions of the product.
    • Changing the retention of a backup that is already running will only apply to the next backup job run. To preserve backups that are already subject to retention (but had not yet been removed) cancel the running procedure completely, change and save retention settings and re-run the procedure.

    Examples

    Default (Always incremental, single-file; Monthly – 6 months, Weekly – 4 weeks, Daily – 7 days)

    Assuming today is 16.Feb.2021 and the backups started on the 01.Jun.2020, ran on a daily basis without failures with weekly backup on Monday, we have:

    • Monthly set:
      • 01.Jun.2020 (8 months and 14 days old – more than 6 months – deleted)
      • 01.Jul.2020 (7 months and 14 days old – more than 6 months – deleted)
      • 01.Aug.2020 (6 months and 14 days old – more than 6 months – deleted)
      • 01.Sep.2020 (5 months and 14 days old – less than 6 months – present)
      • 01.Oct.2020, 01.Nov.2020, 01.Dec.2020, 01.Jan.2021, 01.Feb.2021 (less than 6 months – present)
    • Weekly set:
      • Mondays up to 18.Jan.2021 (4 weeks and 1 day old – more than 4 weeks – deleted)
      • 25.Jan.2021 (3 weeks and 1 day old – less than 4 weeks – present)
      • 01.Feb.2021 – defined as monthly, not related to weekly set (same applies to 01.Jun.2020), the rule is ignored
      • 08.Feb.2021, 15.Feb.2021 (less than 4 weeks – present)
    • Daily set:
      • Any daily backup up to 08.Feb.2021 (more than 7 days – deleted)
      • 09.Feb.2021 (7 days – present, but will be deleted when retention starts)
      • 10-14.Feb.2021 (less than 7 days – present)
      • 15.Feb.2021 – defined as weekly, not related to daily set, the rule is ignored
      • 16.Feb.2021 (less than 7 days – present)

    The number of days/weeks/months to keep a specific backup set will not always be equal to the number of such backups in the archive. Different scenarios will result in different numbers, e.g.:

    • If you set to backup daily but only from Monday to Friday with 7 days retention, the highest amount daily backups you will ever have will be five (four – excluding the weekly backup, if any);
    • A device that had skipped the full month of September will have monthly backups starting from October. Regardless of the number of backups, the calculations are the same: if the backup of the current set exceeds the rule of the set, the software deletes the backup. Hence skipping periods of time within retention rules will result in a smaller amount kept in the archive;

    Always full, weekly on Fridays only (Monthly – 1 month, Weekly – 3 weeks, Daily – 10 days)

    Since the weekly scheme allows creating multiple backups within one week it assumes that “Daily” set is available in the backup and will contain retention rules for all three sets. Under certain conditions, the daily backup may even be present, which we will also cover here. Assuming today is 16.Feb.2021 and the backups started on 12.Jan.2021 ran on a weekly basis without failures with weekly backup on Friday; additionally, manual backups launched on 20.Jan.2021 and 10.Feb.2021.

    • Monthly set
      • 12.Jan.2021 – first backup in January, thus defined as Monthly (1 month and 4 days – more than 1 month – deleted)
      • 01.Feb.2021 (less than 1 month – present)
    • Weekly set
      • 15.Jan.2021 (4 weeks and 4 days – more than 3 weeks – deleted)
      • 22.Jan.2021 (3 weeks and 4 days – more than 3 weeks – deleted)
      • 29.Jan.2021, 05.Feb.2021, 12.Feb.2021 (less than 3 weeks – present)
    • Daily set (manual backups are defined as daily, unless they happen to be first successful backups of the week or the month launched before schedule, in which case the scheduled one will be defined as daily instead)
      • 20.Jan.2021 (more than 10 days – deleted)
      • 10.Feb.2021 (less than 10 days – present)

    Custom, Full backup twice a year at 1 a.m., differential twice a month at 1 a.m. (except when Full is done), Incremental daily, including days of Full and Differential, at 6 a.m. (refer to picture)


     
    This is an example of a relatively complex schedule set up using a custom scheme. In this case, you will have three backup sets: Full, Differential and Incremental. The sets are not tied to daily, weekly or monthly type and retention rules are defined for sets based on the backup type instead. For the purpose of this example, we will set the following retention rules: Full – 10 years, Differential – 2 years, Incremental – 10 weeks (refer to picture)


     
    Assuming today is 16.Feb.2021 the backup had already run for more than 10 years you will see the following backups:

    • Full set
      • Already deleted: 01.Jan.2011 (10 years, 1 month, and 15 days old) and earlier
      • Still present: 01.Jul.2011 (9 years, 7 months, and 15 days old) and later
    • Differential set
      • Already deleted: 15.Feb.2019 (2 years and 1 day old) and earlier
      • Still present: 01.Mar.2019(1 year, 11 months and 15 days) and later
    • Incremental set
      • Already deleted: 15.Dec.2019 (10 weeks and 1 day old) and earlier; keep in mind that the differential backup created on 15.Dec.2019 at 01:00 am is present because it does not belong to incremental set and has its own rule
      • Next for deletion after today’s backup: 16.Dec.2019 (10 weeks exactly)
      • Still present: 17.Dec.2019 (9 weeks and 6 days) and later

    Tags: