- Append-Only Mode
- Other Questions
If you get
Connection closed by remote host. Is borg working on the server?, it is almost always a problem with SSH keys. Double-check the following to debug further:
Have you already assigned a SSH key to the repo on BorgBase.com and is this the same key you are using locally? BorgBase will show the key’s SHA256 fingerprint in Account > SSH Keys. You can compare this to your local fingerprint like this:
$ ssh-keygen -lf ~/.ssh/id_ed25519
Is SSH using the key you assigned? By default only
~/.ssh/id_[rsa | ed25519 | ecdsa]are used. When using a custom key name, you can add
~/.ssh/config. Or to only use this key with BorgBase:
Host *.repo.borgbase.com IdentityFile ~/.ssh/id_custom
- Is SSH trying too many keys? The maximum number of keys (
MaxAuthTries) that can be tried per connection is 6. If you have more keys, specify the key to use, as shown above.
Are the key permissions OK? Private keys and important config files (if in use) need to have a permission of
0600. To change the permission:
$ chmod 0600 ~/.ssh/id_custom ~/.ssh/config
If you still get errors, try to connect to your repo using the
sshcommand with verbose logging enabled:
$ ssh -v firstname.lastname@example.org
This will print a list of keys being tried and potential problems. You won’t get a shell at the end, as BorgBase only supports access via
borg. Once you see
Remote: Key is restricted. or
PTY allocation request failed on channel 0 then the login step still worked.
All our servers are connected with 1Gbit connections at a minimum and located in professional data centers. We rarely get reports of slow backup speeds. If you do encounter slower-than-expected backups or slow upload speeds, you can follow the steps below to find the bottleneck.
First it helps to understand the steps
borg follows when creating an initial or new backup:
- First it will do some housekeeping, like getting the index from the repo, if there is no local copy.
- Next it will compare all the inode ID and other attributes of all files to determine changed files. If you move your files to a new file system, the first backup run can take a bit longer, but no new data will be copied.
- If new data is found, Borg will checksum, compress and encrypt the files as segments of 1-5 MB, skipping any known segments. So if part of a large file changes, only new parts will be uploaded.
- Last, it will upload new segments to BorgBase.
So the upload speed is not always the main bottleneck. Depending on your setup, you should also watch out for CPU usage or disk IO. It’s usually difficult to improve uplink speed, but if you are CPU- or IO-limited, there are a few settings you can tune. Just be aware of the trade-offs. Maybe you are OK with a slower initial backup in order to have a smaller well-compressed backup in the future.
- Make sure you choose an appropriate compression level for your data. In general
lz4(fast, but low compression)
zstd,3(medium compression) and
zstd,8(high compression) will work well.
- If you don’t need additional file flags, you can disable them with
bsd_flags: falsein Borgmatic. This can lead to dramatic performance improvements when your backup consists of many small files. (With Borg >1.2, use
- Ensure the local files cache is working correctly. By default Borg will compare
ctime,size,inodeand process the file if either changes. If you have e.g. a network file system with unstable inodes, try using
- Avoid excessive archive checking:
borg checkcan read all backup segments and confirm their consistency. For large repos this can take a long time. BorgBase already uses different techniques to avoid bitrot in the storage backend, so
borg checkis not strictly necessary for this purpose. In Borgmatic set
consistencysection. If you still need consistency checks, consider using the
repositoryoption to limit the check to the repository. Checking all archive metadata is done on the client and very time consuming. See the official Borg docs for details.
- If you suspect a slow (certain residential internet connections come with restricted upload speed) or unstable network connection, we can temporarily enable
iperf3for you server-side.
If Borg happens to be busy on the client- or server side, it may not send data over the SSH connection for a while. In this case, some ISPs will terminate the connection after a period of inactivity, especially if a NAT is involved. You would then see an error like this:
Remote: packet_write_wait: Connection to xxx.xxx.xxx.xxx: Broken pipe Connection closed by remote host
Remote: client_loop: send disconnect: Broken pipe RemoteRepository: 2.61 kB bytes sent, 1.01 MB bytes received, 52 messages sent Connection closed by remote host
Which means the SSH connection has been terminated and Borg is unable to send data to the server-side process. One possible solution is to have the client send regular keep-alive packages while no data is sent by Borg. On the client machine, you can add the below configuration to
Host *.repo.borgbase.com ServerAliveInterval 10 ServerAliveCountMax 30
This configuration means that the client will send a null packet every 10 seconds to keep the connection alive. If it doesn’t get a response 30 times, the connection will be closed. BorgBase already has the appropriate
ClientAliveInterval configuration applied server-side.
If you still encounter issues, you may be using a VPN, a mobile network that aggressively terminates idle connections or a residential internet connection with short outages. In that case, you can use a simple retry script during the initial upload. It will retry the command if it exits with an error. Borg also adds checkpoint archives every 30 minutes, so data is only uploaded once, even if the backup run is interrupted.
In this mode, Borg will never remove old segments and instead add a new transaction for any change in a transaction log. The result is that no data is ever deleted and unwanted operations (like archive prunes- or deletions) can be undone. This is useful if the client machine shouldn’t get full access to its own backups to e.g. prevent a hacker from deleting backups after taking over a client machine. For full details and instruction on how to roll back, see the official Borg docs.
The server-side Borg process doesn’t know about the high-level commands (
borg prune) you run. It only knows about adding chunks, removing chunks and so on. So with the current architecture, it’s not possible to reject e.g. a
borg delete command right away.
As a result, Borg developers made the decision to fail delete commands “silently”. Effectively this means that while running backups with append-only ssh keys, no disk space will be recovered in your BorgBase repo with pruning. But you can run a prune with an all access ssh key when your free quota is running low, which will then clear pruned backups and free up disk space.
With append-only mode enabled, the repository will have a timestamped transaction log. This allows going back to previous states, even if prune- or delete-commands were issued by the backup client.
If you need to restore an older repo version, you can export the whole repo using
rsync and roll back to a previous transaction.
When using append-only mode, old transactions and segments are never cleaned from the repo and the size can grow over time. To really prune append-only repos and reduce their space usage, you have two options:
- Temporarily set your main key to full access mode. This will remove all old transactions during the next write operation.
- Use a trusted admin machine with a full access key to prune. This will also clear old transactions.
Note that old data is only deleted, if a full access key makes a write operation (create, delete, prune). If you prune, but no archive is actually deleted, it’s not a write operation.
Both regions are currently using hardware RAID-6 backed storage servers with enterprise drives and replacement drives ready on-site. This protects against hardware failure and a degree of bit rot. For a list of the providers we work with, you can also see our GDPR page.
While this provides good durability against hard drive failure, BorgBase doesn’t offer geographical redundancy or redundancy across multiple data centers for single repos. It’s thus not recommended to use our service for archiving purposes, where there is no other copy of the data.
If you want geographic redundancy, you can add two repos in different regions and do parallel backup to both of them. This case is, e.g. well supported by Borgmatic, which allows adding a list of repositories.
We offer free migration services for both, incoming and outgoing transfers. Currently this is done manually. To start a transfer follow these steps:
- For incoming transfers, create an empty repo in your BorgBase account.
- Make sure your old repo data is accessible from the internet somehow, e.g. via SSH, FTP or HTTP. For SSH we will provide you with a one-time public key to use.
- Contact our support and provide the BorgBase repo ID and the login details of the transfer source or target.
BorgBase always displays the actual disk usage, as measured on the file system. This includes some metadata and index files and slight variations from the space usage reported via
borg info under All Archives > Deduplicated Size are expected.
If you see larger variations, you are probably running your repo in append-only mode. This means
borg never really deletes old segments. So the actual disk usage will be higher than what
borg thinks over time. From the docs:
As data is only appended, and nothing removed, commands like prune or delete won’t free disk space, they merely tag data as deleted in a new transaction.
If you are OK to fully remove those old segments, then just write to the repo with a full-access key. This will clean up old segments:
Be aware that as soon as you write to the repo in non-append-only mode (e.g. prune, delete or create archives from an admin machine), it will remove the deleted objects permanently
borg check command can detect issues with your repository, but running it on large repos will make your backups much slower and cause higher load on our storage servers. It’s not really necessary to run it more often, since the underlying RAID controller already does weekly data scrubs to verify the data. As a result we strongly recommend running
borg check once a month only. When using Borgmatic, this can be done like this:
- Run this cron task every day to create a new backup:
borgmatic --create --prune
- And this once a month (some random day a few hours after the backup):
Our Ansible role also supports monthly checks, when
borgmatic_large_repo is set to
True (the default).
If we notice excessive repo checks on your account, we may contact you or interrupt those processes in order to guarantee the best speed for all users.
If you have found another backup service and prefer to remove your account, you can do so any time. Doing so will remove your account data permanently. If you ever choose to start using BorgBase again, you will have to open a new account. If you ever had a paid subscription, we will still keep some invoicing data, as required by law. To remove your account:
- Remove all your repositories and make sure the data is saved elsewhere. You can also transfer your whole archive via
rsync, as described here.
- Log into your account and navigate to Account > Profile. Then click Remove Account
Sometimes it’s necessary to restart services and servers to apply security- and maintenance updates. Usually such updates are not urgent, so we schedule them in a way to minimize any service disruption, while doing them. So the maintenance window for each region is as follows:
Usually there won’t be any downtime during this maintenance window. But if restarts or maintenance are required, it will be done during that time, unless more urgent action is needed.