{"id":1712,"date":"2020-10-31T13:19:39","date_gmt":"2020-10-31T12:19:39","guid":{"rendered":"https:\/\/locallost.net\/?p=1712"},"modified":"2020-10-31T13:33:05","modified_gmt":"2020-10-31T12:33:05","slug":"huge-var-log-lastlog-file-and-backuppc-backup","status":"publish","type":"post","link":"https:\/\/locallost.net\/?p=1712","title":{"rendered":"Huge \/var\/log\/lastlog file and BackupPC backup"},"content":{"rendered":"\n<p>After enabling <a href=\"https:\/\/locallost.net\/?p=1705\" data-type=\"post\" data-id=\"1705\">AD authentication<\/a> on a server you might end up with a \u00ab\u00a0huge\u00a0\u00bb <code>\/var\/log\/lastlog<\/code> file that in my case did not played nice with my BackupPC with rsync xfer method: it resulted in a 300 GB lastlog file!<\/p>\n\n\n\n<p>The reason is that this <code>\/var\/log\/lastlog<\/code> file is a \u00ab\u00a0pre-allocated database\u00a0\u00bb file that stores the login activity for all the users UIDs on the server, and being connected to a Samba AD DC, I now have users with UIDs well above 1 billion! This \u00ab\u00a0pre-allocation\u00a0\u00bb uses the sparse feature and allows to have a 300 GB file allocated using only 28 KB of real disk space:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># ls -lh \/var\/log\/lastlog\n-rw-rw-r-- 1 root utmp 279G oct.  31 12:35 \/var\/log\/lastlog\n\n# ls -sh \/var\/log\/lastlog\n28K \/var\/log\/lastlog<\/code><\/pre>\n\n\n\n<p>Handy feature. However, when you start backuping a host like this, you might end up with disk-usage problems when all your backup chain is not \u00ab\u00a0sparse-aware\u00a0\u00bb and is not able to treat sparse files efficiently.<\/p>\n\n\n\n<p>This showed up with BackupPC using the rsync transfer method, when dumping this new host blew up my disk space.<\/p>\n\n\n\n<p>The problem here is that the BackupPC&rsquo;s rsync transfer do not handle sparse file \u00ab\u00a0efficiently\u00a0\u00bb resulting in effectively retrieving and storing 300 GB of data before compressing it to its cpool.<\/p>\n\n\n\n<p>Warning : do not try to add the <code>--sparse<\/code> flags to the rsync transfer method as the BackupPC&rsquo;s rsync receiver is not compatible with it or you might end up with corrupted files in your backup.<\/p>\n\n\n\n<p>So, the only safe solution for the moment is to exclude the <code>\/var\/log\/lastlog<\/code> from backup.<\/p>\n\n\n\n<p>P.S.: here is a quick one-liner to detect sparse files, ordered by increasing ratio of real-data-size \/ total-sparse-size (lower means a huge sparse file with few data in it):<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>find \/ -type f -printf \"%S\\t%h\/%f\\n\" 2>\/dev\/null | grep -v '^0&#91;&#91;:space:]]' | sort -k1g | head<\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"<p>After enabling AD authentication on a server you might end up with a \u00ab\u00a0huge\u00a0\u00bb \/var\/log\/lastlog file that in my case did not played nice with my BackupPC with rsync xfer method: it resulted in a 300 GB lastlog file! The &hellip; <a href=\"https:\/\/locallost.net\/?p=1712\">Continuer la lecture <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[1],"tags":[],"class_list":["post-1712","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p2Bei9-rC","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/locallost.net\/index.php?rest_route=\/wp\/v2\/posts\/1712","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/locallost.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/locallost.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/locallost.net\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/locallost.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1712"}],"version-history":[{"count":3,"href":"https:\/\/locallost.net\/index.php?rest_route=\/wp\/v2\/posts\/1712\/revisions"}],"predecessor-version":[{"id":1716,"href":"https:\/\/locallost.net\/index.php?rest_route=\/wp\/v2\/posts\/1712\/revisions\/1716"}],"wp:attachment":[{"href":"https:\/\/locallost.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1712"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/locallost.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1712"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/locallost.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1712"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}