Huge /var/log/lastlog file and BackupPC backup

After enabling AD authentication on a server you might end up with a « huge » /var/log/lastlog file that in my case did not played nice with my BackupPC with rsync xfer method: it resulted in a 300 GB lastlog file!

The reason is that this /var/log/lastlog file is a « pre-allocated database » file that stores the login activity for all the users UIDs on the server, and being connected to a Samba AD DC, I now have users with UIDs well above 1 billion! This « pre-allocation » uses the sparse feature and allows to have a 300 GB file allocated using only 28 KB of real disk space:

# ls -lh /var/log/lastlog
-rw-rw-r-- 1 root utmp 279G oct.  31 12:35 /var/log/lastlog

# ls -sh /var/log/lastlog
28K /var/log/lastlog

Handy feature. However, when you start backuping a host like this, you might end up with disk-usage problems when all your backup chain is not « sparse-aware » and is not able to treat sparse files efficiently.

This showed up with BackupPC using the rsync transfer method, when dumping this new host blew up my disk space.

The problem here is that the BackupPC’s rsync transfer do not handle sparse file « efficiently » resulting in effectively retrieving and storing 300 GB of data before compressing it to its cpool.

Warning : do not try to add the --sparse flags to the rsync transfer method as the BackupPC’s rsync receiver is not compatible with it or you might end up with corrupted files in your backup.

So, the only safe solution for the moment is to exclude the /var/log/lastlog from backup.

P.S.: here is a quick one-liner to detect sparse files, ordered by increasing ratio of real-data-size / total-sparse-size (lower means a huge sparse file with few data in it):

find / -type f -printf "%S\t%h/%f\n" 2>/dev/null | grep -v '^0[[:space:]]' | sort -k1g | head
Publié dans Uncategorized | Laisser un commentaire

Quick benchmark of Gzip and Gnupg throughput

Testing Gzip and Gnupg throughput for compressing and encrypting large files on-the-fly on Intel Core i5-6200U @2.30 GHz.

gzip vs. pigs:

  • gzip --best : 23,8 MB/s
  • pigz -p 2 --best : 51 MB/s
  • pigs -p 4 --best : 60 MB/s

gpg vs. gpg2:

  • gpg --encrypt : 19,5 MB/s
  • gpg2 --encrypt : 22,5 MB/s
  • gpg --encrypt --compress-level 0 : 56,5 MB/s
  • gpg2 --encrypt --compress-level 0 : 98,5 MB/s

Summary :

  • Gzip compression (with --best level) is the limiting factor: use pigz to parallelize and maximize compression throughput.
  • Gnupg performs by default a compression which is redundant if you feed him already compressed files: use pigz to parallelize and maximize the throughput AND disable Gnupg’s compression (--compress-level 0). Also, if you leave Gnupg’s compression on, it might not be able to parallelize it as pigz does, hence the benefit of performing the compression separately with pigz to maximize throughput.
  • gpg2 is almost two time faster than gpg when performing AES256 encryption: use gpg2 to maximize throughput.
  • The overall best « combo » for compressing and encrypting large files on-the-fly might be: pigz --best | gpg2 --encrypt --compress-level 0

Publié dans Uncategorized | Laisser un commentaire

« realm join » returns with error « Server not found in Kerberos database »

Trying to join an AD domain (Sama 4 AD DC) from a specific (Ubuntu 20.04) server would fail with a « Server not found in Kerberos database » error:

# realm join -U john.doe -v AD_EXAMPLE_NET
 * Resolving: _ldap._tcp.ad.example.net
 * Performing LDAP DSE lookup on: 10.100.100.2
 * Successfully discovered: ad.example.net
Password for john.doe:
 * Unconditionally checking packages
 * Resolving required packages
 * LANG=C /usr/sbin/adcli join --verbose --domain ad.example.net --domain-realm AD.EXAMPLE.NET --domain-controller 10.100.100.2 --login-type user --login-user john.doe --stdin-password
 * Using domain name: ad.example.net
 * Calculated computer account name from fqdn: SRV
 * Using domain realm: ad.example.net
 * Sending NetLogon ping to domain controller: 10.100.100.2
 * Received NetLogon info from: smb.ad.example.net                                                                                                      
 * Wrote out krb5.conf snippet to /var/cache/realmd/adcli-krb5-dN4Dz2/krb5.d/adcli-krb5-conf-JLqdZ0
 * Authenticated as user: john.doe@AD.EXAMPLE.NET
 ! Couldn't authenticate to active directory: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Server not found in Kerberos database)
adcli: couldn't connect to ad.example.net domain: Couldn't authenticate to active directory: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Server not found in Kerberos database)
 ! Insufficient permissions to join the domain

It turns out that this problem was that the IP address of the AD controller would not resolve back in reverse to it’s original DNS name!

The explanation of this problem (and solution) was found in the following post: https://aws.nz/best-practice/ad-join/

The solution is either to:

  • set a correct DNS reverse PTR that points back to the DNS name of the AD controller
  • or add the option rnds = false in the [libdefaults] setion in `/etc/krb5.conf`

Publié dans Uncategorized | Laisser un commentaire

Notes to self for future breakage involving `intel-microcode` updates

The latest update of intel-microcode package on Ubuntu broke some systems (mine in particular), and was the occasion to learn a bit more about initrd images and Ubuntu boot procedure.

So, here are some notes of things that may be useful for future problems.

The initrd image is mini-filesystem used to boot the OS. It’s a kind of trampoline root fileystem used by the kernel to initialize/activate/configure some hardware and/or features (e.g. full disk encryption, etc.)

Thus, this initrd contains binaries that might need to be updated. This process is automatically performed by Ubuntu when you install your package updates, with some packages triggering a rebuild of this initrd to incorporate the new and updated binaries.

So, when you install an update of intel-microcode, the initrd image is rebuilt to incorporate the new microcode files.

Ubuntu’s initrd seems a bit different from what I used to tinker with in my earlier days in the sense that they are not a CPIO archive, but a kind of aggregate incorporating the microcode binaries (that are applied first-most) and a « classical » initrd archive in CPIO format.

So, to introspect an initrd you’ll have to use something like binwalk to get the offset of the initrd archive, extract it, then you’ll have a « classical » CPIO image.

In the case of this microcode update, the initrd filesystem image was not the culprit, but rather theses microcode firmware/files.

I managed to recover my system by booting from a bootable USB live OS, mounting and chrooting into the broken system, reverting the intel-microcode to the previous version which in turn triggered a rebuild of the initrd image incorporating this previous microcode.

From issues/tickets I learned that I also could have disabled the loading of the microcode by appending the dis_ucode_ldr flag to the kernel from Grub.

Finally, here is the issue on GitHub: microcode-20200609 Release, at least 06-4e-03, hangs user’s system

Publié dans system | Laisser un commentaire

Configuring Linux VM with french MacBook Pro keyboard and VMware Fusion

At last, I found the right keyboard mapping for Linux VMs runing in VMware Fusion with a french MacBook Pro keyboard!

[FR]

$ sudo dpkg-reconfigure keyboard-configuration

Modèle du clavier :                       "PC générique 105 touches (intl)"
Pays d'origine du clavier :               "Français"
Disposition du clavier :                  "Français - Français (Macintosh)"
Touche destinée à se substituer à AltGr : "Touche Alt de gauche"
Touche compose :                          "Touche « logo » de gauche"

[EN]

$ sudo dpkg-reconfigure keyboard-configuration

Keyboard model:                "Generic 105-key (Intl) PC"
Country of origin of keyboard: "French"
Keyboard layout:               "French - French (Macintosh)"
Key to function as AltGr:      "Left Alt"
Compose key:                   "Left logo key"
Publié dans system | Laisser un commentaire

Siara Systems leaking CPE’s MAC addresses?

After a DSL/network outage, I launched a tcpdump on the ethernet interface to see if pppd was renegotiating the PPPoE session, and I saw all these LCP Term-Request frames that seems to come from other customers’ equipments (given the various source MAC addresses) and destined to a Siara Systems equipment (which I guess is a DSLAM, or core router, on which all these equipments are connected).

Is this an expected behavior to receive all these LCP Term-Request frames?

Here is a sample of unique/sorted frames captured during the DSL/network outage:

Continuer la lecture

Publié dans Uncategorized | 2 commentaires

Building a Netgear DM111Pv2 firmware without scfgmgr

Lately you found that your DSL modem/router was wide open, and you want to fix it.

Looking for a way to close this « back » door, I noticed that most of Netgear’s firmwares source codes are available from their site http://kb.netgear.com/app/answers/detail/a_id/2649.

Perhaps we could recompile a firmware with a « fixed » version of scfgmgr?

Looking in the DM111P firmware, you’ll find that the despicable scfgmgr is bundled as a binary in target.tar.bz2, so it cannot be edited and recompiled to get rid of it’s infamous behaviour.

However, perhaps we can simply neutralize it (in the target overlay) hoping that there won’t be any wrong side effects with the overall working of the firmware?

So, here is a try at rebuilding a firmware (version 2.00.27_WW) with a neutralized /usr/sbin/scfgmgr.

WARNING: Use these directives at your own risk. I’m not liable for any damage to your computer system or loss of data. blah blah blah …Insert here the usual disclaimers… blah blah blah.

Continuer la lecture

Publié dans network | Laisser un commentaire

Netgear DM111P backdoor?

After reading @elvanderb‘s tweet about a backdoor in his Linksys WAG200G, I checked my Netgear DM111P DSL modem (running firmware V2.00.27_WW) and found that TCP/32764 was opened and responding in a similar way.

The admin’s HTTP password is also exposed in clear through the scfgmgr process listening on TCP/32764:

$ perl -e 'print pack("(III)<", 0x53634d4d, 0x01, 0x00)' \
| nc 192.168.0.1 32764 \
| perl -n0e 'chomp;print"$_\n"' \
| grep ^http_

Output:

http_username=admin
http_password=***password*in*clear***
http_timeout=5

Continuer la lecture

Publié dans code, network | Laisser un commentaire

Problème de connexion à eurosportplayer.fr avec Safari ?

Si vous n’arrivez pas à vous connecter au player Eurosport avec Safari sur Mac OS X (vous retournez sans cesse sur la page de login), et que vous avez un nom/prénom avec un (ou des) accent(s), le problème peut venir de là.

Quand vous validez la page de login du player eurosport, le serveur envoi un cookie « PlayerAuth » avec une chaîne de la forme « guest=0&pseudo=Àlâïn Gérârd », avec votre nom/prénom encodé en UTF8.

Le problème est que Safari ne supporte pas les caractères non-ASCII dans les cookies, et donc le cookie n’est pas pris en compte par Safari, et il n’est pas retransmis avec les requêtes HTTP suivantes. On revient donc en boucle sur la page de login.

Pour contourner cela, il faut alors ouvrir une session avec Firefox, allez dans votre profil et changer le nom/prénom par des versions sans accents. Sauvegardez le profil, et vous devriez alors pouvoir vous connecter avec Safari.

http://stackoverflow.com/questions/5327341/strange-problem-with-cookies-in-safari-and-asp-net
http://stackoverflow.com/questions/1969232/allowed-characters-in-cookies

Publié dans code | Laisser un commentaire

Il est toujours aisé d’être logique. Il est presque impossible d’être logique jusqu’au bout.

Le mythe de Sisyphe, Albert Camus

Publié le par me | Laisser un commentaire