You are not logged in.

#1 2017-09-14 12:41:19

deep42thought
Administrator
From: Jena, Germany
Registered: 2017-06-17
Posts: 608

python-libnacl / libsodium

I'm out of ideas, so I post the status quo here:

python-libnacl fails to compile (during check()), which might be a libsodium issue.

Since the check is not verbose, failing with a segfault, I changed the PKGBUILD quite a lot:

diff --git a/python-libnacl/trunk/PKGBUILD b/python-libnacl/trunk/PKGBUILD
index 61f9b7e65d3..757bbc08b0f 100644
--- a/python-libnacl/trunk/PKGBUILD
+++ b/python-libnacl/trunk/PKGBUILD
@@ -16,6 +16,10 @@ source=("$pkgbase-$pkgver.tar.gz::https://github.com/saltstack/libnacl/archive/v
 sha512sums=('b4aa26b407855a73ac08ab5abc684f37338e960f534c04edb471634029388b9b11a79b02220eced99fcc94df7e1c05cbf4a0df48bb8b45b5f79e43cddfee790a')
 
 prepare() {
+  sed -i '
+#    17d
+    18,39d
+  ' libnacl-$pkgver/tests/unit/test_aead.py
   cp -r libnacl-$pkgver{,-py2}
 }
 
@@ -29,10 +33,13 @@ build() {
 
 check() {
   cd "$srcdir"/libnacl-$pkgver
-  python -m unittest discover --start-directory tests -v
 
-  cd "$srcdir"/libnacl-$pkgver-py2
-  python2 -m unittest discover --start-directory tests -v
+  python -m unittest tests/unit/test_aead.py -v
+
+#  python -m unittest discover --start-directory tests -v
+
+#  cd "$srcdir"/libnacl-$pkgver-py2
+#  python2 -m unittest discover --start-directory tests -v
 }
 
 package_python-libnacl() {

So only one test is being run (test_aead.py) and fails, but if its line 17 is also deleted, check() succeeds (the comment in sed).
line 17 is

clear1 = box.decrypt(ctxt, len(aad))

, which gives me the impression, that it might be libsodium's fault. However, I recompiled that and it still failes.

Any suggestions (or fixes)?

Cheers,
deep42thought

Offline

#2 2017-09-14 13:13:39

andreas_baumann
Administrator
From: Zurich, Switzerland
Registered: 2017-08-10
Posts: 823
Website

Re: python-libnacl / libsodium

I actually get an "Illegal instruction" which sounds more like python itself has an issue (or the C part of libnacl):

test_gcm_aead (unit.test_aead.TestAEAD) ... /startdir/PKGBUILD: line 30:   337 Illegal instruction     (core dumped) python -m unittest discover --start-directory tests -v
Dump of assembler code for function crypto_aead_aes256gcm_beforenm:
   0xf6de2460 <+0>:     sub    $0x9c,%esp
   0xf6de2466 <+6>:     pxor   %xmm0,%xmm0
   0xf6de246a <+10>:    mov    0xa4(%esp),%edx
   0xf6de2471 <+17>:    mov    0xa0(%esp),%eax
   0xf6de2478 <+24>:    movdqu (%edx),%xmm4
   0xf6de247c <+28>:    movaps %xmm4,0x10(%eax)
   0xf6de2480 <+32>:    shufps $0x10,%xmm4,%xmm0
   0xf6de2484 <+36>:    movdqa %xmm4,%xmm2
   0xf6de2488 <+40>:    movaps %xmm4,0x60(%esp)
   0xf6de248d <+45>:    pxor   %xmm0,%xmm2
   0xf6de2491 <+49>:    movdqu 0x10(%edx),%xmm3
   0xf6de2496 <+54>:    shufps $0x8c,%xmm2,%xmm0
=> 0xf6de249a <+58>:    aeskeygenassist $0x1,%xmm3,%xmm1
  0xf6de24a0 <+64>:    movaps %xmm3,0x70(%esp)
   0xf6de24a5 <+69>:    pshufd $0xff,%xmm1,%xmm1
   0xf6de24aa <+74>:    movaps %xmm3,0x20(%eax)
   0xf6de24ae <+78>:    pxor   %xmm0,%xmm1

Doesn't look very i686is to me, http://www.felixcloutier.com/x86/AESKEYGENASSIST.html

So the question is, how is this module built? Does some assembly optimization creep in?

Last edited by andreas_baumann (2017-09-14 13:14:06)

Offline

#3 2017-09-14 13:18:05

andreas_baumann
Administrator
From: Zurich, Switzerland
Registered: 2017-08-10
Posts: 823
Website

Re: python-libnacl / libsodium

Mmh. My Intel(R) Core(TM)2 CP has no AES extensions? AES extensions are around since 2008, weird.

Ah, so even my 64-bit build fails, too old machine. False alarm. :-)

Last edited by andreas_baumann (2017-09-14 13:22:29)

Offline

#4 2017-09-14 14:44:11

deep42thought
Administrator
From: Jena, Germany
Registered: 2017-06-17
Posts: 608

Re: python-libnacl / libsodium

the 64 bit build was failing on one of my boxes, too - see FS#55078
But the error was somewhat different.

Offline

#5 2017-09-14 14:46:36

andreas_baumann
Administrator
From: Zurich, Switzerland
Registered: 2017-08-10
Posts: 823
Website

Re: python-libnacl / libsodium

Aha. Also a CPU lacking AES support. I wonder if AES comes from libsodium or where from. Seems
to big badly ported without checking the capabilities of the CPU.

Offline

#6 2017-09-14 15:14:11

andreas_baumann
Administrator
From: Zurich, Switzerland
Registered: 2017-08-10
Posts: 823
Website

Re: python-libnacl / libsodium

Testing both with python2 and pyhton3 results in a programmed abort in:

#0  0xf7792db9 in __kernel_vsyscall ()
#1  0xf75c0142 in raise () from /usr/lib/libc.so.6
#2  0xf75c18d8 in abort () from /usr/lib/libc.so.6
#3  0xf6db51fc in crypto_aead_aes256gcm_decrypt_detached_afternm () from /usr/lib/libsodium.so
#4  0xf6db6bf3 in crypto_aead_aes256gcm_decrypt_afternm () from /usr/lib/libsodium.so
#5  0xf6db6feb in crypto_aead_aes256gcm_decrypt () from /usr/lib/libsodium.so

So this looks really like a libsodium problem.

Offline

#7 2017-09-14 18:57:31

andreas_baumann
Administrator
From: Zurich, Switzerland
Registered: 2017-08-10
Posts: 823
Website

Re: python-libnacl / libsodium

I recompiled libsodium and python-libnacl inside the i686 staging chroot, then I run the unit tests.

I get:

#3  0xf6d3ff11 in sodium_misuse () at sodium/core.c:188
#4  0xf6d7dee2 in crypto_aead_aes256gcm_decrypt_detached_afternm (m=0xf697ff50 "", nsec=0x0, c=0x0, 
    clen=373509687568, mac=0xf6e98910 "\006", ad=0x0, adlen=21320452328, npub=0x0, ctx_=0xffd58990)
    at crypto_aead/aes256gcm/aesni/aead_aes256gcm_aesni.c:666
#5  0xf6d82134 in crypto_aead_aes256gcm_decrypt_afternm (m=0xf697ff50 "", mlen_p=0x0, nsec=0x0, c=0x0, 
    clen=373509687584, ad=0x0, adlen=21320452328, npub=0x0, ctx_=0xffd58990)
    at crypto_aead/aes256gcm/aesni/aead_aes256gcm_aesni.c:811

So it's a programmed 'abort' in:

#2  0xf75ec8d8 in abort () from /usr/lib/libc.so.6
(gdb) up
#3  0xf6d3ff11 in sodium_misuse () at sodium/core.c:188
188         abort();
(gdb) list
183             if (sodium_crit_leave() == 0 && handler != NULL) {
184                 handler();
185             }
186         }
187     /* LCOV_EXCL_START */
188         abort();
189     }

...

665         if (clen > crypto_aead_aes256gcm_MESSAGEBYTES_MAX) {
666             sodium_misuse(); /* LCOV_EXCL_LINE */
667         }

So the library is triggered with an illegal length.

(gdb) p clen
$1 = 373509687568
(gdb) p crypto_aead_aes256gcm_MESSAGEBYTES_MAX
$2 = 4294967279

If I add a print in the test I get:

        box2 = libnacl.aead.AEAD(box.sk).useAESGCM()
        print( len(aad) )
        clear1 = box.decrypt(ctxt, len(aad))
test_gcm_aead (unit.test_aead.TestAEAD) ... 4

So, the data gets distorted when being passed down.

Offline

#8 2017-09-14 20:14:17

andreas_baumann
Administrator
From: Zurich, Switzerland
Registered: 2017-08-10
Posts: 823
Website

Re: python-libnacl / libsodium

Here I start to smell some parameter passing trouble in the ctypes wrapper in __init__.py:

    ret = nacl.crypto_aead_aes256gcm_decrypt(
        m, mlen,
        None,
        ctxt, ctypes.c_ulonglong(len(ctxt)),
        aad, ctypes.c_ulonglong(len(aad)),
        nonce, key)
    if ret:

Debugging there with pdbtest and pdb I see it crashes in this call from Python to C.

I hope ctypes.c_ulonglong is really unsigned long long on 32-bit and 64-bit (should be both 64 bit, right?) and matching the prototype in sodium's
sodium/crypto_aead_aes256gcm.h:

int crypto_aead_aes256gcm_decrypt(unsigned char *m,
                                  unsigned long long *mlen_p,
                                  unsigned char *nsec,
                                  const unsigned char *c,
                                  unsigned long long clen,
                                  const unsigned char *ad,
                                  unsigned long long adlen,
                                  const unsigned char *npub,
                                  const unsigned char *k)

Offline

#9 2017-09-14 20:19:31

andreas_baumann
Administrator
From: Zurich, Switzerland
Registered: 2017-08-10
Posts: 823
Website

Re: python-libnacl / libsodium

Oh! There are also pointers. So on a 32-bit system a 'unsigned long long *' is a 32-bit pointer to a 64-bit value,
on 64-bit it is a 64-bit pointer to a 64-bit value. So the ' unsigned long long *mlen_p,' shifts the
stack by 4 bytes thus creating funny values for 'clen', given those values are passed on the stack
(long time no ABI for me)?

Last edited by andreas_baumann (2017-09-14 20:20:54)

Offline

#10 2017-09-14 20:25:41

andreas_baumann
Administrator
From: Zurich, Switzerland
Registered: 2017-08-10
Posts: 823
Website

Re: python-libnacl / libsodium

Aha, got it, see:

https://docs.python.org/2/library/ctypes.html

15.17.1.9. Passing pointers (or: passing parameters by reference)

We have to fix __init__.py like that:

...
   ret = nacl.crypto_aead_aes256gcm_decrypt(
        m, ctypes.byref(mlen),
        None,
        ctxt, ctypes.c_ulonglong(len(ctxt)),
        aad, ctypes.c_ulonglong(len(aad)),
        nonce, key)
...
    ret = nacl.crypto_aead_chacha20poly1305_ietf_decrypt(
        m, ctypes.byref(mlen),
        None,
        ctxt, ctypes.c_ulonglong(len(ctxt)),
        aad, ctypes.c_ulonglong(len(aad)),
        nonce, key)
...

Last edited by andreas_baumann (2017-09-14 20:45:42)

Offline

#11 2017-09-14 20:26:45

andreas_baumann
Administrator
From: Zurich, Switzerland
Registered: 2017-08-10
Posts: 823
Website

Re: python-libnacl / libsodium

BTW: I like verbose debugging like this, even if you risk to make a fool of yourself. :-)
But it shows the process of how to find those kind of bugs and serves as a future reference to similar problems.

Offline

#12 2017-09-14 20:40:38

andreas_baumann
Administrator
From: Zurich, Switzerland
Registered: 2017-08-10
Posts: 823
Website

Re: python-libnacl / libsodium

Offline

#13 2017-09-15 12:33:39

deep42thought
Administrator
From: Jena, Germany
Registered: 2017-06-17
Posts: 608

Re: python-libnacl / libsodium

Thanks for your work.
Looks like we should wait some time for upstream to fix this (looks, like python-libnacl doesn't block anything atm).
Though, I would patch it myself, if it becomes necessary.

btw: I'm in favour of "verbose debugging", too, however, I'm afraid, I can't compete with you ;-)

Offline

#14 2017-09-15 12:52:44

deep42thought
Administrator
From: Jena, Germany
Registered: 2017-06-17
Posts: 608

Re: python-libnacl / libsodium

ah, what the heck - I'm too impatient - I'll just create a fix and remove it again, when upstream fixes this issue.

Offline

Board footer

Powered by FluxBB