Building whole Postgres with LLVM

Most posts I could find are about building just the libjit with LLVM/CLANG.

So here's my 5 cents on the subject.

I'll be using [V]irtual[M]achine running Debian [12] Testing, with basic building pakages already installed : 

binutils make gcc-12 libtool libss-dev libreadline-dev libxslt1-dev libxml2-dev libpth-dev zlib1g-dev

My usual configure parameters for GCC :

./configure --prefix=/media/pg --with-pam --with-libxml --with-libxslt --with-openssl --with-pgport=5433 --with-llvm LLVM_CONFIG=llvm-config-15 CLANG=clang-15 CFLAGS="-s -O3 -march=native -mtune=native -fno-plt -flto=4" CXXFLAGS="-s -O3 -march=native -mtune=native -fno-plt -flto=4" LDFLAGS="-Wl,-O3,--as-needed,-flto=4"

Packages to add :

libllvm15 lld-15 llvm-15 llvm-15-dev llvm-15-linker-tools llvm-15-runtime llvm-15-tools

clang-15 libclang-common-15-dev libclang-cpp15 libclang1-15

I won't be using https://libcxx.llvm.org since it's not fully baked yet :

C++11 - Complete
C++14 - Complete
C++17 - In Progress
C++20 - In Progress
C++23 - In Progress
C++2c - In Progress

 

My goal is to check if CLANG can handle Postgres sources ;]

So here's my configure for LLVM :

./configure --prefix=/media/pg13 --with-pam --with-libxml --with-libxslt --with-openssl --with-pgport=5433 --with-llvm LLVM_CONFIG=llvm-config-15 CLANG=clang-15 CC=clang-15 CXX=clang-15 CFLAGS="-fuse-ld=lld-15 -s -O3 -march=native -fno-plt -mtune=native -flto" CXXFLAGS="-fuse-ld=lld-15 -s -O3 -march=native -flto -fno-plt -mtune=native" LD=lld-15 AR=llvm-ar-15 RANLIB=llvm-ranlib-15 STRIP=llvm-strip-15

.. and result

configure: using compiler=Debian clang version 15.0.6
configure: using CFLAGS=-Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -Wno-unused-command-line-argument -Wno-compound-token-split-by-macro -Wno-deprecated-non-prototype -fuse-ld=lld -s -O3 -march=native -fno-plt -mtune=native -flto
configure: using CPPFLAGS= -D_GNU_SOURCE -I/usr/include/libxml2
configure: using LDFLAGS= -L/usr/lib/llvm-15/lib  -Wl,--as-needed
configure: using CXX=clang
configure: using CXXFLAGS=-Wall -Wpointer-arith -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fuse-ld=lld -s -O3 -march=native -flto -fno-plt -mtune=native
configure: using CLANG=clang
configure: using BITCODE_CFLAGS= -fno-strict-aliasing -fwrapv -Xclang -no-opaque-pointers -Wno-unused-command-line-argument -Wno-compound-token-split-by-macro -Wno-deprecated-non-prototype -O2
configure: using BITCODE_CXXFLAGS= -fno-strict-aliasing -fwrapv -Xclang -no-opaque-pointers -O2

 

Using both "-fuse-ld=lld" and "LD=lld" might be redundant, but without at least one of them - even with setting all the llvm/clang tools - clang will use default linker - ld.

 

Even running 'make -d V=1' you won't see which linker clang calls in the end, bummer.

I had to run 'make V=1 &>../zbuild.log &' in background and watch processes in htop. 

 

Everything build nicely for me, resulting binaries a tad smaller than those build with GCC-12.

Postgres - I've tested sources vesion 13.10 and 13.11, server runs, so far no errors.

Is it faster ? maybe ;]

Wildfly 26 - dump http request/reponse with body

I've been messing around with WF26 at work lately.

Sadly good 'ol

  <system-properties>
    <property name="org.apache.cxf.logging.enabled" value="true"/>
    <property name="com.sun.xml.ws.transport.http.HttpAdapter.dump" value="true"/>
  </system-properties>

doesn't work anymore :(

Most i could find is posts about io.undertow.server.handlers.RequestDumpingHandler

Based on https://issues.redhat.com/browse/UNDERTOW-854

I got WF26 to dump Response + body.

In standalone.xml - in  section <subsystem xmlns="urn:jboss:domain:undertow:12.0"

  <server name="default-server">
..
      <host name="default-host" alias="localhost">
          <location name="/" handler="welcome-content"/>
..
          <filter-ref name="request-dumper"/>
          <filter-ref name="store-response"/>
      </host>
  </server>
  <servlet-container name="default">
..
  </handlers>
  <filters>
    <filter name="request-dumper" class-name="io.undertow.server.handlers.RequestDumpingHandler" module="io.undertow.core"/>
    <filter name="store-response" class-name="io.undertow.server.handlers.StoredResponseHandler" module="io.undertow.core"/>
  </filters>

Adding EagerFormParsingHandler gave me errors - so skip that. 

Ta-ducking-da

2023-03-17 09:30:52,700 INFO  [io.undertow.request.dump] (default task-1)
----------------------------REQUEST---------------------------
               URI=/server-example-rest/rest/test/
 characterEncoding=null
     contentLength=70
       contentType=[application/json;charset=UTF-8]
            header=Connection=Keep-Alive
            header=Accept=application/json;charset=UTF-8,application/xml;charset=UTF-8
            header=Authorization=Basic xxxxxxxxxxxx
            header=Content-Type=application/json;charset=UTF-8
            header=Content-Length=70
            header=User-Agent=Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)
            header=Host=10.20.30.40:8443
            locale=[]
            method=POST
          protocol=HTTP/1.1
       queryString=
        remoteAddr=test/10.20.30.41:55061
        remoteHost=test
            scheme=https
              host=10.20.30.40:8443
        serverPort=8443
          isSecure=true
--------------------------RESPONSE--------------------------
     contentLength=54
       contentType=application/json; charset=UTF-8
            header=Expires=0
            header=Connection=keep-alive
            header=Cache-Control=no-cache, no-store, must-revalidate
            header=Pragma=no-cache
            header=Content-Type=application/json; charset=UTF-8
            header=Content-Length=54
            header=Date=Fri, 17 Mar 2023 08:30:52 GMT
            status=200
body=
{"errors":[],"data":[{"something":"Test","example"}]}

Sending e-mail via python script - version 2, simplified

Usage :  ./script.sh   "E-Mail Subject"   /var/log/dmesg.log

This version puts contents of /var/log/dmesg.log in to an email message directly


script.sh :

#!/usr/bin/env python

import sys
from os import path
from smtplib import SMTP
from email.mime.text import MIMEText
from email.MIMEBase import MIMEBase
from email import Encoders

me = 'author@server.com'
you = 'recipient@example.com'
cc = 'other@carbon.copy'
bcc = 'nsa@gov.us'

fp = open(sys.argv[2], 'rb')
msg = MIMEText(fp.read(), 'plain', _charset="UTF-8")
fp.close()

msg['Subject'] = sys.argv[1]
msg['From'] = me
msg['To'] = you
msg['Cc'] = cc
msg['Bcc'] = bcc

sndm = SMTP()
sndm.set_debuglevel(1)
sndm.connect('smtp.server.com', 25)
sndm.starttls()
try :
    sndm.esmtp_features['auth'] = 'DIGEST-MD5 LOGIN'
    sndm.login('author@server.com', 'Pas5_W0rd')
    sndm.sendmail(me, [you,cc,bcc], msg.as_string())
    sndm.quit()
except Exception, e:
    print e

PostgreSQL - Fixing Sequences

Tested on version 8.4

-- DROP FUNCTION public.SeqFix();
CREATE OR REPLACE  Function public.SeqFix() Returns void AS $$
DECLARE LIST record; MaxIDValue INTEGER; CurrentValue iNTEGER;
BEGIN
    FOR LIST iN Select table_schema, table_name, column_name, split_part(column_default,'''' ,2) AS seqname FROM information_schema.columns
                        Where table_catalog=current_database() AND column_default iS NOT NULL AND Position('nextval' iN column_default) =1 order by 1,2,3
    LOOP
        EXECUTE 'SELECT MAX(' || LIST.column_name || ') FROM ' || LIST.table_schema || '.' || LIST.table_name
        iNTO MaxIDValue;
        EXECUTE 'SELECT COUNT(*) FROM information_schema.sequences WHERE sequence_catalog=current_database() AND sequence_schema='''||LIST.table_schema||'''
             AND sequence_name='''||split_part(LIST.seqname, '.',2)||''''
        INTO CurrentValue;
        IF CurrentValue = 0 THEN
            RAISE WARNING E'?? SEQ ::\t%\t :: does not exists  ??', LIST.seqname ;
        ELSE
            EXECUTE 'SELECT last_value FROM ' || LIST.seqname
            INTO CurrentValue;
            IF CurrentValue < MaxIDValue THEN
                RAISE WARNING E'!! SEQ :: \t% = %\t<\tMAX(%.%.% = %) ', LIST.seqname, CurrentValue, LIST.table_schema, LIST.table_name, LIST.column_name, MaxIDValue;
               -- PERFORM pg_catalog.setval(LIST.seqname, MaxIDValue+1, false);
            END IF;
        END IF;
    END loop;
END;
$$ LANGUAGE plpgsql;

SELECT public.SeqFix();

Jboss 7, HTTPS and EC browser support

While messing around with Jboss AS 7.1 at work I've made some discoveries.

1. JDK7 supports EC key algorithm

./jdk1.7.0_03/bin/keytool -v -genkey -alias https -keyalg EC -keystore /opt/jboss/https.keystore -keysize 409 -validity 730 -dname "CN=*.example.com, OU=TW, O=Home, L=OL, ST=WiM, C=PL" -storepass s0m3p15s -keypass S0m3p15s

max size is -keysize 571

2. Firefox (14.0a1) also supports EC .. partially - only keys generated with -keysize 256 and -keysize 384

3. Opera (12 alpha) doesn't support EC keys at all :(.

---

To use keys generated with keytool

keytool -v -genkey -alias https -keyalg DSA -keystore /opt/httpdsa.keystore -keysize 1024  -validity 730 -dname "CN=my.domain.com, OU=Lap, O=Home, L=City, ST=State, C=UK" -storepass s0m3Pa5s -keypass s0m3Pa5s

Subsytem must be set : native="false"

<subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host" native="false">
  <connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http"/>
  <connector name="https" protocol="HTTP/1.1" scheme="https" socket-binding="https" secure="true">
     <ssl name="ssl" key-alias="https" password="s0m3Pa5s" certificate-key-file="/opt/httpdsa.keystore" protocol="TLSv1" verify-client="false"/>
  </connector>
</subsystem>

---

To use keys generated with openssl
Subsytem must be set : native="true"

<subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host" native="true">
  <connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http"/>
  <connector name="https" protocol="HTTP/1.1" scheme="https" socket-binding="https" secure="true">
      <ssl certificate-key-file="/opt/https-rsa4key.pem" protocol="TLSv1" verify-client="false" certificate-file="/opt/https-rsacert.pem" keystore-type="PKCS12" truststore-type="PKCS12"/>
  </connector>
</subsystem>

---

Generating RSA keys with password

openssl genrsa -des3 -out https-rsa4key.pem 4096
openssl req -new -key https-rsa4key.pem  -out https.csr
openssl x509 -req -days 720 -in https.csr -signkey https-rsa4key.pem -out https-rsacert.pem

To test if browser can handle keys types/size - type password (set at runinng `openssl genrsa...`) when asked

openssl s_server -www -accept 443 -cert https-rsacert.pem -key https-rsa4key.pem

To use in Jboss AS7 standalone.xml (don't froget password="S0m3Pa5s"):

<ssl password="S0m3Pa5s" certificate-key-file="/opt/https-rsa4key.pem" protocol="TLSv1" verify-client="false" certificate-file="/opt/https-rsacert.pem" keystore-type="PKCS12" truststore-type="PKCS12"/>

---

I've managed to get Jboss AS7.1 to start with openssl DSA keys

openssl dsaparam -out dsaparam 1024
openssl gendsa -out https-dsa.pem dsaparam
openssl req -new -key https-dsa.pem -out https.csr
openssl x509 -req -days 720 -in https.csr -signkey https-dsa.pem -out https-dsacert.pem

To test if browser can handle keys types/size

openssl s_server -www -accept 443 -cert https-dsacert.pem -key https-dsa.pem

and in standalone.xml

<ssl certificate-key-file="/opt/https-dsa.pem" protocol="TLSv1" verify-client="false" certificate-file="/opt/https-dsacert.pem" keystore-type="PKCS12" truststore-type="PKCS12"/>

Linux JDK 6 installation - the hard way.

To clarify few things :
- I'm not a big fan of java (or any other bloated framework, especially if major versions are not backwards compatible)
- I hate installers that do things that we could really live without, or without users consent (eyecandy, toolbars, auto updaters etc.)

Last week at work I wanted to get JDk6 for JBoss based project. Since there are no tar.gz/tgz versions for 6u31 (like for JDK7), I dl'ed the i586-bin installer. My office server is openSuse 12.1 x86_64. I tried running the installer in different combinations.

chmod 755 jdk-6u31-linux-i586.bin
./jdk-6u31-linux-i586.bin

sh jdk-6u31-linux-i586.bin

Outcome was mostly the same :

jdk-6u31-linux-i586.bin: line 113: ./install.sfx : Permission denied

Installer is mix of shell script - few hundred lines at beginning of the file, and a self extracting binary.
To determinate where the script ends and sfx starts run

less -N jdk-6u31-linux-i586.bin

or

less jdk-6u31-linux-i586.bin

and press = to see line numbers currently seen on screen.

exit 0
^?ELF^A^A^A^@^@^..............
jdk-6u31-linux-i586.bin lines 144-189/327000 byte 5779/85292206 0%

To get the sfx part run

tail -n +189 jdk-6u31-linux-i586.bin > install.sfx

To run (uncompress) the self extracting binary

chmod 755 install.sfx
./install.sfx

I thought that I got what I needed, but at start JBoss 6 spitted out weired error. So I tried to check if everything is fine with the JDK.

cd jdk1.6.0_31
./bin/java -version
Error occurred during initialization of VM
java/lang/NoClassDefFoundError: java/lang/Object

After some googlin' for a reason to what could be the cause of the problem, I found a forum post suggesting that rt.jar could be missing.

find ./ -name rt.*

returned :

./jre/lib/rt.pack

In bin folder I found unpack200

./bin/unpack200 --help

So I had the tool and means, next I had to find what needed to be unpacked

find ./ -name *.pack

returned :

./lib/tools.pack
./jre/lib/charsets.pack
./jre/lib/jsse.pack
./jre/lib/deploy.pack
./jre/lib/javaws.pack
./jre/lib/plugin.pack
./jre/lib/rt.pack
./jre/lib/ext/localedata.pack

So let's get to it

./bin/unpack200  ./lib/tools.pack  ./lib/tools.jar
.
.
./bin/unpack200  ./jre/lib/ext/localedata.pack  ./jre/lib/ext/localedata.jar

After that everything worked fine.
If any1 want's to take a look at the script part of the installer :

head -n 188 jdk-6u31-linux-i586.bin > install.sh

Manual PostgreSQL instalation on Windows

This post is for those unlucky ppl who for some dumb reason had to install Postgres on Windows, and had no luck with it. Most common problems I came across are :
1. installation finishes but database isn't initialized - error says that libintl-8.dll is missing
2. installation stops at the beginning - VisualC Redist Setup crashes - most common on Win7

Other reasons to install PG by hand is that even when using installer with command line options, you can't get the result you wanted, like database encoding, service user etc.

What will need is :
1. Postgres binaries
2. Ntrights.exe

Let's copy all files from postgres zip to c:\pgsql.
To create service-user - in cmd as admin

net user pgsql S0m3Pa5sW0rd /add

Now to properly configure service-user

ntrights.exe -u pgsql +r SeServiceLogonRight
ntrights.exe -u pgsql -r SeInteractiveLogonRight
wmic.exe USERACCOUNT WHERE "name='pgsql'" SET PasswordExpires=FALSE

Service-user needs full control over c:\pgsql

cacls "c:\pgsql" /T /E /G "pgsql":F

To properly initialize database we need to run cmd as service-user pgsql. Still as admin run

runas /user:pgsql cmd
typ password when asked S0m3Pa5sW0rd

Now in new cmd window

cd c:\pgsql
mkdir data
cd bin
initdb.exe -D ../data -E LATIN2 --locale="Czech, Czech Republic"
exit

Now back in admins cmd

cd c:\pgsql\bin
pg_ctl.exe register -N PG84 -U pgsql -P S0m3Pa5sW0rd -D "c:\pgsql\data" -w

On Windows Vista and newer you need to uncomment last line in c:\pgsql\data\pg_hba.conf, since those versions have ipv6 support turned on by default - to test if your system qualifies

ping ::1
echo %ERRORLEVEL%

if echo returns 0, change last line in pg_hba.conf like so (remove # at the beginning of the line)

host    all     all     ::1/128      trust

To start the server - run services.msc , find and start PG84.