(Català) Accelerar el temps de càrrega de les aplicacions RubyOnRails
Mar 23rd
Sorry, this entry is only available in Català.
Linux desktops for windows users
Feb 28th
2012 business presentation
Feb 6th
Presentation of our products in the Primer Simpòsium that took place in Vilafranca del Penedès, Barcelona, the 3th of February of 2012. The contents are in Catalan.
Expeditious P2V method
Jan 20th
This is the P2V method that we used to convert a physical Linux server to a virtual machine running on KVM. It is a quick virtualization method that dumps the entire disk over the network to a new host machine. Ideally the origin server should be stopped in order to boot in rescue mode, thus making the copy with unmounted file systems. In our scenario it was not possible to halt the physical server, and this procedure did the trick.
Copy the disc
At the destination server start netcat listening to a port (in this example 43333), redirecting all its input to a file. If you connect using SSH do it in a screen so you can close the session.
At the origin server use dd to dump its disk. It is then compressed and sent to the netcat at port 43333 waiting with the command above. Gzip compresses the stream and reduces the amount of data to be transferred over the net. Our disk was 72 GB and took 4 hours to 10 Mbps, via the Internet.
Note: The syntax of nc depends on the package you have installed. In this example it is netcat-openbsd.
Review the file systems
Now you have a disk image named disk.raw in the desination server. But the origin server was accessing its filesystems while dd was reading: you should run an fsck of them. If you are lucky the filesystem should not be badly damaged.
In Linux you can configure a file to be treated as a device assigning it to a loop device. We check the next free loop device.
/dev/loop0
Assign disk.raw to loop0 and it can be accessed using /dev/loop0
Now read loop0 partitions and create appropriate devices to access it, we use kpartx.
The partitions are accessible via /dev/mapper/loop0p[n]. Do an fsck.
fsck.ext3 -f /dev/mapper/loop0p2
...
At this point you can mount the partitions to verify that the files are indeed there and maybe fix the device names in /etc/fstab.
Before starting the virtual machine, you need to undo this process to prevent damage to the filesystem.
losetup -d /dev/loop0
Create the virtual machine
Create a KVM virtual machine with disk.raw attached and try booting.
It may not boot because the device name of the disc may have changed and the kernel does not find the root filesystem. Using the KVM console you will have to fix the line in GRUB or LILO, correcting the root = option in the kernel line. When it finally boots remember to fix the boot loader and /etc/fstab.
In our case the physical device /dev/cciss/c0d0p[n] became /dev/hda[n] in the VM.
Wrapping up
You must configure the new virtual network interface and verify that the services work properly.
During the virtualization the data on the original physical server has been updated and it must be synchronized before any real server change. So stop the service within a maintenance window, sync the data and then you can change DNS zones or whatever. Downtime has been minimized.
PEPPOL’s technical issues
Nov 16th
Here I will explain the technical aspects of sending and receiving documents via PEPPOL , which are the initials of “Pan-European Public Procurement OnLine.” It is a pilot project on public procurement to exchange electronic documents with the European public administration. This technical information is public, but is scattered around the Internet in forums and documents of all kinds.
Architecture
The three elements of the architecture are the AP, the SMP and an SML.
AP = Access Point. Is the service to receive documents. For example ap.b2brouter.com
SMP = Metadata Service Publisher. Is the service to publish data from participants. For example it tells you to which AP you have to send certain types of documents. An example of SMP is smp.b2brouter.com
SML = Locator Service Metadata. A DNS server that, given a recipient, tells you which SML has its data. The DNS domain zone is peppolcentral.org.
Certificates
Participants are identified and authorized via PKI X.509. The pilot project has a root CA and three intermediate CAs where “PEPPOL Test Root CA” signs the intermediate CAs.
It takes two certificates on the AP and the SMP. There is a third certificate, called STS, which is not used. These certificates must be requested if you want to participate in PEPPOL pilot .
- The AP certificate is signed by the “PEPPOL Access Point Test CA”. Used to send and receive from other AP.
- The SMP certificate is signed by the “Service Metadata Publisher PEPPOL Test CA”. Used to publish DNS entries in the SML.
- The STS is signed by the “Test CA PEPPOL Security Token Service” and it is currently useless.
Send documents
In order to send something clearly we need to know the address of the recipient, and in the case of Peppol, the address of the recipient is a Participant ID, Process ID and a Document ID.
To send a document the first step is to find out which is the AP of the recipient. To do this you take the address of the recipient (participant, process and document) and use the SML and an SMP:
- The Participant ID becomes a DNS zone using an MD5 hash, a prefix “B-” and a suffix. For example, Participant ID “9912: esb63276174” becomes “B-ae2866398fd1d4c0d35343e8464a5258.iso6523-actorid-upis.sml.peppolcentral.org”
- The DNS zone is resolved and you get an IP. This is the IP of the SMP, which is the place where information concerning the Participant ID is. In our example, the zone resolves to CNAME smp.b2brouter.com and to the IP of then SMP.
- Finally, HTTP is used to read the data of the Participant ID that the SMP has published in the URL http://B-ae2866398fd1d4c0d35343e8464a5258.iso6523-actorid-upis.sml.peppolcentral.org/complete/iso6523-actorid-upis::9912:esb63276174
The XML document returned by the SMP contains the reference to the AP that we seek. Within the XML there is the list of documents accepted by the participant, and for each document the list of processes. For example, to send an invoice “urn: oasis: names: specification: UBL: schema: xsd: Invoice-2:: Invoice process (…)” to the process “urn: www.cenbii.eu:profile:bii04:ver1 .0” in this case the AP would be ap.b2brouter.com .
Once you know the URL of the destination AP just send the document using what they call “Secure Trusted Asynchronous Reliable Transport” or just START, which defines how the AP must use these 5 technologies to communicate:
- SOAP 1.1 – Simple Object Access Protocol.
- WS-Security 1.1 – a flexible and feature-rich extension to SOAP web service to apply to security.
- WS-Transfer – transfer in defining the specification of an XML representation of an WS-addressable resource – as a standard approach to accessing the message channels.
- WS-ReliableMessaging 1.1 – SOAP in Protocol That Allows messages to be delivered reliably between distributed applications.
- SAML 2.0 – Security Assertion Markup Language – an XML-based open standard for exchanging authentication and authorization data between security domains.
Luckily the reference implementation is in Java and Java already has all the necessary WS-* libraries.
Java Keystores
It is very important to be clear about which certificates to use in each element of the architecture:
- Both the client and the server need the AP certificate signed by the “Test CA PEPPOL Access Point” to send and receive. We will create a keystore, let’s name it ap.jks
- The SMP requires a certificate signed by the “Service Metadata Publisher PEPPOL Test CA” to update the DNS entries in the SML. We will create another keystore, let’s name it smp.jks
- We also need an additional keystore (the trust store) with the certificates that we trust (the ones from Peppol CAs). Let’s name it peppol-cacerts.jks
To create keystores you can use Portecle because Java’s keytool is rather cryptic. Things to consider:
- Use the alias “1” fot the private key.
- Use “peppol” as password for both the keystore and the private key.
- Use the “Import CA Reply” , with the right mouse button over “1”, to add your certificate signed by the CA.
- The AP server keystore (ap.jks) should have only one certificate because it just uses the first it founds.
Implementations
The implementations that we have used are all open source:
- Server Access Point – Java web application, Netbeans project
- Client to send to an Access Point – Java application, Netbeans project
- Metadata Server Service Publisher – Java web application, Maven Project
- Client to manage entries in the Metadata Service Publisher – Java application, Maven Project
- Client to manage entries in the Metadata Service Locator – Java desktop application, Maven Project
We’ve also developed some auxiliary tools that can be found at github.
Avoid Postfix’s backscatter with Exchange as backend
Nov 14th
Backscatter? Well, this happens when there is an spam filter in front of an MTA, but the filter don’t know which mailboxes do really exist in the MTA and which ones not.
The procedure explained here is about how to tell a Postfix filter which are the recipients of an Exchange MTA and thus avoid becoming a backscatter
What is Backscatting?
When a server receives an email for an address that does not exist, it should reject it during the SMTP conversation, when the origin server is still connected. Thus the origin can inform the real sender about the problem. Otherwise, if the server accepts an email which is unable to deliver, it must generate a bounce that will be sent to the mail “From” header. Spammers manipulate the sender header of messages, and a misconfigured server ends up sending spam bounces. This type of spam is called Backscatter.
Postfix in front of an Exchange
In our scenario we have a Postfix MTA that receives mail, checks it with for spam, virus and other policies, and finally sends it to a Microsoft Exchange. Since this Postfix MTA does not have mailboxes, we were accepting all mail addressed to our domains. After a while our IP was added to a blacklist for Backscatting.
To fix this we need to know which mailboxes exist in Exchange, in order to reject emails to recipients that does not exist.
The easy part is the Postfix setup, at main.cf we add relay_recipient_maps, to check recipients.
relay_recipient_maps = hash:/etc/postfix/relay_recipients
we create /etc/postfix/relay_recipients with mailbox list, like this:
bar@example.com OK
we need to do postmap and reload postfix
/etc/init.d/postfix reload
Keep the list updated
We are not generating Backscatting any more, but must keep the mailbox list up to date. That list is stored in an ActiveDirectory, to get it we used this VBS script, with some changes:
' file virtual.txt
'
' Ferdinand Hoffmann & Patrick Koetter
' 20021100901
' Shamelessly stolen from
' http://www.microsoft.com/windows2000/techinfo/ \
' planning/activedirectory/bulksteps.asp
'Global variables
Dim Container
Dim OutPutFile
Dim FileSystem
'Initialize global variables
Set FileSystem = WScript.CreateObject("Scripting.FileSystemObject")
Set OutPutFile = FileSystem.CreateTextFile("virtual.txt", True)
Set Container=GetObject("LDAP://DC=example,DC=com")
EnumerateUsers Container
'Clean up
OutPutFile.Close
Set FileSystem = Nothing
Set Container = Nothing
'Say Finished when your done
WScript.Echo "Finished"
WScript.Quit(0)
'List all Users
Sub EnumerateUsers(Cont)
Dim User
'Go through all Users and select them
For Each User In Cont
Select Case LCase(User.Class)
'If you find Users and Groups
'Added groups after Милен Панков mailed me about it :)
Case "user", "group"
'Select all proxyAddresses
Dim Alias
If IsEmpty(User.proxyAddresses) Then
' do nothing.
ElseIf (TypeName(User.proxyAddresses) = "String") Then
OutPutFile.WriteLine "alias: " & User.proxyAddresses
Else
For Each Alias in User.proxyAddresses
OutPutFile.WriteLine "alias: " & Alias
Next
End If
Case "organizationalunit", "container"
EnumerateUsers User
End Select
Next
End Sub
the script output looks like this:
alias: smtp:bar@example.com
the script is called daily from a scheduled task and writes the mailbox list to a file named virtual.txt
To access this file from Postfix server we decided to mount its folder with cifs+autofs. We installed and configured autofs:
/auto /etc/auto.cifs
# /etc/auto.cifs
activedirectory -fstype=cifs,rw,username=Administrator,domain=EXAMPLE,password=secret ://192.168.1.17/spamassassin
so when accessing to /auto/activedirectory it will mount //192.168.1.17/spamassassin through samba, that’s where we left the VBS script and the virtual.txt file
Finally another script, called from cron, updates the relay_recipients file and restarts postfix if anything changed
USERSFILE=/auto/activedirectory/virtual.txt
TMPFILE=/root/exchangesync/users_exchange.txt
TMPFILE2=/root/exchangesync/users_exchange.parsed
echo "--- Start: `date` ----------"
# Try to access file 10 times so autofs
# have time to mount folder through samba
for i in `seq 1 10`;
do
if [ -f $USERSFILE ]; then
echo "found $USERSFILE"
break
else
if [ $i -eq 10 ]; then
echo "can't access $USERSFILE after 10 retries"
echo "------ Fi: `date` ----------"
exit 1
fi
echo "can't access $USERSFILE, retrying..."
sleep 2
fi
done
cat $USERSFILE > $TMPFILE
dos2unix $TMPFILE
awk -F: '/alias: (SMTP|smtp):/ {printf("%s\tOK\n",$3)}' $TMPFILE | sort > $TMPFILE2
oldmd5=`cat /etc/postfix/relay_recipients.md5`
newmd5=`md5sum $TMPFILE2 | cut -d" " -f1`
if [ "$oldmd5" != "$newmd5" ]; then
echo "md5 differ, reloading postfix"
diff -b /etc/postfix/relay_recipients $TMPFILE2
cp /etc/postfix/relay_recipients /etc/postfix/relay_recipients.old
cp $TMPFILE2 /etc/postfix/relay_recipients
echo -n $newmd5 > /etc/postfix/relay_recipients.md5
/usr/sbin/postmap /etc/postfix/relay_recipients && /etc/init.d/postfix reload
else
echo "md5 match, not reloading postfix"
fi
echo "------ End: `date` ----------"
echo
PHP4 and PHP5 in Debian Squeeze
Nov 7th
One of out clients needs a PHP4 stack to run legacy web applications. We use Debian and we do not want to replace Squeeze’s PHP5, as many other Debian-packaged applications like squirrelmail or PhpMyAdmin need it. Our solution is to run PHP4 through fcgid.
PHP4 as CGI
In order to install php4-cgi we choose to add the amazing debian snapshots archive to apt sources.list. This gives us access to old Debian packages (there are other methods).
# installed from http://snapshot.debian.org/package/php4/
deb http://snapshot.debian.org/archive/debian/20070612T000000Z lenny main
install php4 normally
apt-get install php4-cgi php4-cli php4-common php4-curl php4-dev php4-domxml php4-gd php4-imap php4-mcal php4-mcrypt php4-mysql
to configure an Apache VirtualHost to use PHP4 instead of PHP5 add this configuration:
(...)
# PHP4
<FilesMatch "\.ph(p3?|tml)$">
Options ExecCGI FollowSymLinks
SetHandler fcgid-script
FCGIWrapper /usr/bin/php-cgi .php
</FilesMatch>
</VirtualHost>