Multiprocessing for Frozen Python

Recently I built a GUI application using PySide, and do some background calculation using multiprocessing. Using Python interpreter to run this application works smoothly. But if I freeze it to Windows executable file (exe) and run it, guess what, I got two main windows showing up!

 

After searching a while, I found following code should be added

It should be placed right after:

 

The freeze_support() is mainly used to pass initialization data from parent process to newly created process using pipe, including modules, process name, current working directory, etc. In Unix-based system this function is not needed, because fork will do these things for us (multiprocessing.Process will call os.fork() in start method on Linux)

 

the low-level implementation is in Lib/multiprocessing/forking.py and a wrapper in Lib/multiprocessing/__init__.py.

By reading the source code, we can see Python will detect –multiprocessing-fork in command line arguments to determine whether current process is child process or not. And the last command line argument is the pipe file handle. The data in main process is serialized using pickle, then pass to child process using pipe.

Shiboken Multiple Include Path on Windows

In shiboken documentation, it’s saying the separator of include path should be : (colon)

But on Windows every full path is containing colon (e.g. C:\Windows\cmd), it will mess with the colon separator.

In its source code printUsage() function I found it’s actually using a PATH_SPLITTER macro to represent the separator. And under windows it’s semicolon..

 

At past I’m using colon as separator:

–include-paths=D:\Qt\4.8.7\include\:D:\Qt\4.8.7\include\QtCore\:D:\Python27\Lib\site-packages\PySide\include\PySide

And I got lots of error messages like following:

type is specified in typesystem, but not defined

After replacing colon with semicolon, these errors are gone.

Memory Forensic for DateTime Type

In C or C++, a DateTime value is often represented by time_t type, it’s a UNIX timestamp format, which is number of seconds elapsed since 1970-1-1.  This type is defined as long (4 bytes long) on 32 bit machine, and long long (8 bytes long) on 64 bit machine.

 
And Datetime value has one important trait:

In short time the high byte won’t change.

Suppose a DateTime value is DDCCBBAA, so in memory it will be AA BB CC DD (in little endian), the highest byte (DD) won’t change in short time, and the second highest byte (CC) don’t change much. This is because the low 2 bytes can represent 65536 seconds, which is about 18 hours, that means after every 18 hours the second highest byte will only increase by 1. And the highest byte only change after 18*256 hours (nearly 192 days) passed by.

 

Next we will see how to use Python to convert bytes array to timestamp value, and display it as readable format

The data variable contains bytes needed work on, after some observing and investigation,  we think byte 4 to byte 8 is the DateTime value. Then we use struct.unpack function to convert the bytes to timestamp value, ‘<L’ means little endian and long type. Next we will convert timestamp value to human-readable format.

 

Note that time_t doesn’t contain timezone information, you need discover timezone at other places.

NSIS File command not working after calling SetOutPath

I wrote a NSIS script which copies two MSI files to $INSTDIR\Installers directory and then execute them, it’s pretty easy and following is the code

Unfortunately above code will not work and it’s saying NSIS cannot find these two msi files, but I’m sure they existed!

 
Then I tried to put $INSTDIR at beginning of the path

Now a MSI Help Dialog pops up with information about how to use msiexec command.. Apparently it failed again.

 

To solve this issue, we need call SetOutPath before ExecWait, like following

The first three lines will copy SqlLocalDB.msi and OutsideX64.msi files to $INSTDIR\Installers, and the last three lines will install MSI files at $INSTDIR\Installers.

It’s because that ExecWait (Exec, ExecShell) command will use $OUTDIR as working directory, and this path can be set by SetOutPath command (By default $OUTDIR is set to $INSTDIR).

If we don’t call SetOutPath command explicitly before ExecWait, its working directory will be $INSTDIR\Installers, and it will try to install MSI files under $INSTDIR\Installers\Installers directory, which is apparently wrong.

 

(And this guy had same problem as well http://stackoverflow.com/questions/17105455/install-sql-server-2008-r2-express-sp2-with-nsis)

Outside In Viewer Quiet Install not working

I need to install Outside In Viewer component silently in my NSIS installer. The Outside In Viewer installer is a MSI file, to install it silently, we need run following command in command line.

msiexec /i OutsideX64.msi /q

Above command will finish immediately, but it’s finished so soon that I think the installation is likely failed.

Since the command line didn’t return anything, I need enable logging to check what’s wrong with it

msiexec /i OutsideX64.msi /q /log log.txt

(Note that the log.txt file should be created in advance, otherwise an error message will show.)

In the log file we see such message

MSI (s) (C4:BC) [14:54:57:760]: Product: OutsideX64 — Error 1303. The installer has insufficient privileges to access this directory: C:\Program Files\OIX. The installation cannot continue. Log on as administrator or contact your system administrator.

By reading this log message, we know we need run this installer as administrator

 

To request Administrator privilege, add following line into our NSIS installer (at first line)

 

MinGW unable to find pkg-config

Recently I need build libewf on Windows, and MinGW is chosen as the compilation tool.

Open MSYS and run autogen.sh in libewf source directory, then following error is appeared

unable to find: pkg-config

 

I read source code of autogen.sh and realized that it’s trying to locate pkg-config at /mingw/bin directory,  Then I checked MinGW installation directory and bin folder, found there is no such pkg-config file. So we need install pkg-config manually.

 

Install pkg-config

At MinGW Wiki page it introduces two methods to install pkg-config, but using a pre-compiled version is much easier. Here is pre-compiled pkg-config download address from SourceForge. After downloading is completed, extract pkg-config.exe and move it to MinGW/bin directory, retry running autogen.sh we will found this problem is solved

WPF Set StartupUri in code

Introduction

StartupUri is a property of Application class, which sets the first UI to show when application starts. By default it’s set in App.xaml file, e.g.

In above code the StartupUri value is MainWindow.xaml, and using default value is OK for most of times.

 

Why Set StartupUri in Code?

But what if you want to show other windows according to different conditions? Assume our application can open .proj files, so the file name will be passed in as command line arguments.  Now we want to show a file chooser dialog at startup if user run our application directly (without double clicking .proj files), and open another window (call it Project Explorer) when user start application by double clicking .proj file.

In this case we need to check command line arguments, then set corresponding StartupUri in code.

 

How to Do it

First in App.xaml.cs, add an override method OnStartup for Application class.

This method will be called at startup of application.

Setting StartupUri in OnStartup method is just a recommended way, you can set it in constructor as well. The execution order is: Constructor -> XAML Parsing -> OnStartup, so setting StartupUri in constructor will be overridden by XAML and OnStartup, and StartupUri in XAML will be overriden by OnStartup

 

Next we set the StartupUri according to whether filename is passed in.

In above code we call System.Environment.GetCommandLineArgs method to get command line arguments. You can see the StartupUri is actually an Uri object, its UriKind is UriKind.Relative. (If we don’t specify the UriKind parameter, we will get an UriFormatException)

 

 

Apache Derby Database Browser

Background

Apache Derby is an embed database implemented in pure Java. It’s implemented in pure Java, thus it’s more compatible with Java.

There are mainly three database browsers for Derby

Since the first two softwares are commercial, in this tutorial we will introduce how to use SQuirreL.

 

Prerequisites

  • JRE (or JDK)

Download JRE (or JDK) from java.com and install it. Don’t forget to add its bin path to environment variable.

  • SQuirreL

Download SQuirreL from installation page and install it using following command

java -jar squirrel-sql-<version>-install.jar

(or you can double click the jar file if you’re under Windows)

  • Derby Client and Derby Embedded Driver jar files

Download the zip file from this link, then extract derby.jar and derbyclient.jar from the zip file. Next copy these two files into SQuirreL’s lib folder.

If the two jar files are copied to correct path, you will see in SQuirreL’s Drivers panel Apache Derby Client and Apache Derby Embedded are enabled.

SQuirreL Drivers Panel

SQuirreL Drivers Panel

 

 

Create a connection to Derby database

In Alias Panel, click Add Alias button

SQuirrel add alias

SQuirrel add alias

The Name field is not important, and you can fill in with anything. In URL field you enter the server address and database name, e.g. jdbc:derby://localhost/derby_test

 

And if you want to connect to embedded file database,  you need set JDBC Connection URL like following:

jdbc:derby:PATH

Assume the database is located at D:\derby_test, then the JDBC URL should be

jdbc:derby:D:\derby_test

SQuirreL add embed derby database

SQuirreL add embed derby database

 

 

C# Int (Integer) Infinity

Background

I’m implement a data structure which is used to convert a time range to human-readable text, e.g. 0-7 days to “Within 1 Week”, 7-30 days to “1 Week to 1 Month”. Here I use Tuple<int,int> to represent the time range but not two integers, because we can define a Dictionary<Tuple<int, int>, String> to represent a map from time range to text.

But how could we represent “Over 2 Weeks”? Apparently the second parameter of the Tuple should be infinity.

Infinity

To represent infinity for an integer, we can use int.MaxValue (it’s actually Int32.MaxValue)

So we can solve above problem using following code

 

IKVM.NET Convert JAR to .NET (C#, VB.NET) DLL

Why Convert JAR to .NET DLL

Imagine your company builds an online file converter, and its backend programming language is Java because of its position in web development. And in this project you build a file parsing library (jar format certainly) which is used to parse different types of files.

After this project is done, your boss decides to create a Windows desktop version for this software. Now the best choice is .NET, but how can you reuse your java parsing library? That’s why IKVM.NET existed.

IKVM.NET Introduction

IKVM.NET is a .NET implementation of Java which allows us to run Java code in .NET (C#, VB.NET, etc.) easily. It includes following three components:

  • JVM in .NET (which means you can run java byte code in .NET)
  • Java class library implementation
  • A compiler converting java byte code to .NET IL

There are many famous Java projects converted to .NET using IKVM.NET already, such as Tika, Curator, etc. Even Mono has include it as a built-in component.

Start Converting

This will generate a DLL file named libpst.dll from libpst.jar in current directory.

Without -target:library option, it will generate an exe file.