RuntimeError: view size is not compatible with input tensor’s size and stride (at least one dimension spans across two contiguous subspaces)

RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.

The causing code is

This error didn't happen before, so it's introduced by new version of pytorch (now my current used pytorch is 1.8.1).

And printing the array, I found it's a boolean array.

Solution

Add .contiguous() before view() or use reshape to replace view

So change the line

to

or

Automatically generate requirements.txt for pip

pip is popular package manager for Python.

There is often a requirements.txt file in lots of Python projects specifying several dependencies need to install.
We can install several packages using pip at once using following command

pip -r requirements.txt

to generate requirements for pip, certainly we can type the required package names line by line,
but there is a tool called pipreqs can help us.

pipreqs

pipreqs can analyze your python project for dependencies and generate requirements.txt
used by pip automatically.

Installation

pip install pipreqs

Generate requirements.txt

pipreqs .

This command will generate requirements.txt using current directory as source root directory.
Then you can run pip -r requirements.txt to install dependencies.

Python No module named feedparser

I want to use following tool to download paper from arxiv

https://github.com/pvskand/arXiv_download

I installed it by pip

When trying to run python arXiv, following error happened.

Install feedparser using pip

Run python arXiv again, the issue is fixed

Python TypeError: Error when calling the metaclass bases

When running the code, I got following error:

TypeError: Error when calling the metaclass bases
module.__init__() takes at most 2 arguments (3 given)

 

The source code is like following:

AboutWindow.py

 

GitFtpWindow.py

 

This is because in AboutWindow.py file, the GitFtpWindow identifier is a package, but not a class.

To fix this problem, we need modify AboutWindow.py file like following:

 

 

No module named pefile

Today I want to build a python script to EXE file using PyInstaller. But after running PyInstaller, I got following error

No module named pefile

The full stack trace is

 

Starting from PyInstaller 3.2, a new module named pefile is introduced. This module is used to work with Windows binary file (EXE file, also called PE file). Before PyInstaller 3.2, PE related operation is done using its own module, like setting PE header and DOS header)

 

Solution

Assume the Python installation direcotry is C:\Python27.

Open Command Prompt window, and navigate to C:\Python27\Scripts

cd C:\Python27\Scripts

Then install pefile module using following command

pip install pefile

 

(pip is built into latest Python distribution, 2.7.12 for now)

Multiprocessing for Frozen Python

Recently I built a GUI application using PySide, and do some background calculation using multiprocessing. Using Python interpreter to run this application works smoothly. But if I freeze it to Windows executable file (exe) and run it, guess what, I got two main windows showing up!

 

After searching a while, I found following code should be added

It should be placed right after:

 

The freeze_support() is mainly used to pass initialization data from parent process to newly created process using pipe, including modules, process name, current working directory, etc. In Unix-based system this function is not needed, because fork will do these things for us (multiprocessing.Process will call os.fork() in start method on Linux)

 

the low-level implementation is in Lib/multiprocessing/forking.py and a wrapper in Lib/multiprocessing/__init__.py.

By reading the source code, we can see Python will detect --multiprocessing-fork in command line arguments to determine whether current process is child process or not. And the last command line argument is the pipe file handle. The data in main process is serialized using pickle, then pass to child process using pipe.

PySide PyInstaller Packaging

when using pyinstaller to package the pyside application with sqlite dependency, it doesn't copy sqldrivers to destination folder (qt4_plugins). When run the application, I will get error message complaining about SQLite driver is not found.

I need copy them to qt4_plugins directory from Qt directory

My solution is wrting a separate script to copy the sqldrivers files to executable directory after packaging.

But there should be a better method by configuring PyInstaller?

Memory Forensic for DateTime Type

In C or C++, a DateTime value is often represented by time_t type, it's a UNIX timestamp format, which is number of seconds elapsed since 1970-1-1.  This type is defined as long (4 bytes long) on 32 bit machine, and long long (8 bytes long) on 64 bit machine.

 
And Datetime value has one important trait:

In short time the high byte won't change.

Suppose a DateTime value is DDCCBBAA, so in memory it will be AA BB CC DD (in little endian), the highest byte (DD) won't change in short time, and the second highest byte (CC) don't change much. This is because the low 2 bytes can represent 65536 seconds, which is about 18 hours, that means after every 18 hours the second highest byte will only increase by 1. And the highest byte only change after 18*256 hours (nearly 192 days) passed by.

 

Next we will see how to use Python to convert bytes array to timestamp value, and display it as readable format

The data variable contains bytes needed work on, after some observing and investigation,  we think byte 4 to byte 8 is the DateTime value. Then we use struct.unpack function to convert the bytes to timestamp value, '<L' means little endian and long type. Next we will convert timestamp value to human-readable format.

 

Note that time_t doesn't contain timezone information, you need discover timezone at other places.

No module named Crypto.PublicKey

When I tried to import the paramiko module, I got following error:

No module named Crypto.PublicKey

That means we need to install pycrypto library. Run following command to install pycrypto library by using pip:

In the installation process I got a another error message:

error: Unable to find vcvarsall.bat

That means we need a C compiler to compile this extension. But we can also download a pre-built extension for windows: http://www.voidspace.org.uk/python/modules.shtml#pycrypto

 

After installing this extension and tried to import "paramiko" module, I got the same error "No module named Crypto.PublicKey" again. This is because the installer installs pycrypto library with lowercase name, but paramiko tries to import it with uppercase name. We should rename crypto directory under "Lib/site-packages" to Crypto, then importing paramiko will work.

Unable to find vcvarsall.bat when installing gmpy using pip

GMPY is a multiple-precision arithmetic module, which is a extension module written by C. I tried to install it by using pip, then I got following error message:

Unable to find vcvarsall.bat

It's because it's trying to find a c compiler (Visual C++ or MinGW) to compile this extension but failed. To fix this error, one method is to install a C compiler (Visual C++ or MinGW). And another easier way is to download an windows installer: https://code.google.com/p/gmpy/downloads/list