Month: December 2017

mongodb InternalError: failed to create service entry worker thread

gbstack 1 Comment

Following is content from /var/log/mongodb/mongod.log:

2017-12-18T01:21:41.953+0800 W EXECUTOR [conn4444] Terminating session due to error: InternalError: failed to create service entry worker thread
2017-12-18T01:21:41.953+0800 I NETWORK [listener] end connection 127.0.0.1:36868 (1001 connections now open)
2017-12-18T01:21:41.953+0800 I NETWORK [listener] connection accepted from 127.0.0.1:36870 #4445 (1002 connections now open)
2017-12-18T01:21:41.954+0800 I - [listener] pthread_create failed: Resource temporarily unavailable
2017-12-18T01:21:41.954+0800 W EXECUTOR [conn4445] Terminating session due to error: InternalError: failed to create service entry worker thread
2017-12-18T01:21:41.954+0800 I NETWORK [listener] end connection 127.0.0.1:36870 (1001 connections now open)
2017-12-18T01:21:41.979+0800 I NETWORK [listener] connection accepted from 127.0.0.1:36872 #4446 (1002 connections now open)
2017-12-18T01:21:41.979+0800 I - [listener] pthread_create failed: Resource temporarily unavailable
2017-12-18T01:21:41.979+0800 W EXECUTOR [conn4446] Terminating session due to error: InternalError: failed to create service entry worker thread
2017-12-18T01:21:41.979+0800 I NETWORK [listener] end connection 127.0.0.1:36872 (1001 connections now open)

 

This problem occured after web request (which will create new MongoClient instance) started for a while, it's not happened immediately, so it's likely caused by the connection limit is reached.

Through the log we can find the connection count limit is about 1000.

 

Every time I open Mongo Console, I got startup warnings

MongoDB shell version v3.6.0
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.6.0
Server has startup warnings:
2017-12-19T16:29:45.510+0800 I STORAGE [initandlisten]
2017-12-19T16:29:45.510+0800 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2017-12-19T16:29:45.510+0800 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
2017-12-19T16:29:46.006+0800 I CONTROL [initandlisten]
2017-12-19T16:29:46.006+0800 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database.
2017-12-19T16:29:46.006+0800 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted.
2017-12-19T16:29:46.006+0800 I CONTROL [initandlisten]
2017-12-19T16:29:46.008+0800 I CONTROL [initandlisten]
2017-12-19T16:29:46.008+0800 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2017-12-19T16:29:46.008+0800 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2017-12-19T16:29:46.008+0800 I CONTROL [initandlisten]
2017-12-19T16:29:46.008+0800 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2017-12-19T16:29:46.008+0800 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2017-12-19T16:29:46.008+0800 I CONTROL [initandlisten]
2017-12-19T16:29:46.008+0800 I CONTROL [initandlisten] ** WARNING: soft rlimits too low. rlimits set to 1024 processes, 64000 files. Number of processes should be at least 32000 : 0.5 times number of files.
2017-12-19T16:29:46.008+0800 I CONTROL [initandlisten]
We can see the connection count limit is actually 1000.

 

So to fix this issue, either is to increase the connecton count limit, or to close the created MongoClient explicitly in code.
In my program every request a new MongoClient is created, I didn't explicitly close it after using it. The solution is calling close method of MongoClient or using singleton MongoClient (the MongoClient library recommended this way for using connection pooling).

By using this method, the problem is solved.