Greenplum 增加連線數目後 系統無法順利啟動

筆者: 歐立威 Hank


Greenplum 在四版時，增加過多的 max_connection 至1000 以上時確實會遇見問題 (參考Pivotal KB: "https://discuss.pivotal.io/hc/en-us/articles/201462656-GPDB-startup-initilization-failed-with-could-not-create-semaphores-No-space-left-on-device-")，

筆者在五版時有遇過僅從預設 250 增加至 350 便遇見系統無法正常啟動的狀況，

原因可能是Segment Instance較多導致 semaphores 不足 (OS系統的信號燈，用以管控多執行緒環境共用資源的機制)，

但問題解決並不複雜，可透過調整OS參數作解決:

筆者透過將 Greenplum 官方建議值調高成底下值便解決問題，詳細能夠調整至多大，還請詢問系統管理員，預設 Greenplum 群集不會和其他系統共用，測試結果調整至底下值並無其他影響發生。

$ vi /etc/sysctl.conf
kernel.sem=32000 1024000000 500 32000
## kernel.sem={SEMMSL, SEMMNS, SEMOPM, SEMMNI}


調整完畢後記得檢查OS參數是否正確讀取，以免系統又啟動失效還得修復:

$ sysctl -p
$ sysctl -a | grep sem
kernel.sem = 32000 1024000000 500 32000


底下附上遇見問題時的GPDB LOG訊息:

2017-12-26 17:51:15.565211 CST,,,p17519,th1956788096,,,,0,,,seg-1,,,,,"FATAL","XX000","could not create semaphores: No space left on device (pg_sema.c:129)","Failed system call was semget(40010001, 17, 03600).","This error does *not* mean that you have run out of disk space.

It occurs when either the system limit for the maximum number of semaphore sets (SEMMNI), or the system wide maximum number of semaphores (SEMMNS), would be exceeded.  You need to raise the respective kernel parameter.  Alternatively, reduce PostgreSQL's consumption of semaphores by reducing its max_connections parameter (currently 1753).

It occurs when either the system limit for the maximum number of semaphore sets (SEMMNI), or the system wide maximum number of semaphores (SEMMNS), would be exceeded.  You need to raise the respective kernel parameter.  Alternatively, reduce PostgreSQL's consumption of semaphores by reducing its max_connections parameter (currently 1753).

It occurs when either the system limit for the maximum number of semaphore sets (SEMMNI), or the system wide maximum number of semaphores (SEMMNS), would be exceeded.  You need to raise the respective kernel parameter.  Alternatively, reduce PostgreSQL's consumption of semaphores by reducing its max_connections parameter (currently 1753).


Greenplum運作時的表現與OS息息相關，學習過程中也不可忽略 Linux 系統方面的相關知識，
釐清問題發生在GPDB層面或是OS層面才能有效率的找出問題並解決問題。