Program runs very slow on a Oracle database server of Linux 64-bit -

i have program takes in average 30 minutes run on general 64-bit linux machines. takes 150 minutes run on 64-bit linux machine oracle database running.

vmstat shows program need in average 6gb memory , bi=0 on general machines. uses no more 2gb memory on oracle database server , bi high.

the oracle database server has total memory=500g, sga=42112m , pga=20g.

my question oracle setup restrict memory usage of other program.

the following little part of top output when program running:

top - 15:34:08 114 days, 12:27, 11 users,  load average: 0.48, 0.60, 0.54 tasks: 872 total,   2 running, 870 sleeping,   0 stopped,   0 zombie cpu(s): 15.8%us,  3.2%sy,  0.0%ni, 64.6%id, 16.3%wa,  0.0%hi,  0.2%si,  0.0%st mem:  529409524k total, 523431316k used,  5978208k free,   788964k buffers swap: 20971516k total,   247040k used, 20724476k free, 234761252k cached  pid user      pr  ni  virt  res  shr s %cpu %mem    time+  command                                                          49096 oracle    20   0 41.5g 1.1g 1.0g s 47.5  0.2   0:14.90 oracle                                                           17862 oracle    20   0 41.5g  46m  32m r 18.3  0.0   5:09.58 oracle                                                           17874 oracle    20   0 41.5g  47m  31m s 16.5  0.0   4:50.99 oracle                                                           35934 root      rt   0  769m 101m  55m s 12.8  0.0   5830:15 osysmond.bin                                                     17864 oracle    20   0 41.5g  46m  31m s 11.0  0.0   4:53.54 oracle                                                           17882 oracle    20   0 41.5g  46m  31m s  7.3  0.0   5:20.39 oracle                                                           52363 applmgr   20   0 15636 1760  828 r  7.3  0.0   0:00.08 top                                                              35064 root      20   0 1974m  47m  14m s  3.7  0.0 642:05.39 ohasd.bin                                                        26123 oracle    20   0 33.4g  33m  16m s  1.8  0.0  74:18.77 oracle                                                           27764 oracle    20   0 33.4g  21m  19m s  1.8  0.0   0:41.09 oracle                                                           36387 root      20   0  814m  20m  10m s  1.8  0.0 384:21.37 octssd.bin                                                       38011 oracle    -2   0 1237m  14m  12m s  1.8  0.0   1212:52 oracle                                                           38130 root      20   0 2910m  60m  20m s  1.8  0.0   1070:16 crsd.bin                                                         38192 oracle    20   0 1239m  17m  15m s  1.8  0.0  21:21.70 oracle                                                           39657 root      20   0 1582m  23m  11m s  1.8  0.0   1108:05 orarootagent.bi                                                  41587 oracle    20   0 33.4g  23m  20m s  1.8  0.0   0:00.14 oracle                                                           49809 oracle    20   0 41.4g  57m  54m s  1.8  0.0   0:00.73 oracle                                                           1 root      20   0 21460 1564 1240 s  0.0  0.0  44:37.17 init                                                             2 root      20   0     0    0    0 s  0.0  0.0   1:05.13 kthreadd                                                         3 root      20   0     0    0    0 s  0.0  0.0  18:12.77 ksoftirqd/0                                                      6 root      rt   0     0    0    0 s  0.0  0.0 163191:59 migration/0                                                      7 root      rt   0     0    0    0 s  0.0  0.0   0:50.35 watchdog/0                                                       8 root      rt   0     0    0    0 s  0.0  0.0 164526:35 migration/1                                                      10 root      20   0     0    0    0 s  0.0  0.0  18:48.93 ksoftirqd/1                                                      12 root      rt   0     0    0    0 s  0.0  0.0   0:51.10 watchdog/1                                                       13 root      rt   0     0    0    0 s  0.0  0.0 161719:04 migration/2                                                      15 root      20   0     0    0    0 s  0.0  0.0   8:58.82 ksoftirqd/2                                                      16 root      rt   0     0    0    0 s  0.0  0.0   1:00.06 watchdog/2                                                       17 root      rt   0     0    0    0 s  0.0  0.0 162585:32 migration/3                                                      19 root      20   0     0    0    0 s  0.0  0.0   6:18.01 ksoftirqd/3                                                      20 root      rt   0     0    0    0 s  0.0  0.0   0:38.44 watchdog/3                                                       21 root      rt   0     0    0    0 s  0.0  0.0 158088:12 migration/4                                                      23 root      20   0     0    0    0 s  0.0  0.0  12:46.60 ksoftirqd/4                                                      24 root      rt   0     0    0    0 s  0.0  0.0   0:44.32 watchdog/4                                                       25 root      rt   0     0    0    0 s  0.0  0.0 164162:22 migration/5                                                      27 root      20   0     0    0    0 s  0.0  0.0  14:00.94 ksoftirqd/5                                                      28 root      rt   0     0    0    0 s  0.0  0.0   0:39.26 watchdog/5                                                       29 root      rt   0     0    0    0 s  0.0  0.0 162112:29 migration/6                                                      31 root      20   0     0    0    0 s  0.0  0.0   7:36.89 ksoftirqd/6                                                      32 root      rt   0     0    0    0 s  0.0  0.0   0:36.18 watchdog/6                                                       33 root      rt   0     0    0    0 s  0.0  0.0 160716:53 migration/7                                                      35 root      20   0     0    0    0 s  0.0  0.0   3:33.29 ksoftirqd/7                                                      36 root      rt   0     0    0    0 s  0.0  0.0   0:32.82 watchdog/7                                                       37 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 cpuset                                                           38 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 khelper                                                          39 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 netns                                                            40 root      20   0     0    0    0 s  0.0  0.0   0:38.28 sync_supers                                                      41 root      20   0     0    0    0 s  0.0  0.0   0:06.53 bdi-default                                                      42 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kintegrityd                                                      43 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kblockd                                                          44 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 ata_sff                                                          45 root      20   0     0    0    0 s  0.0  0.0   0:00.01 khubd                                                            46 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 md                                                               50 root      20   0     0    0    0 s  0.0  0.0   0:30.08 khungtaskd                                                       51 root      20   0     0    0    0 s  0.0  0.0  13:30.48 kswapd0                                                          52 root      25   5     0    0    0 s  0.0  0.0   0:00.00 ksmd                                                             53 root      20   0     0    0    0 s  0.0  0.0   0:00.03 fsnotify_mark                                                    54 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 crypto                                                           60 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kthrotld                                                         62 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kpsmoused                                                        349 root      20   0     0    0    0 s  0.0  0.0   0:00.00 scsi_eh_0                                                        351 root      20   0     0    0    0 s  0.0  0.0   0:14.72 hpsa                                                             364 oracle    20   0 33.4g  37m  31m s  0.0  0.0   0:14.78 oracle                                                           366 oracle    20   0 33.4g  34m  28m s  0.0  0.0   0:04.32 oracle                                                           374 oracle    20   0 33.4g  37m  30m s  0.0  0.0   0:05.47 oracle                                                           377 root      20   0     0    0    0 s  0.0  0.0   0:00.05 scsi_eh_1                                                        378 root      20   0     0    0    0 s  0.0  0.0   0:00.01 scsi_eh_2                                                        402 root      20   0  102m 4024 3052 s  0.0  0.0   0:00.26 sshd                                                             484 root      20   0 36268 1340 1048 s  0.0  0.0   0:00.44 vncconfig                                                        485 root      20   0  150m 3916 2220 s  0.0  0.0   0:00.75 xterm                                                            486 root      20   0  127m 2184 1484 s  0.0  0.0   0:00.52 twm                                                              493 root      20   0     0    0    0 s  0.0  0.0   0:00.00 scsi_eh_3                                                        494 root      20   0     0    0    0 s  0.0  0.0   0:00.76 usb-storage                                                      497 root      20   0  107m 1860 1460 s  0.0  0.0   0:00.08 bash                                                             537 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         539 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         558 root      20   0     0    0    0 s  0.0  0.0   0:04.83 jbd2/dm-1-8                                                      559 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 ext4-dio-unwrit                                                  633 root      20   0     0    0    0 s  0.0  0.0   0:15.35 kworker/3:0                                                      665 root      16  -4 17568 7752  460 s  0.0  0.0  22631:49 udevd                                                            673 oracle    20   0  102m 2004 1032 s  0.0  0.0   0:00.05 sshd                                                             697 oracle    20   0  108m 2048 1444 s  0.0  0.0   0:00.06 bash                                                             771 oracle    20   0 33.4g  25m  21m s  0.0  0.0   0:30.99 oracle                                                           792 root      20   0     0    0    0 s  0.0  0.0   0:00.00 scsi_eh_4                                                        1017 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 qla2xxx_4_dpc                                                    1018 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 scsi_wq_4                                                        1021 root      20   0     0    0    0 s  0.0  0.0   0:00.00 scsi_eh_5                                                        1037 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 qla2xxx_5_dpc                                                    1038 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 scsi_wq_5                                                        1075 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 qlcnic                                                           1135 hpsmh     20   0  176m  11m 4380 s  0.0  0.0   0:25.39 hpsmhd                                                           1677 oracle    20   0 41.4g 118m 113m s  0.0  0.0   0:01.45 oracle                                                           1798 oracle    20   0 41.4g  56m  51m s  0.0  0.0   0:00.59 oracle                                                           1974 root      20   0     0    0    0 s  0.0  0.0   0:00.02 flush-252:109                                                    1987 root      20   0  102m 4060 3092 s  0.0  0.0   0:00.29 sshd                                                             1993 sysmon    20   0  102m 2004 1036 s  0.0  0.0   0:00.78 sshd                                                             1994 sysmon    20   0  105m 1888 1440 s  0.0  0.0   0:00.73 bash                                                             2084 root      20   0  158m 4072 2592 s  0.0  0.0   0:00.08 sudo                                                             2110 root      20   0  107m 1580 1236 s  0.0  0.0   0:00.01 sh                                                               2488 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kmpathd                                                          2489 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kmpath_handlerd                                                  2606 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2612 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2616 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2620 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2624 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2628 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2635 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2638 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2645 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2652 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2658 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2662 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2667 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2672 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2676 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2683 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2686 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2687 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2692 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2699 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2708 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2715 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2718 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2720 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2728 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2734 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2743 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2750 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2753 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2755 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2761 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2766 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2774 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2777 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2782 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2784 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2787 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2790 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2801 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2812 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2814 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2820 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2823 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2827 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2833 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2837 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2842 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2848 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2854 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2864 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2877 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2888 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2901 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2905 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2912 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2920 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2922 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2931 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2940 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2952 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2953 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2955 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2972 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2977 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2983 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2989 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2990 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         2996 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         3003 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         3010 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         3014 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         3021 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         3027 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         3033 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         3046 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         3053 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         3064 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         3072 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         3092 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         3097 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         3108 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         3114 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         3119 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         3126 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         3133 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         3139 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         3148 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         3153 oracle    20   0 4560m 304m  12m s  0.0  0.1 157:16.04 java                                                             3154 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         3162 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         3397 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         3399 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         3427 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         3491 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         3492 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         3494 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         3496 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         3499 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         3502 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         3506 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         3511 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         3520 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         3523 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         3529 root       0 -20     0    0    0 s  0.0  0.0   0:00.00 kdmflush                                                         3613 root      20   0     0    0    0 s  0.0  0.0   0:00.02 kjournald                                                        

we have not enough information determine happening in server. ways know oracle setup can affect process behavior (without considering ram/cpu/disk overload) are:

• ulimit: oracle setup uses command limit maximum number of open file descriptors , limit maximum number of process per user. command can used limit max amount of resident memory per process, oracle not use in way.
• kernel shared memory limit: oracle setup defines shmmax , shmall , these parameters limit size of shared memory available in linux kernel.

but these limits affects memory consumption of program if:

• it working many opened files simultaneously (then affected ulimit)
• it spawning processes (to perform parallel tasks) (then affected ulimit)
• it allocating , using shared memory (the same oracle uses sga) (then affected shmmax , shmall).

there other linux kernel parameters, process , services can affect maximum size of memory resident available process (the popular ones are: via limits.conf, via upstart , via cgroup – see https://afrobeard.wordpress.com/2012/02/01/linux-limiting-resident-memory/). ways not derived oracle setup, derived kernel configuration. must check them.

note: determine if limit in ram usage derived kernel configuration can create program (using java, c, c++ or language want) reserves , use 6g (e.g. creating arrays of arrays , setting values) , test if can use more 2g.

to determine if oracle workload interferes program can try test program in oracle server database instance down (if possible).