hdf5 - MATLAB h5write has inconsistent performance -
i'm trying use savefast function on matlab fileexchange. noticed it's performance on setup not consistent. when run below code, below times. notice on 4th run took 20x longer! there between 2 , 5 slow runs per 10, randomly distributed (sometimes 3 in row slow). variation entirely in low level h5ml.hdf5lib2 function.
i've tried on 4 different machines. on 2 of them (server 2008 r2, windows 7), see wild variation below. on 2 others (server 2008, server 2012), there no variation. 2 server 2008 boxes have identical hardware, anti-virus, , nothing running on them. problem machine specific. however, boxes relatively powerful 64-bit think problem should not happening. there tricky low level setting make hdf5 work more consistently?
when try using save
or fwrite
command instead of code below, times have little variation makes me think hdf5 specific issue.
clear classes; delete('test_*.mat'); x = rand(10000, 1000); %% disp('using h5write:'); j = 1:10 % make new file fname = sprintf('test_h5_%d.mat', j); dummy = 0; save(fname, '-v7.3', 'dummy'); varname = '/x'; h5create(fname, varname, size(x), 'datatype', class(x)); tic; h5write(fname, varname, x); toc; end disp('using save:'); j = 1:10 tic; fname = sprintf('test_save_%d.mat', j); save(fname, 'x'); toc; end disp('using fwrite:'); j = 1:10 tic; fname = sprintf('test_fwrite_%d.mat', j); fileid = fopen(fname,'w'); fwrite(fileid, x, 'double'); fclose(fileid); toc; end disp('using memory map:'); j = 1:10 tic; fname = sprintf('test_fwrite_%d.mat', j); m = memmapfile(fname, ... 'format','double'); m.writable = true; m.data = x; toc; end clear m;
output of above code:
>> using h5write: elapsed time 0.120691 seconds. elapsed time 0.123445 seconds. elapsed time 0.109917 seconds. elapsed time 2.773781 seconds. elapsed time 0.104730 seconds. elapsed time 0.115631 seconds. elapsed time 0.106282 seconds. elapsed time 0.119643 seconds. elapsed time 0.109200 seconds. elapsed time 0.110083 seconds. using save: elapsed time 2.774994 seconds. elapsed time 2.776656 seconds. elapsed time 2.777405 seconds. elapsed time 2.811365 seconds. elapsed time 2.780027 seconds. elapsed time 2.775661 seconds. elapsed time 2.776308 seconds. elapsed time 2.800510 seconds. elapsed time 2.783472 seconds. elapsed time 2.782114 seconds. using fwrite: elapsed time 0.035606 seconds. elapsed time 0.094308 seconds. elapsed time 0.033619 seconds. elapsed time 0.034147 seconds. elapsed time 0.033776 seconds. elapsed time 0.034627 seconds. elapsed time 0.033806 seconds. elapsed time 0.035084 seconds. elapsed time 0.034562 seconds. elapsed time 0.039331 seconds. using memory map: elapsed time 0.079036 seconds. elapsed time 0.035585 seconds. elapsed time 0.035654 seconds. elapsed time 0.036398 seconds. elapsed time 0.035054 seconds. elapsed time 0.039047 seconds. elapsed time 0.036286 seconds. elapsed time 0.035016 seconds. elapsed time 0.036145 seconds. elapsed time 0.036497 seconds.
output of ver command:
>> ver ---------------------------------------------------------------------------------------------------- matlab version: 8.2.0.701 (r2013b) matlab license number: xxxxxx operating system: microsoft windows server 2008 r2 version 6.1 (build 7601: service pack 1) java version: java 1.7.0_11-b21 oracle corporation java hotspot(tm) 64-bit server vm mixed mode ---------------------------------------------------------------------------------------------------- matlab version 8.2 (r2013b) database toolbox version 5.0 (r2013b) statistics toolbox version 8.3 (r2013b)
update: i've tried:
- rebooting box.
- i have identical box (h/w, os) problem not occur on. killed processes on slow box not on normal box. did not make difference.
- when change data random, makes no difference.
- when use save command instead, times consistent. makes me think not general disk issue, setup , hdf5.
Comments
Post a Comment