python - Error handling in PySpark reading in non existent files -

-

i have massive list of directories , files potentially read from. of them may not exist not problem - ignore error - using try method. there way can allow in pyspark.

here returned error message:

py4j.protocol.py4jjavaerror: error occurred while calling z:org.apache.spark.api.python.pythonrdd.collectandserve. : org.apache.hadoop.mapred.invalidinputexception: input path not exist: file:

i building series of files @ following:

sci = sc.textfile(",".join(paths))

where paths list of paths possible files. check file system , see if exist, there more elegent way of doing this?

the following should work:

for f in file_list:    try:        read_file(f)    except org.apache.hadoop.mapred.invalidinputexception:        deal_with_absent_file(f)

Comments

Post a Comment

Popular posts from this blog

How has firefox/gecko HTML+CSS rendering changed in version 38? -

-
i have a little website header no longer renders correctly, since updating firefox 38.0.5 [the faceplant logo missing top right, , main logo no longer centred]. still renders expected in other browsers. how has engine's rendering changed in update? according mdn , list of changes is: css support ruby-position , ruby-align have been added , available default ( bug 1055676 , bug 1123917 , bug 1039006 ). the :unresolved pseudo-class has been implemented custom elements ( bug 1111633 ). the predefined style ethiopic-numeric uses space, instead of dot, suffix match recent change spec ( bug 1120721 ). css transitions on generated content (with ::before , ::after ) on both inline , block splits them start expected specification ( bug 1110277 ). the implementation of css logical properties made big progress. [...] how css transitions start has been modified match recent change of specifications, aiming @ having interoperable behavio
Read more

javascript - Complex json ng-repeat -

-
i try ng-repeat on object, tried several times still stuck. if have idea, , if it's possible share tips kind of situation. [ { "abonnement": { "id": "1", "nom": "bas prix", "tarif": "2.49", "communication": "2h de communication", "vers": "vers fixes et mobiles", "zone": "zone nationale" } }, { "abonnement": { "id": "2", "nom": "national", "tarif": "9.99", "communication": "t\u00e9l\u00e9phonie et sms illimit\u00e9s", "vers": "vers fixes et mobiles", "zone": "zone nationale" } }, { "abonnement": { "id": "3", "nom": "euro", "tarif&
Read more

jquery - Cloning of rows and columns from the old table into the new with colSpan and rowSpan -

-
i using clone jquery. necessary clone gray area in new table. problem cloned rows , columns can has rowspan , colspan. the gray area intersection data-fix-cols , data-fix-rows. my solution not work: <table id="old-table" data-fix-cols='3' data-fix-rows='4'> <tbody> <tr> <td class="gray">clone me</td> <td colspan="2" rowspan="2" class="gray">clone me</td> <td>ashgabat</td> <td>barcelona</td> <td>berlin</td> </tr> <tr> <td rowspan="3" class="gray">clone me</td> <td>bucharest</td> <td>warsaw</td> <td>washington</td> </tr> <tr> <td class="gray">clone me</td>
Read more