ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
spark-instrumented-optimizer/ec2
History
Simon Hafner 70beb808e1 [SPARK-8821] [EC2] Switched to binary mode for file reading 
Otherwise the script will crash with

    - Downloading boto...
    Traceback (most recent call last):
      File "ec2/spark_ec2.py", line 148, in <module>
        setup_external_libs(external_libs)
      File "ec2/spark_ec2.py", line 128, in setup_external_libs
        if hashlib.md5(tar.read()).hexdigest() != lib["md5"]:
      File "/usr/lib/python3.4/codecs.py", line 319, in decode
        (result, consumed) = self._buffer_decode(data, self.errors, final)
    UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

In case of an utf8 env setting.

Author: Simon Hafner <hafnersimon@gmail.com>

Closes #7215 from reactormonk/branch-1.4 and squashes the following commits:

e86957a [Simon Hafner] [SPARK-8821] [EC2] Switched to binary mode

(cherry picked from commit 83a621a5a8)
Signed-off-by: Shivaram Venkataraman <shivaram@cs.berkeley.edu>
2015-07-07 09:43:16 -07:00
..
deploy.generic/root/spark-ec2 [SPARK-6402][DOC] - Remove some refererences to shark in docs and ec2 2015-03-19 08:02:06 -04:00
README Remove remaining references to incubation 2014-03-02 01:00:16 -08:00
spark-ec2 [SPARK-5434] [EC2] Preserve spaces in EC2 path 2015-01-28 12:56:03 -08:00
spark_ec2.py [SPARK-8821] [EC2] Switched to binary mode for file reading 2015-07-07 09:43:16 -07:00

README 

This folder contains a script, spark-ec2, for launching Spark clusters on
Amazon EC2. Usage instructions are available online at:

http://spark.apache.org/docs/latest/ec2-scripts.html