ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
spark-instrumented-optimizer/dev
History
Brennon York df3550084c [HOTFIX][SPARK-4123]: Updated to fix bug where multiple dependencies added breaks Github output 
Currently there is a bug whereby if a new patch introduces more than one new dependency (or removes more than one) it breaks the Github post output (see [this build](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29399/consoleFull)). This hotfix will remove `awk` print statements in place of `printf` so as not to automatically add the newline character which is then escaped and added directly at the end of the `awk` statement. This should take a failed build output such as:

```json
data: {"body": "  [Test build #29400 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29400/consoleFull) for   PR 5266 at commit [`2aa4be0`](2aa4be0e1d).\n * This patch **passes all tests**.\n * This patch merges cleanly.\n * This patch adds the following public classes _(experimental)_:\n  * `class IDF extends Estimator[IDFModel] with IDFParams `\n  * `class Normalizer extends UnaryTransformer[Vector, Vector, Normalizer] `\n\n * This patch **adds the following new dependencies:**\n   * `avro-1.7.7.jar`
   * `breeze-macros_2.10-0.11.2.jar`
   * `breeze_2.10-0.11.2.jar`\n * This patch **removes the following dependencies:**\n   * `avro-1.7.6.jar`
   * `breeze-macros_2.10-0.11.1.jar`
   * `breeze_2.10-0.11.1.jar`"}
```

and turn it into:

```json
data: {"body": "  [Test build #29400 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29400/consoleFull) for   PR 5266 at commit [`2aa4be0`](2aa4be0e1d).\n * This patch **passes all tests**.\n * This patch merges cleanly.\n * This patch adds the following public classes _(experimental)_:\n  * `class IDF extends Estimator[IDFModel] with IDFParams `\n  * `class Normalizer extends UnaryTransformer[Vector, Vector, Normalizer] `\n\n * This patch **adds the following new dependencies:**\n   * `avro-1.7.7.jar`\n   * `breeze-macros_2.10-0.11.2.jar`\n   * `breeze_2.10-0.11.2.jar`\n * This patch **removes the following dependencies:**\n   * `avro-1.7.6.jar`\n   * `breeze-macros_2.10-0.11.1.jar`\n   * `breeze_2.10-0.11.1.jar`"}
```

I've tested this locally and all worked.

/cc srowen pwendell nchammas

Author: Brennon York <brennon.york@capitalone.com>

Closes #5269 from brennonyork/HOTFIX-SPARK-4123 and squashes the following commits:

a441068 [Brennon York] Updated awk to use printf and to manually insert newlines so that the JSON github string when posted is corrected
2015-03-30 12:48:26 -07:00
..
audit-release [Release] Bring audit scripts up-to-date 2014-11-12 16:35:39 -08:00
create-release HOTFIX: Changes to release script. 2015-03-12 18:37:19 -07:00
tests [HOTFIX][SPARK-4123]: Updated to fix bug where multiple dependencies added breaks Github output 2015-03-30 12:48:26 -07:00
change-version-to-2.10.sh SPARK-5143 [BUILD] [WIP] spark-network-yarn 2.11 depends on spark-network-shuffle 2.10 2015-03-04 21:00:51 -08:00
change-version-to-2.11.sh SPARK-5143 [BUILD] [WIP] spark-network-yarn 2.11 depends on spark-network-shuffle 2.10 2015-03-04 21:00:51 -08:00
check-license [SPARK-5474][Build]curl should support URL redirection in build/mvn 2015-02-05 12:03:13 -08:00
github_jira_sync.py SPARK-2596 HOTFIX: Deal with non-existent JIRAs. 2014-07-19 20:06:28 -07:00
lint-python [SPARK-6219] [Build] Check that Python code compiles 2015-03-19 12:46:10 -07:00
lint-scala [SPARK-2627] [PySpark] have the build enforce PEP 8 automatically 2014-08-06 12:58:24 -07:00
merge_spark_pr.py SPARK-5776 JIRA version not of form x.y.z breaks merge_spark_pr.py 2015-02-12 20:14:45 +00:00
mima [SPARK-4501][Core] - Create build/mvn to automatically download maven/zinc/scalac 2014-12-27 13:26:38 -08:00
README.md Merge pull request #565 from pwendell/dev-scripts. Closes #565. 2014-02-08 23:13:34 -08:00
run-tests [SPARK-6477][Build]: Run MIMA tests before the Spark test suite 2015-03-24 10:33:04 +00:00
run-tests-codes.sh [SPARK-6477][Build]: Run MIMA tests before the Spark test suite 2015-03-24 10:33:04 +00:00
run-tests-jenkins [SPARK-4123][Project Infra]: Show new dependencies added in pull requests 2015-03-29 12:37:53 +01:00
scalastyle [SPARK-4501][Core] - Create build/mvn to automatically download maven/zinc/scalac 2014-12-27 13:26:38 -08:00

README.md

Spark Developer Scripts

This directory contains scripts useful to developers when packaging, testing, or committing to Spark.

Many of these scripts require Apache credentials to work correctly.