Fixes feature introduced in #7508 to use the default value if nothing is specified in command line
cc liancheng rxin pwendell
Author: Shivaram Venkataraman <shivaram@cs.berkeley.edu>
Closes#7558 from shivaram/merge-script-fix and squashes the following commits:
7092141 [Shivaram Venkataraman] Use default primary author if unspecified
It's a common case that some contributor contributes an initial version of a feature/bugfix, and later on some other people (mostly committers) fork and add more improvements. When merging these PRs, we probably want to specify the original author as the primary author. Currently we can only do this by running
```
$ git commit --amend --author="name <email>"
```
manually right before the merge script pushes to Apache Git repo. It would be nice if the script accepts user specified primary author information.
Author: Cheng Lian <lian@databricks.com>
Closes#7508 from liancheng/spark-9179 and squashes the following commits:
218d88e [Cheng Lian] Allows committers to specify primary author of the PR to be merged
This commit allows `dev/merge_spark_pr.py` to use personal GitHub OAuth tokens in order to make authenticated requests. This is necessary to work around per-IP rate limiting issues.
To use a token, just set the `GITHUB_OAUTH_KEY` environment variable. You can create a personal token at https://github.com/settings/tokens; we only require `public_repo` scope.
If the script fails due to a rate-limit issue, it now logs a useful message directing the user to the OAuth token instructions.
Author: Josh Rosen <joshrosen@databricks.com>
Closes#7136 from JoshRosen/pr-merge-script-oauth-authentication and squashes the following commits:
4d011bd [Josh Rosen] Fix error message
23d92ff [Josh Rosen] Support GitHub OAuth tokens in dev/merge_spark_pr.py
This patch addresses a critical issue in the PySpark tests:
Several of our Python modules' `__main__` methods call `doctest.testmod()` in order to run doctests but forget to check and handle its return value. As a result, some PySpark test failures can go unnoticed because they will not fail the build.
Fortunately, there was only one test failure which was masked by this bug: a `pyspark.profiler` doctest was failing due to changes in RDD pipelining.
Author: Josh Rosen <joshrosen@databricks.com>
Closes#7032 from JoshRosen/testmod-fix and squashes the following commits:
60dbdc0 [Josh Rosen] Account for int vs. long formatting change in Python 3
8b8d80a [Josh Rosen] Fix failing test.
e6423f9 [Josh Rosen] Check return code for all uses of doctest.testmod().
Looks like this was added by accident when pwendell merged a commit back in September: fe2b1d6a20
Author: Kay Ousterhout <kayousterhout@gmail.com>
Closes#6485 from kayousterhout/SPARK-7933 and squashes the following commits:
7c6164a [Kay Ousterhout] [SPARK-7933] Remove Patrick's username/pw from merge script
jira-pytyhon package should be installed by
sudo pip install jira
cc pwendell
Author: Davies Liu <davies@databricks.com>
Closes#6367 from davies/fix_jira_python2 and squashes the following commits:
fbb3c8e [Davies Liu] Fix install jira-python
The issue is that the behavior of the ASF JIRA silently
changed. Now when the "Resolve Issue" transition occurs,
the default resolution is "Pending Closed". We used to
count on the default behavior being to set the
resolution as "Fixed".
The solution is to explicitly set the resolution as "Fixed" and not
count on default behavior.
Author: Patrick Wendell <patrick@databricks.com>
Closes#6103 from pwendell/merge-script-fix and squashes the following commits:
dcc16a6 [Patrick Wendell] Always set resolution to "Fixed" in PR merge script.
Cleans up the pull request title in the merge script to follow conventions outlined in the wiki under Contributing Code.
https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark#ContributingtoSpark-ContributingCode
[MODULE] SPARK-XXXX: Description
Author: texasmichelle <texasmichelle@gmail.com>
Closes#5149 from texasmichelle/master and squashes the following commits:
9b6b0a7 [texasmichelle] resolved variable scope issue
7d5fa20 [texasmichelle] only prompt if title has been modified
8c195bb [texasmichelle] removed erroneous line
4f1ed46 [texasmichelle] Deque removal, logic simplifications, & prompt user to pick a title (orig or modified)
df73f6a [texasmichelle] reworked regex's to enforce brackets around JIRA ref
43b5aed [texasmichelle] Merge remote-tracking branch 'apache/master'
25229c6 [texasmichelle] Merge remote-tracking branch 'apache/master'
aa20a6e [texasmichelle] Move code into main() and add doctest for new text parsing method
48520ba [texasmichelle] SPARK-1684: Corrected import statement
042099d [texasmichelle] SPARK-1684 Merge script should standardize SPARK-XXX prefix
8f4a7d1 [texasmichelle] SPARK-1684 Merge script should standardize SPARK-XXX prefix
Consider only x.y.z verisons from JIRA. CC JoshRosen who will probably know this script well.
Alternative is to call the version "2.0.0" after all in JIRA.
Author: Sean Owen <sowen@cloudera.com>
Closes#4570 from srowen/SPARK-5776 and squashes the following commits:
fffafde [Sean Owen] Consider only x.y.z verisons from JIRA
This will fix SPARK-4507.
For pull requests that reference multiple JIRAs in their titles, it would be helpful if the PR merge script offered to close all of them.
Author: Takayuki Hasegawa <takayuki.hasegawa0311@gmail.com>
Closes#3428 from hase1031/SPARK-4507 and squashes the following commits:
bf6d64b [Takayuki Hasegawa] SPARK-4507: try to resolve issue when no JIRAs in title
401224c [Takayuki Hasegawa] SPARK-4507: moved codes as before
ce89021 [Takayuki Hasegawa] SPARK-4507: PR merge script should support closing multiple JIRA tickets
The merge script builds up a big command array and sometimes
this contains both unicode and ascii strings. This doesn't work
if you try to join them into a single string. Longer term a solution
is to go and make sure the source of all strings is unicode.
This patch provides a simpler solution... just print the array
rather than joining. I actually prefer printing an array here
anyways since joining on spaces is lossy in the case of arguments
that themselves contain spaces.
Author: Patrick Wendell <pwendell@gmail.com>
Closes#2645 from pwendell/merge-script and squashes the following commits:
167b792 [Patrick Wendell] HOTFIX: Fix unicode error in merge script.
Closes#2387
Author: Matthew Farrellee <matt@redhat.com>
Closes#2301 from mattf/SPARK-3425 and squashes the following commits:
20f3c09 [Matthew Farrellee] [SPARK-3425] do not set MaxPermSize for OpenJDK 1.8
As sryza reported, spark-shell doesn't accept any flags.
The root cause is wrong usage of spark-submit in spark-shell and it come to the surface by #1801
Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp>
Author: Cheng Lian <lian.cs.zju@gmail.com>
Closes#1715, Closes#1864, and Closes#1861Closes#1825 from sarutak/SPARK-2894 and squashes the following commits:
47f3510 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-2894
2c899ed [Kousuke Saruta] Removed useless code from java_gateway.py
98287ed [Kousuke Saruta] Removed useless code from java_gateway.py
513ad2e [Kousuke Saruta] Modified util.sh to enable to use option including white spaces
28a374e [Kousuke Saruta] Modified java_gateway.py to recognize arguments
5afc584 [Cheng Lian] Filter out spark-submit options when starting Python gateway
e630d19 [Cheng Lian] Fixing pyspark and spark-shell CLI options
This commit fixes a couple of issues in the merge_spark_pr.py developer script:
- Allow recovery from failed cherry-picks.
- Fix detection of pull requests that have already been merged.
Both of these fixes are useful when backporting changes.
Author: Josh Rosen <joshrosen@apache.org>
Closes#1668 from JoshRosen/pr-script-improvements and squashes the following commits:
ff4f33a [Josh Rosen] Default SPARK_HOME to cwd(); detect missing JIRA credentials.
ed5bc57 [Josh Rosen] Improvements for backporting using merge_spark_pr:
Currently all of the commits with 'X' in them cause person X to
receive e-mails every time someone makes a public fork of Spark.
marmbrus who requested this.
Author: Patrick Wendell <pwendell@gmail.com>
Closes#1239 from pwendell/strip and squashes the following commits:
22e5a97 [Patrick Wendell] Strip '@' symbols when merging pull requests.
Discovered in #992
Author: Patrick Wendell <pwendell@gmail.com>
Closes#1007 from pwendell/hotfix and squashes the following commits:
af90aa0 [Patrick Wendell] HOTFIX: Support empty body in merge script
A minor change to use env default python instead of fixed `/usr/bin/python`.
Author: Xiangrui Meng <meng@databricks.com>
Closes#965 from mengxr/merge-pr-python and squashes the following commits:
1ae0013 [Xiangrui Meng] use env default python in merge_spark_pr.py
Author: Reynold Xin <rxin@apache.org>
Closes#875 from rxin/pep8-dev-scripts and squashes the following commits:
04b084f [Reynold Xin] Made dev Python scripts PEP8 compliant.
Adds an automated hook in the merge script that can close the JIRA,
set the fix versions, and leave a comment on the JIRA indicating the
PR in which it was resolved. This ensures that (a) we always close JIRA's
when issues are merged and (b) there is a link to the pull request in every JIRA.
This requires a python library called `jira-client`. We could look at embedding this
library in our project, but it seemed simple enough to just gracefully disable this
feature if it is not installed. It can be installed with `pip install jira-client`.
Author: Patrick Wendell <pwendell@gmail.com>
Closes#570 from pwendell/jira-pr-merge and squashes the following commits:
3022b96 [Patrick Wendell] SPARK-1648 Support closing JIRA's as part of merge script.
Sometimes if there is a small conflict it's nice to be able to just
manually fix it up rather than have another RTT with the contributor.
Author: Patrick Wendell <pwendell@gmail.com>
Closes#342 from pwendell/merge-conflicts and squashes the following commits:
cdce61a [Patrick Wendell] SPARK-1431: Allow merging conflicting pull requests
Author: Thomas Graves <tgraves@apache.org>
Closes#47 from tgravescs/fix_merge_script and squashes the following commits:
8209ab1 [Thomas Graves] Update dev merge script to use spark.git instead of incubator-spark
The first line of a git commit message is the line that's used with many git
tools as the most concise textual description of that message. The most
common use that I see is in the short log, which is a one line per commit
log of recent commits.
This commit moves the line
Merge pull request #%s from %s.
Lower into the message to reserve the first line of the resulting commit for
the much more important pull request title.
http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html
Author: Andrew Ash <andrew@andrewash.com>
Closes#574 from ash211/gh-pr-merge-title and squashes the following commits:
b240823 [Andrew Ash] More merge_message improvements
d2986db [Andrew Ash] Keep GitHub pull request title as commit summary
Fixes bug where merges won't close associated pull request.
Previously we added "Closes #XX" in the title. Github will sometimes
linbreak the title in a way that causes this to not work. This patch
instead adds the line in the body.
This also makes the commit format more concise for merge commits.
We might consider just dropping those in the future.
Author: Patrick Wendell <pwendell@gmail.com>
Closes#569 and squashes the following commits:
732eba1 [Patrick Wendell] Fixes bug where merges won't close associated pull request.
SPARK-1066: Add developer scripts to repository.
These are some developer scripts I've been maintaining in a separate public repo. This patch adds them to the Spark repository so they can evolve here and are clearly accessible to all committers.
I may do some small additional clean-up in this PR, but wanted to put them here in case others want to review. There are a few types of scripts here:
1. A tool to merge pull requests.
2. A script for packaging releases.
3. A script for auditing release candidates.
Author: Patrick Wendell <pwendell@gmail.com>
== Merge branch commits ==
commit 5d5d331d01f6fd59c2eb830f652955119b012173
Author: Patrick Wendell <pwendell@gmail.com>
Date: Sat Feb 8 22:11:47 2014 -0800
SPARK-1066: Add developer scripts to repository.
