ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-21181] Release byteBuffers to suppress netty error messages

Browse source 
## What changes were proposed in this pull request?
We are explicitly calling release on the byteBuf's used to encode the string to Base64 to suppress the memory leak error message reported by netty. This is to make it less confusing for the user.

### Changes proposed in this fix
By explicitly invoking release on the byteBuf's we are decrement the internal reference counts for the wrappedByteBuf's. Now, when the GC kicks in, these would be reclaimed as before, just that netty wouldn't report any memory leak error messages as the internal ref. counts are now 0.

## How was this patch tested?
Ran a few spark-applications and examined the logs. The error message no longer appears.

Original PR was opened against branch-2.1 => https://github.com/apache/spark/pull/18392

Author: Dhruve Ashar <dhruveashar@gmail.com>

Closes #18407 from dhruve/master.
This commit is contained in:
Dhruve Ashar 2017-06-23 10:36:29 -07:00 committed by Marcelo Vanzin
parent b803b66a81
commit 1ebe7ffe07
1 changed files with 22 additions and 4 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

26
common/network-common/src/main/java/org/apache/spark/network/sasl/SparkSaslServer.java
View file

@ -34,6 +34,7 @@ import java.util.Map;
import com.google.common.base.Preconditions;
import com.google.common.base.Throwables;
import com.google.common.collect.ImmutableMap;
import io.netty.buffer.ByteBuf;
import io.netty.buffer.Unpooled;
import io.netty.handler.codec.base64.Base64;
import org.slf4j.Logger;
@ -187,14 +188,31 @@ public class SparkSaslServer implements SaslEncryptionBackend {
/* Encode a byte[] identifier as a Base64-encoded string. */
public static String encodeIdentifier(String identifier) {
Preconditions.checkNotNull(identifier, "User cannot be null if SASL is enabled");
return Base64.encode(Unpooled.wrappedBuffer(identifier.getBytes(StandardCharsets.UTF_8)))
.toString(StandardCharsets.UTF_8);
return getBase64EncodedString(identifier);
}
/** Encode a password as a base64-encoded char[] array. */
public static char[] encodePassword(String password) {
Preconditions.checkNotNull(password, "Password cannot be null if SASL is enabled");
return Base64.encode(Unpooled.wrappedBuffer(password.getBytes(StandardCharsets.UTF_8)))
.toString(StandardCharsets.UTF_8).toCharArray();
return getBase64EncodedString(password).toCharArray();
}
/** Return a Base64-encoded string. */
private static String getBase64EncodedString(String str) {
ByteBuf byteBuf = null;
ByteBuf encodedByteBuf = null;
try {
byteBuf = Unpooled.wrappedBuffer(str.getBytes(StandardCharsets.UTF_8));
encodedByteBuf = Base64.encode(byteBuf);
return encodedByteBuf.toString(StandardCharsets.UTF_8);
} finally {
// The release is called to suppress the memory leak error messages raised by netty.
if (byteBuf != null) {
byteBuf.release();
if (encodedByteBuf != null) {
encodedByteBuf.release();
}
}
}
}
}