ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-14254][CORE] Add logs to help investigate the network performance

Browse source 
## What changes were proposed in this pull request?

It would be very helpful for network performance investigation if we log the time spent on connecting and resolving host.

## How was this patch tested?

Jenkins unit tests.

Author: Shixiong Zhu <shixiong@databricks.com>

Closes #12046 from zsxwing/connection-time.
This commit is contained in:
Shixiong Zhu 2016-03-29 21:14:48 -07:00
parent b66b97cd04
commit 7320f9bd19
1 changed files with 4 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

5
common/network-common/src/main/java/org/apache/spark/network/client/TransportClientFactory.java
View file

@ -123,7 +123,10 @@ public class TransportClientFactory implements Closeable {
public TransportClient createClient(String remoteHost, int remotePort) throws IOException {
// Get connection from the connection pool first.
// If it is not found or not active, create a new one.
long preResolveHost = System.nanoTime();
final InetSocketAddress address = new InetSocketAddress(remoteHost, remotePort);
long hostResolveTimeMs = (System.nanoTime() - preResolveHost) / 1000000;
logger.info("Spent {} ms to resolve {}", hostResolveTimeMs, address);
// Create the ClientPool if we don't have it yet.
ClientPool clientPool = connectionPool.get(address);
@ -235,7 +238,7 @@ public class TransportClientFactory implements Closeable {
}
long postBootstrap = System.nanoTime();
logger.debug("Successfully created connection to {} after {} ms ({} ms spent in bootstraps)",
logger.info("Successfully created connection to {} after {} ms ({} ms spent in bootstraps)",
address, (postBootstrap - preConnect) / 1000000, (postBootstrap - preBootstrap) / 1000000);
return client;