A little shocking discovery we had on the Java ThreadPoolExecutor implementation: our large training cluster would somehow never filled up with a specific kind of training task. The culprit was eventually identified in the ThreadPoolExecutor, which gracefully queued up all tasks. I always thought new threads will be created when the queue is not empty and the number of threads is less than the maxPoolSize. Obviously, I did not read the fine-print in JavaDoc. There were quite a few different strategies described in ThreadPoolExecutor: Direct Handoffs (never queue, always create new thread till reach max), Unbounded queues (always queue, and never reach max threads) and Bounded queues (queue to a certain size, then start ceating threads till max threads, then rejects).
My assumption, Direct Handoffs+Unbounded Queue (use as many threads till reach maxPoolSize, then queue up unbounded), was obviously not one of them, even though I would think this is definitely favourable strategy in most cases. Anyway, we implemented this using a customized LinkedBlockingQueue.
Lessons learned: (like dealing with the bank,) always read the fine-print;).
-
Hadoop! Hadoop! Hadoop!
ReplyDeleteI know, I know, I know...
ReplyDeleteDo your guys use a pure java trainer for AMs or make system calls to binaries? That may make a quite big difference on the design of the parallelization platform. However I wonder if there exists toolkits for AM training in pure java...
ReplyDeleteWorks for me with such customized queue:
ReplyDeletestatic class ThreadPoolTaskQueue extends LinkedBlockingQueue{
private ThreadPoolExecutor threadPool = null;
@Override
public boolean offer(Runnable e) {
if(threadPool != null && threadPool.getPoolSize() < threadPool.getMaximumPoolSize())
return false;
else
return super.offer(e);
}
public void setThreadPool(ThreadPoolExecutor threadPool){
this.threadPool = threadPool;
}
}
Hi, Great.. Tutorial is just awesome..It is really helpful for a newbie like me.. I am a regular follower of your blog. Really very informative post you shared here. Kindly keep blogging. If anyone wants to become a Java developer learn from Java Training in Chennai. or learn thru Java Online Training from India . Nowadays Java has tons of job opportunities on various vertical industry.
ReplyDelete