Thursday, July 15, 2010

Java ThreadPool and Large Training

A little shocking discovery we had on the Java ThreadPoolExecutor implementation: our large training cluster would somehow never filled up with a specific kind of training task. The culprit was eventually identified in the ThreadPoolExecutor, which gracefully queued up all tasks. I always thought new threads will be created when the queue is not empty and the number of threads is less than the maxPoolSize. Obviously, I did not read the fine-print in JavaDoc. There were quite a few different strategies described in ThreadPoolExecutor: Direct Handoffs (never queue, always create new thread till reach max), Unbounded queues (always queue, and never reach max threads) and Bounded queues (queue to a certain size, then start ceating threads till max threads, then rejects).

My assumption, Direct Handoffs+Unbounded Queue (use as many threads till reach maxPoolSize, then queue up unbounded), was obviously not one of them, even though I would think this is definitely favourable strategy in most cases. Anyway, we implemented this using a customized LinkedBlockingQueue.

Lessons learned: (like dealing with the bank,) always read the fine-print;).

8 comments:

  1. Do your guys use a pure java trainer for AMs or make system calls to binaries? That may make a quite big difference on the design of the parallelization platform. However I wonder if there exists toolkits for AM training in pure java...

    ReplyDelete
  2. Works for me with such customized queue:

    static class ThreadPoolTaskQueue extends LinkedBlockingQueue{

    private ThreadPoolExecutor threadPool = null;

    @Override
    public boolean offer(Runnable e) {
    if(threadPool != null && threadPool.getPoolSize() < threadPool.getMaximumPoolSize())
    return false;
    else
    return super.offer(e);
    }

    public void setThreadPool(ThreadPoolExecutor threadPool){
    this.threadPool = threadPool;
    }
    }

    ReplyDelete
  3. Hi, Great.. Tutorial is just awesome..It is really helpful for a newbie like me.. I am a regular follower of your blog. Really very informative post you shared here. Kindly keep blogging. If anyone wants to become a Java developer learn from Java Training in Chennai. or learn thru Java Online Training from India . Nowadays Java has tons of job opportunities on various vertical industry.

    ReplyDelete
  4. Nice Information Your first-class knowledge of this great job can become a suitable foundation for these people. I did some research on the subject and found that almost everyone will agree with your blog.
    Cyber Security Course in Bangalore

    ReplyDelete
  5. Writing in style and getting good compliments on the article is hard enough, to be honest, but you did it so calmly and with such a great feeling and got the job done. This item is owned with style and I give it a nice compliment. Better!
    Cyber Security Training in Bangalore

    ReplyDelete
  6. Really, this article is truly one of the best, information shared was valuable and resourceful Very good work thank you.
    Data Scientist Training in Hyderabad

    ReplyDelete