Background

Deployment system uses minimum 2 instances to deploy job. But only 1 is needed. So planning to use Apache Curator for leader election. So only 1 instance will launch the job and other will be blocked

Problem

1. Default behavior is the that the follower becomes leader when original leader is restarted.
2. Now that previously the follower was blocking is unblocked and restarts / redeploys the job again
3. This is not expected, only 1 instance should control the job.
4. Is there any way to stored status data in ZK itself and retrieve before starting the job, so that if previous status is blank, then current host is true leader and if it had follower status previously the execution can be skipped.

public void deploy() throws Exception {
    if (isLeaderElectionEnabled) {
      // Blocking if not a leader
      while (!leaderElector.isLeader()) { // This will be true when original leader stops and the follower becomes leader. In that case too we don't want to deploy job
// Can some data stored in ZK be checked here
        continue;
      }
    }
    // Started leader election check if this is leader
    LOG.info("DEPLOYING JOB");
    // Job launch code
}

Any other recommendation for implementing this ?

Edit 1

Was able to rope in Apache Curator library and proceed to leader election. But this problem is not exactly same as leader election.

It should handle restart and redeploy scenario by avoiding multiple restarts.

Edit 2

Elaborating scenarios and a possible solution

Premise

• Host A and Host B
• A : Leader, launched the job and has leader latch ( curator lib )
• B : Follower blocking

Restart/Redeploy

• Case 1 --> A restarts first

  • A looses the leader latch
  • B is leader and is unblocked <-- restart / curr. version
  • A comes up and acts as follower
  • B restarts now losing the leadership
  • A is unblocked being a leader <-- restart / new version = B comes up and acts as follower

• Case 2 --> B restarts first

  • B comes up and acts as follower = A restarts now losing the leadership
  • B is leader and is unblocked <-- restart / new version
  • A comes up and acts as follower

Note

The question though mentions Java, this is what I am trying to achieve. The approach is expected, the solution need not be in Java specifically.