On manual reindex, we do a sync delete, following by many async reindexing. If we kick off a cluster loading request right after submitted all async jobs, there is a big chance that cluster peer is seeing a incomplete index dump(especially when there is a lot of async jobs).

This fix enforce a backend MessageBus synchronizer right after we submit all async jobs. Due to Liferay ThreadPool ensures FIFO, once we notice all search writer threads are synced up on the synchronizer, we know for sure all async reindexing are finish, then it is safe to trigger a cluster index files loading.

There is a window time that starting from first search writer thread arrives synchronizer, ending at last search writer thread arrives synchronizer, within this window time search writer ThreadPool's performance is declining linearly down to 0, then immediately backup to 100%.
The window time Tw depends on the ThreadPool size N and potential reindexing complexity for single element C.
Tw=(N-1)*ave(C)

Generally, N is about 5~20, C is at about 1~100 ms level, so Tw is actually really small.
But to ensure robustness, a timeout is added to synchronizer waiting. Default to 1 min, should be more than enough. On timeout(this will be really rare case), the current reindex may not be properly replicated to cluster peer(but the local index files should be correct), an error log will be outputed, administrator should schedule another reindex to fix this.