splitBrain

Class PRNetDownTest

java.lang.Object

parReg.ParRegTest

splitBrain.PRNetDownTest

public class PRNetDownTest
extends ParRegTest

A version of parReg.ParRegTest that allows time to re-establish redundancy after the network connections have been dropped before validation. Surviving side VMs execute concurrent operations, and then validate the cache and PR metadeta. In this version, VMs will wait SplitBrainPrms.dropWaitTimeSec after a network drop before performing validation.

Field Summary

Fields

Modifier and Type	Field and Description
protected static String	CONCURRENT_LEADER_THREADID

Fields inherited from class parReg.ParRegTest

aRegion, bridgeOrderingWorkaround, cacheDef, cacheIsClosed, CLOSE_CACHE, CLOSE_REGION, concVerifyCoordinator, CREATE_INDEX_OPERATION, createCallbackPrefix, currentIntervalKeyIndex, currentValidateKeyIndex, cycleVMs_notChosenVMs, cycleVMs_targetVMs, destroyCallbackPrefix, destroyedKeys, DISCONNECT, disconnected, diskFilesRecorded, distLockService, ENTRY_ADD_OPERATION, ENTRY_DESTROY_OPERATION, ENTRY_DESTROY_PERCENT_OPERATION, ENTRY_GET_NEW_OPERATION, ENTRY_GET_OPERATION, ENTRY_INVALIDATE_OPERATION, ENTRY_LOCAL_DESTROY_OPERATION, ENTRY_LOCAL_DESTROY_PERCENT_OPERATION, ENTRY_LOCAL_INVALIDATE_OPERATION, ENTRY_PUT_IF_ABSENT_AS_CREATE_OPERATION, ENTRY_PUT_IF_ABSENT_AS_GET_OPERATION, ENTRY_PUT_IF_ABSENT_OPERATION, ENTRY_PUTALL_OPERATION, ENTRY_REMOVE_AS_NOOP_OPERATION, ENTRY_REMOVE_OPERATION, ENTRY_REPLACE_AS_NOOP_OPERATION, ENTRY_REPLACE_AS_UPDATE_OPERATION, ENTRY_REPLACE_NO_INVAL_OPERATION, ENTRY_REPLACE_OLD_AS_NOOP_OPERATION, ENTRY_REPLACE_OLD_AS_UPDATE_OPERATION, ENTRY_REPLACE_OLD_NO_INVAL_OPERATION, ENTRY_REPLACE_OLD_OPERATION, ENTRY_REPLACE_OPERATION, ENTRY_UPDATE_OPERATION, exceptionCount, expectOfflineExceptionKey, getCallbackPrefix, hasLRUEntryEviction, hasPRCacheLoader, highAvailability, invalidateCallbackPrefix, isBridgeClient, isBridgeConfiguration, isClientCache, isDataStore, isDataStoreKey, isEmptyClient, isSerialExecution, isThinClient, LOCAL_DESTROY, LOCK_NAME, LOCK_SERVICE_NAME, lockOperations, lowerThreshold, LRUEntryMax, MAX_CHOICES, MAX_CHOICES_FOR_CLIENT_SERVER, minTaskGranularityMS, minTaskGranularitySec, numOpsPerTask, numThreadsInClients, numThreadsInThisVM, numVMsToStop, offHeapVerifyTargetCount, QUERY_OPERATION, randomValues, RECENT_HISTORY, redundantCopies, regDef, regionDestroyCallbackPrefix, regionInvalidateCallbackPrefix, regionLocallyDestroyed, regionSnapshot, registerInterestCoordinator, REMOVE_INDEX_OPERATION, secondsToRun, shutDownAllInProgressKey, testInstance, thisVmRunningHAController, txDestroyedKeys, txRegionSnapshot, uniqueHostsOn, uniqueKeyIndex, uniqueKeys, updateCallbackPrefix, upperThreshold, VmIDStr

Constructor Summary

Constructors

Constructor and Description
PRNetDownTest()

Method Summary

Modifier and Type	Method and Description
protected void	concVerify() Modified version of ParRegTest.concVerify().
protected void	doConcOpsAndVerify() Do random operations and verification for concurrent tests.
static void	HydraTask_doConcOpsAndVerify() Hydra task method for concurrent tests with verification.
static void	HydraTask_initialize()
static void	HydraTask_initializeBridgeServer()

Methods inherited from class parReg.ParRegTest

addEntry, checkForLastIteration, checkForLastIteration, clearBBCriticalState, createObject, cycleVms, cycleVMsNoWait, deleteDir, deleteExistingDiskDirs, destroyEntry, destroyPercent, doConcOps, doEntryOperations, doIntervalOps, doRROpsAndVerify, doShutDownAllMembers, getCachePrmsName, getClientRegionConfigName, getIsDataStore, getKey, getNewKey, getNewKey, getOffHeapVerifyTargetCount, getOperation, getOperation, getRecentKey, getRegionConfigName, getUpdateObject, getValueForBB, getValueForKey, HA_reinitializeClientRegion, HA_reinitializeRegion, HA_reinitializeRegion, HAController, HADoEntryOps, handleException, HydraTask_assignBucketsToPartitions, HydraTask_disconnect, HydraTask_doConcOps, HydraTask_doEntryOps, HydraTask_doEntryOpsDataLoss, HydraTask_doIntervalOps, HydraTask_doRROpsAndVerify, HydraTask_HA_initializeAccessor, HydraTask_HA_initializeDataStore, HydraTask_HA_reinitializeAccessor, HydraTask_HA_reinitializeDataStore, HydraTask_HA_reinitializeWithClientCache, HydraTask_HAController, HydraTask_HADoEntryOps, HydraTask_initializeBridgeServerAccessor, HydraTask_initializeWithClientCache, HydraTask_initializeWithRegDef, HydraTask_loadToUpperThreshold, HydraTask_logLocalSize, HydraTask_prepareForRecovery, HydraTask_prepareForValidation, HydraTask_recoveryController, HydraTask_recoveryEntryOps, HydraTask_restoreBackups, HydraTask_shutDownAllHAController, HydraTask_shutDownAllHADoEntryOps, HydraTask_startVmAndRebalance, HydraTask_stopStartVMs, HydraTask_validateInternalPRState, HydraTask_validateIntervalOps, HydraTask_validatePR, HydraTask_validateRegionContents, HydraTask_verifyFromSnapshotAndSync, HydraTask_verifyRecoveredVMsFromSnapshotAndSync, HydraTask_waitForMyStartupRecovery, HydraTask_waitForStartupRecovery, HydraTask_writeDiskDirsToBB, initForIntervalValidation, initializeClientRegion, initializeInstance, initializeRegion, initializeRegion, initializeWithRegDef, initPdxDiskStore, invalidateEntry, logExecutionNumber, loseAndRecreatePR, putAll, putIfAbsent, putIfAbsentAsCreate, putIfAbsentAsGet, recordFailedOps, recoveryController, recoveryEntryOps, registerInterest, remove, removeAccessors, removeAsNoop, replace, replaceAsNoop, replaceAsUpdate, replaceOld, replaceOldAsNoop, replaceOldAsUpdate, restoreRegionSnapshot, runRestoreScript, saveRegionSnapshot, setUniqueKeyIndex, shutDownAllHAController, shutDownAllHADoEntryOps, updateEntry, validateAfterRestore, validateIntervalOps, verifyFromSnapshot, verifyFromSnapshotOnly, verifyHDFSRegionFromSnapshot, verifyInternalPRState, verifyServerKeysFromSnapshot, verifyThinClientFromSnapshot, waitForRecoveryAfterCycleVMs, waitForSelfRecovery, writeHDFSRegionSnapshotToBB

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

CONCURRENT_LEADER_THREADID

protected static String CONCURRENT_LEADER_THREADID

Constructor Detail

PRNetDownTest

public PRNetDownTest()

Method Detail

HydraTask_initializeBridgeServer

public static void HydraTask_initializeBridgeServer()

HydraTask_initialize

public static void HydraTask_initialize()

HydraTask_doConcOpsAndVerify

public static void HydraTask_doConcOpsAndVerify()

Hydra task method for concurrent tests with verification.

doConcOpsAndVerify

protected void doConcOpsAndVerify()

Do random operations and verification for concurrent tests. The task starts up and all threads concurrently do random operations. The operations run for maxTaskGranularitySec or numOpsPerTask, depending the on the defined hydra parameters, then all threads will pause. During the pause, one thread goes first and writes all known keys/values to the blackboard. Then all other threads read the blackboard and verify they have the same view. After all threads are done with verification, the task ends. In this PRNetDownTest version, the ConcurrentLeader coordinates with SBUtil.dropConnections via a hydra.blackboard.SharedLock. Either the ConcurrentLeader holds the lock (and verification is taking place) OR the dropConnections thread holds the lock and verification must wait for the lock to be released.

Overrides:

doConcOpsAndVerify in class ParRegTest

concVerify

protected void concVerify()

Modified version of ParRegTest.concVerify(). The ConcurrentLeader coordinates with SBUtil.dropConnections via a blackboard.SharedLock to protect verification from network drops. The lock is released in doConcOpsAndVerify (once all VMs had verified against the snapshot in the blackboard.

Overrides:

concVerify in class ParRegTest