Oracle Berkeley DB Java Edition 12c R1 Change Log

Release 6.2.31

Upgrading from JE 6.1 or earlier

In JE 6.2 the on-disk file format moved to 10. The file format change is forward compatible in that JE files created with release 6.1 and earlier can be read when opened with JE 6.2 or later. The change is not backward compatible in that files created with JE 6.2 or later cannot be read by earlier releases. After an existing environment is opened read/write using JE 6.2, the environment can no longer be read by earlier releases.

Upgrading from JE 6.0 or earlier

An API change in JE 6.1.3 [#23330] requires application changes if write operations are performed on a non-replicated database in a replicated environment. A code change is necessary for applications with the following characteristics:

In order to perform write operations in such cases, the application must now call TransactionConfig.setLocalWrite(true) and use this configuration to create a Transaction for performing writes to the non-replicated database.

In addition, it is no longer possible to use a single transaction to write to both replicated and a non-replicated databases. IllegalOperationException will be thrown if this is attempted.

These changes were necessary to prevent corruption when a transaction contains write operations for both replicated and non-replicated databases, and a failover occurs that causes a rollback of this transaction. The probability of corruption is low, but it can occur under the right conditions.

For more information see the javadoc for TransactionConfig.setLocalWrite(true), and the "Non-replicated Databases in a Replicated Environment" section of the ReplicatedEnvironment class javadoc.

Upgrading from JE 5.0 or earlier

In addition to the file format changes, a change was made involving partial Btree and duplicate comparators. Partial comparators are an advanced feature that few applications use. As of JE 6.0, using partial comparators is not recommended. Applications that do use partial comparators must change their comparator classes to implement the new PartialComparator tag interface, before running the application with JE 6. Failure to do so may cause incorrect behavior during transaction aborts. See the PartialComparator javadoc for more information.

Upgrading from JE 4.1 or earlier

There are two important notes about the file format change in JE 5.0.
  1. The file format change enabled significant improvements in operation performance, memory and disk footprint, and concurrency of databases with duplicate keys. Due to these changes, an upgrade utility must be run before opening an environment with this release, if the environment was created using JE 4.1 or earlier. See the Upgrade Procedure below for more information.
  2. An application which uses JE replication may not upgrade directly from JE 4.0 to JE 5.0 or later. Instead, the upgrade must be done from JE 4.0 to JE 4.1 and then to JE 5.0 or later. Applications already at JE 4.1 are not affected. Upgrade guidance can be found in the new chapter, "Upgrading a JE Replication Group", in the "Getting Started with BDB JE High Availability" guide.
Due to the format changes in JE 5, a special utility program must be run for an environment created with JE 4.1 or earlier, prior to opening the environment with JE 5.0 or later. The utility program is part of JE 4.1. JE 4.1.20, or a later version of JE 4.1, must be used.

One of two utility programs must be used, which are available in the release package for JE 4.1.20, or a later release of JE 4.1. If you are currently running a release earlier than JE 4.1.20, then you must download the latest JE 4.1 release package in order to run these utilities.

The steps for upgrading are as follows.

  1. Stop the application using BDB JE.
  2. Run the DbPreUpgrade_4_1 or DbRepPreUpgrade_4_1 utility. If you are using a regular non-replicated Environment: 
        java -jar je-4.1.20.jar DbPreUpgrade_4_1 -h <dir>
    If you are using a JE ReplicatedEnvironment: 
        java -jar je-4.1.20.jar DbRepPreUpgrade_4_1
         -h <dir>
         -groupName <group name>
         -nodeName <node name>
         -nodeHostPort <host:port>
  3. Finally, start the application using the current JE 5.0 (or later) release of BDB JE.

The second step -- running the utility program -- does not perform data conversion. This step simply performs a special checkpoint to prepare the environment for upgrade. It should take no longer than an ordinary startup and shutdown.

During the last step -- when the application opens the JE environment using the current release (JE 5 or later) -- all databases configured for duplicates will automatically be converted before the Environment or ReplicatedEnvironment constructor returns. Note that a database might be explicitly configured for duplicates using DatabaseConfig.setSortedDuplicates(true), or implicitly configured for duplicates by using a DPL MANY_TO_XXX relationship (Relationship.MANY_TO_ONE or Relationship.MANY_TO_MANY).

The duplicate database conversion only rewrites internal nodes in the Btree, not leaf nodes. In a test with a 500 MB cache, conversion of a 10 million record data set (8 byte key and data) took between 1.5 and 6.5 minutes, depending on number of duplicates per key. The high end of this range is when 10 duplicates per key were used; the low end is with 1 million duplicates per key.

To make the duplicate database conversion predictable during deployment, users should measure the conversion time on a non-production system before upgrading a deployed system. When duplicates are converted, the Btree internal nodes are preloaded into the JE cache. A new configuration option, EnvironmentConfig.ENV_DUP_CONVERT_PRELOAD_ALL, can be set to false to optimize this process if the cache is not large enough to hold the internal nodes for all databases. For more information, see the javadoc for this property.

If an application has no databases configured for duplicates, then the last step simply opens the JE environment normally, and no data conversion is performed.

If the user fails to run the DbPreUpgrade_4_1 or DbRepPreUpgrade_4_1 utility program before opening an environment with JE 5 or later for the first time, an exception such as the following will normally be thrown by the Environment or ReplicatedEnvironment constructor: 

  com.sleepycat.je.EnvironmentFailureException: (JE 6.0.1) JE 4.1 duplicate DB
  entries were found in the recovery interval. Before upgrading to JE 5.0, the
  following utility must be run using JE 4.1 (4.1.20 or later):
  DbPreUpgrade_4_1.  See the change log.
  UNEXPECTED_STATE: Unexpected internal state, may have side effects.
    at com.sleepycat.je.EnvironmentFailureException.unexpectedState(EnvironmentFailureException.java:376)
    at com.sleepycat.je.recovery.RecoveryManager.checkLogVersion8UpgradeViolations(RecoveryManager.java:2694)
    at com.sleepycat.je.recovery.RecoveryManager.buildTree(RecoveryManager.java:549)
    at com.sleepycat.je.recovery.RecoveryManager.recover(RecoveryManager.java:198)
    at com.sleepycat.je.dbi.EnvironmentImpl.finishInit(EnvironmentImpl.java:610)
    ...

If the user fails to run the DbPreUpgrade_4_1 or DbRepPreUpgrade_4_1 utility program, but no exception is thrown when the environment is opened with JE 5 or later, this is probably because the application performed an Environment.sync before last closing the environment with JE 4.1 or earlier, and nothing else happened to be written (by the application or JE background threads) after the sync operation. In this case, running the upgrade utility is not necessary.

Changes in 6.2.31

  1. Added EnvironmentConfig.CLEANER_USE_DELETED_DIR. This can be set to true when trying to reproduce and analyze LOG_FILE_NOT_FOUND problems. See the javadoc for details. More information was also added to the EnvironmentConfig.CLEANER_EXPUNGE javadoc on the same topic. [#23830] (6.2.8)

  2. Added debugging information when an internal latch deadlock occurs due to a bug where a latch is not released. Note that latches are not user-visible entities and are unrelated to record locking. Latches are used internally for thread safety and only held for short durations. A latch deadlock is detected via a timeout mechanism. An EnvironmentFailureException is thrown in the thread that times out. Now, additionally a full thread dump is written to the je.info log at logging level SEVERE. The thread dump can be used to find the deadlock.

    In addition, the EnvironmentConfig.ENV_LATCH_TIMEOUT parameter has been exposed to provide control over the timeout interval for atypical applications. This parameter has been present internally since latch timeouts were added in JE 6.0.3; however, the parameter was previously undocumented.

    [#23897] (6.2.9)


  3. Fixed two bugs having to do with lock conflicts. The two problems are distinct, but both occurred while creating a LockConflictException due to a lock timeout. [#23894] (6.2.10)

  4. Fixed a bug that could cause the following exception when calling Cursor.count, skipNext or skipPrev. The bug is likely to occur only when BINs (bottom internal nodes of the Btree) are frequently being evicted. Although the Environment is invalidated by this exception and must be closed, the problem is transient -- the Environment can be re-opened and no data loss or corruption will have occurred. 
    (JE 6.2.6) ... Latch not held: BIN17923 currentThread: ...  currentTime: ...
exclusiveOwner: -none- UNEXPECTED_STATE_FATAL: Unexpected internal state,
unable to continue. Environment is invalid and must be closed.
at com.sleepycat.je.EnvironmentFailureException.unexpectedState(EnvironmentFailureException.java:405)
at com.sleepycat.je.latch.LatchImpl.release(LatchImpl.java:109)
at com.sleepycat.je.tree.IN.releaseLatch(IN.java:519)
at com.sleepycat.je.dbi.CursorImpl.skipInternal(CursorImpl.java:2737)
at com.sleepycat.je.dbi.CursorImpl.skip(CursorImpl.java:2612)
at com.sleepycat.je.Cursor.countHandleDups(Cursor.java:4055)
at com.sleepycat.je.Cursor.countInternal(Cursor.java:4028)
at com.sleepycat.je.Cursor.count(Cursor.java:1804) at
...
    The last line above is a call to Cursor.count. The same problem could happen if Cursor.skipNext or skipPrev is called, and only the last few lines of the stack trace above would be different.

    [#23872] (6.2.25)


  5. The HA Feeder output threads now batch network writes whenever possible to reduce the resource overheads associated with transmitting small network packets. These changes enhance replication performance; improvements in the range of 5% have been observed for write intensive workloads.

    [#23274] (6.2.25)


  6. Added new statistics to count the number of user (non-internal) CRUD operations that are performed entirely on BIN deltas.

    [#23883] (6.2.25)


  7. Fixed a bug where no exception was thrown when using ReplicaAckPolicy.ALL and performing a write transaction in a two node replication group, and the replica node was down/unresponsive. InsufficientAcksException is now thrown in this situation, as specified in the documentation. [#23934] (6.2.26)

  8. Fixed a bug in the internal SortedLSNTreeWalker class, which is used to implement the Database.preload() and Environment.preload() methods. When these methods are called, the bug can lead to the creation of a corrupted BTree, and as a result, subsequent loss of data. The bug was introduced in JE 6.0.

    [#23952] (6.2.27)


  9. Added EntityIndex.getDatabase. [#23971] (6.2.27)

  10. Fixed a bug where an assertion incorrectly fired during CRUD operations. This happened when there was concurrent activity in other threads that changed the number of records in the same portion of the Btree. An example stack trace is below. 
    java.lang.AssertionError
 at com.sleepycat.je.dbi.CursorImpl.getCurrentKey(CursorImpl.java:500)
 at com.sleepycat.je.dbi.CursorImpl.getCurrentKey(CursorImpl.java:483)
 at com.sleepycat.je.Cursor.dupsGetNextOrPrevDup(Cursor.java:2882)
 at com.sleepycat.je.Cursor.retrieveNextHandleDups(Cursor.java:2836)
 at com.sleepycat.je.Cursor.retrieveNext(Cursor.java:2816)
 at com.sleepycat.je.Cursor.getNextDup(Cursor.java:1150)
 [ app specific portion ... ]
    In the stack trace above the Cursor.getNextDup method is being called. There are other operations where the same thing could happen. The common factor is the call to the internal CursorImpl.getCurrentKey method, which fires the assertion.

    [#23971] (6.2.29)


  11. Fixed a bug that prevents recovery, i.e., prevents the Environment from being opened. The bug has always been present in JE but has appeared in tests only recently, and has not been reported in the field. Deleting records in a large range of keys might make the bug more likely to occur. An example of the stack trace when the failure occurs is below: 
    com.sleepycat.je.EnvironmentFailureException: (JE 6.2.29) ... last LSN=0x533/0x41f59 LOG_INTEGRITY: Log information is incorrect, problem is likely persistent. Environment is invalid and must be closed.
        at com.sleepycat.je.recovery.RecoveryManager.traceAndThrowException(RecoveryManager.java:3031)
        at com.sleepycat.je.recovery.RecoveryManager.readINs(RecoveryManager.java:1010)
        at com.sleepycat.je.recovery.RecoveryManager.buildINs(RecoveryManager.java:804)
        at com.sleepycat.je.recovery.RecoveryManager.buildTree(RecoveryManager.java:717)
        at com.sleepycat.je.recovery.RecoveryManager.recover(RecoveryManager.java:352)
        at com.sleepycat.je.dbi.EnvironmentImpl.finishInit(EnvironmentImpl.java:670)
        at com.sleepycat.je.dbi.DbEnvPool.getEnvironment(DbEnvPool.java:208)
        at com.sleepycat.je.Environment.makeEnvironmentImpl(Environment.java:251)
        at com.sleepycat.je.Environment.(Environment.java:232)
        at com.sleepycat.je.Environment.(Environment.java:188)
        at com.sleepycat.je.rep.ReplicatedEnvironment.(ReplicatedEnvironment.java:573)
        at com.sleepycat.je.rep.ReplicatedEnvironment.(ReplicatedEnvironment.java:443)
        [ app specific portion ... ]
Caused by: com.sleepycat.je.EnvironmentFailureException: (JE 6.2.29) ... fetchIN of 0x35c/0x3f7f9 parent IN=11688 IN class=com.sleepycat.je.tree.IN lastFullVersion=0x533/0x5d47d lastLoggedVersion=0x533/0x5d47d parent.getDirty()=false state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed.
        at com.sleepycat.je.tree.IN.fetchINWithNoLatch(IN.java:1866)
        at com.sleepycat.je.tree.IN.fetchINWithNoLatch(IN.java:1764)
        at com.sleepycat.je.tree.Tree.getParentINForChildIN(Tree.java:1346)
        at com.sleepycat.je.recovery.RecoveryManager.recoverChildIN(RecoveryManager.java:2025)
        at com.sleepycat.je.recovery.RecoveryManager.recoverIN(RecoveryManager.java:1834)
        at com.sleepycat.je.recovery.RecoveryManager.replayOneIN(RecoveryManager.java:1099)
        at com.sleepycat.je.recovery.RecoveryManager.readINs(RecoveryManager.java:988)
        ... 16 more
Caused by: java.io.FileNotFoundException: .../0000035c.jdb (No such file or directory)
        at java.io.RandomAccessFile.open(Native Method)
        at java.io.RandomAccessFile.(RandomAccessFile.java:241)
        at java.io.RandomAccessFile.(RandomAccessFile.java:122)
        at com.sleepycat.je.log.FileManager$DefaultRandomAccessFile.(FileManager.java:3260)
        at com.sleepycat.je.log.FileManager$6.createFile(FileManager.java:3288)
        at com.sleepycat.je.log.FileManager.openFileHandle(FileManager.java:1311)
        at com.sleepycat.je.log.FileManager.getFileHandle(FileManager.java:1183)
        at com.sleepycat.je.log.LogManager.getLogSource(LogManager.java:1135)
        at com.sleepycat.je.log.LogManager.getLogEntry(LogManager.java:822)
        at com.sleepycat.je.log.LogManager.getLogEntryAllowInvisibleAtRecovery(LogManager.java:787)
        at com.sleepycat.je.tree.IN.fetchINWithNoLatch(IN.java:1801)
        ... 22 more
    [#23990] (6.2.31)

  12. Fixed a bug that can cause data log corruption. This has been reported only as a rare occurrence, but could impact any application where not all Btree internal nodes fit in cache. An example stack trace is below, although other stack traces could also apply where an IN (internal node) is being fetched. 
    com.sleepycat.je.EnvironmentFailureException: (JE 6.2.9) ...
fetchIN of 0x10cbc/0x696373 parent IN=84363 IN
class=com.sleepycat.je.tree.IN lastFullVersion=0x10e00/0x82006e
lastLoggedVersion=0x10e00/0x82006e parent.getDirty()=false state=0
LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is
invalid and must be closed.
        at com.sleepycat.je.tree.IN.fetchINWithNoLatch(IN.java:1866)
        at com.sleepycat.je.tree.IN.fetchINWithNoLatch(IN.java:1752)
        at com.sleepycat.je.tree.Tree.search(Tree.java:2293)
        at com.sleepycat.je.tree.Tree.search(Tree.java:2193)
        at com.sleepycat.je.tree.Tree.getParentBINForChildLN(Tree.java:1481)
        at com.sleepycat.je.cleaner.FileProcessor.processLN(FileProcessor.java:836)
        ... 5 more
Caused by: java.io.FileNotFoundException: /local/pyrox/DS2/asinst_1/OUD/db/Europe/00010cbc.jdb (No such file or directory)
        at java.io.RandomAccessFile.open(Native Method)
        at java.io.RandomAccessFile.(RandomAccessFile.java:241)
        at java.io.RandomAccessFile.(RandomAccessFile.java:122)
        at com.sleepycat.je.log.FileManager$DefaultRandomAccessFile.(FileManager.java:3208)
        at com.sleepycat.je.log.FileManager$6.createFile(FileManager.java:3236)
        at com.sleepycat.je.log.FileManager.openFileHandle(FileManager.java:1305)
        at com.sleepycat.je.log.FileManager.getFileHandle(FileManager.java:1177)
        at com.sleepycat.je.log.LogManager.getLogSource(LogManager.java:1151)
        at com.sleepycat.je.log.LogManager.getLogEntry(LogManager.java:843)
        at com.sleepycat.je.log.LogManager.getLogEntryAllowInvisibleAtRecovery(LogManager.java:808)
        at com.sleepycat.je.tree.IN.fetchINWithNoLatch(IN.java:1801)
        ... 10 more
    [#24046] (6.2.31)

  13. Fixed a bug that can cause data log corruption when using a deferred-write database. In the one reported instance of the problem, missing records were reported. A corruption (e.g., LOG_FILE_NOT_FOUND) is also posssible. [#24066] (6.2.31)

Changes in 6.2.7

  1. A cursor may now be optionally configured to be "non-sticky". This has certain performance advantages: For more information, see the javadoc for CursorConfig.setNonSticky.

    [#23775] (6.2.0)


  2. Further exploitation of BIN-deltas for CRUD operations.

    For backgroud and previous work in this area, see the changelog for the 6.1 release. In this release we have extended the set of CRUD operations that are performed in BIN-deltas, without the need to mutate them to full BINs (and thus saving the disk reads that would be required to fetch the full BINs in memory). Specifically, the following additional operations can now exploit BIN-deltas:

    Insertions and updates, when no tree node splits are required and the key of the record to be inserted/updated is found in a BIN-delta.

    Blind operations: we say that a record operation (insertion, update, or deletion) is performed "blindly" in a BIN-delta, when the delta does not contain a slot with the operation's key and we don't need to access the full BIN to check whether such a slot exists there or to extract any information from the full-BIN slot, if it exists. The condition that no tree node splits are required applies to blind operations as well. The following operations can be performed blindly: - Replay of insertions at replica nodes. - Insertions during recovery redo. - Updates and deletes during recovery redo, for databases with duplicates.

    A new statistic has been added to count the number blind operations performed, including the blind put operations described below. This count can be obtained via the EnvironmentStats.getNBINDeltaBlindOps() method.

    [#23680] (6.2.0)


  3. Blind put operations in BIN-deltas.

    Normally, blind puts are not possible: we need to know whether the put is actually an update or an insertion, i.e., whether the key exists in the full BIN or not. Furthermore, in case of update we also need to know the location of the previous record version to make the current update abortable. However, it is possible to answer at least the key existence question by adding a small amount of extra information in the deltas. If we do so, puts that are actual insertions can be done blindly.

    To answer whether a key exists in a full BIN or not, each BIN-delta stores a bloom filter, which is a very compact, approximate representation of the set of keys in the full BIN. Bloom filters can answer set membership questions with no false negatives and very low probability of false positives. As a result, put operations that are actual insertions can almost always be performed blindly.

    To make possible the blind puts optimization in JE databases that use custom BTree and/or duplicates comparators, these comparators must perform "binary equality", that is, they must consider two keys (byte arrays) to be equal if and only if they have the same length and they are equal byte-per-byte. To communicate to the JE engine that a comparator does binary equality, the comparator must implement the new BinaryEqualityComparator tag interface.

    [#23768] (6.2.1)


  4. Added LockMode.READ_UNCOMMITTED_ALL. When using this mode, unlike READ_UNCOMMITTED, deleted records will not be skipped by read operations when the deleting transaction is still open (and may later abort, in which case the record will no longer be deleted). See the LockMode javadoc for further details.

    [#23660] (6.2.1)


  5. Added two optimizations for secondary DB read operations.

    [#23326] (6.2.2)


  6. Fixed a bug that could cause the following Collections API and DPL methods to incorrectly return an empty result (no records). The problem occurs when searching for the last record in a key range, while another thread is concurrently inserting records near the end of the key range. An empty result is returned, regardless of the number of records in the key range.

    [#23687] (6.2.2)


  7. Fixed bugs in the computation of the nINCompactKey and nINNoTarget stats (EnvironmentStats.getNINCompactKeyIN and getNINNoTarget). Prior to the fixes, these stats would sometimes have negative values. [#23718] (6.2.3)

  8. Fixed a bug that caused a DB to become unusable when it is removed or truncated (by calling Environment.removeDatabase or truncateDatabase) using a read-committed transaction, and the transaction aborts (explicitly, or due to a crash before commit). In this case the DB will not be accessible -- it cannot be opened, truncated or removed. When attempting to open the DB, an exception such as the following is thrown: 
    Exception in thread "main" com.sleepycat.je.DatabaseNotFoundException: (JE 6.1.5) Attempted to remove non-existent database ...
at com.sleepycat.je.dbi.DbTree.lockNameLN(DbTree.java:869)
at com.sleepycat.je.dbi.DbTree.doRemoveDb(DbTree.java:1130)
at com.sleepycat.je.dbi.DbTree.dbRemove(DbTree.java:1183)
at com.sleepycat.je.Environment$1.runWork(Environment.java:947)
at com.sleepycat.je.Environment$DbNameOperation.runOnce(Environment.java:1172)
at com.sleepycat.je.Environment$DbNameOperation.run(Environment.java:1155)
at com.sleepycat.je.Environment.removeDatabase(Environment.java:941)
...
    A workaround for the problem in earlier releases is to avoid using read-committed for a transaction used to perform a DB remove or truncate operation.

    [#23821] (6.2.3)


  9. Fixed a bug that caused an exception during log cleaning, although it has been observed very rarely. It could also potentially cause data corruption, but this has never been reported or observed in tests. Examples of the exceptions that have been observed are below. 
    com.sleepycat.je.EnvironmentFailureException: Environment invalid because of
previous exception: (JE 6.1.0) ...
    at com.sleepycat.je.EnvironmentFailureException.unexpectedException(EnvironmentFailureException.java:315)
    at com.sleepycat.je.log.LogManager.serialLog(LogManager.java:477)
    at com.sleepycat.je.log.LogManager.logItems(LogManager.java:419)
    at com.sleepycat.je.log.LogManager.multiLog(LogManager.java:324)
    at com.sleepycat.je.log.LogManager.log(LogManager.java:272)
    at com.sleepycat.je.log.LogManager.log(LogManager.java:261)
    at com.sleepycat.je.log.LogManager.log(LogManager.java:223)
    at com.sleepycat.je.dbi.EnvironmentImpl.rewriteMapTreeRoot(EnvironmentImpl.java:1285)
    at com.sleepycat.je.cleaner.FileProcessor.processFile(FileProcessor.java:701)
    at com.sleepycat.je.cleaner.FileProcessor.doClean(FileProcessor.java:274)
    at com.sleepycat.je.cleaner.FileProcessor.onWakeup(FileProcessor.java:137)
    at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:148)
    at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 111
    at com.sleepycat.util.PackedInteger.writeInt(PackedInteger.java:188)
    at com.sleepycat.je.log.LogUtils.writePackedInt(LogUtils.java:155)
    at com.sleepycat.je.cleaner.DbFileSummary.writeToLog(DbFileSummary.java:79)
    at com.sleepycat.je.dbi.DatabaseImpl.writeToLog(DatabaseImpl.java:2410)
    at com.sleepycat.je.dbi.DbTree.writeToLog(DbTree.java:2050)
    at com.sleepycat.je.log.entry.SingleItemEntry.writeEntry(SingleItemEntry.java:114)
    at com.sleepycat.je.log.LogManager.marshallIntoBuffer(LogManager.java:745)
    at com.sleepycat.je.log.LogManager.serialLogWork(LogManager.java:611)
    at com.sleepycat.je.log.LogManager.serialLog(LogManager.java:461)
    ... 11 more
    Another instance of the same problem with a slightly different stack trace is below: 
    java.nio.BufferOverflowException UNEXPECTED_EXCEPTION_FATAL: Unexpected
internal Exception, unable to continue. Environment is invalid and must be
closed.
    at com.sleepycat.je.EnvironmentFailureException.unexpectedException(EnvironmentFailureException.java:315)
    at com.sleepycat.je.log.LogManager.serialLog(LogManager.java:481)
    at com.sleepycat.je.log.LogManager.logItems(LogManager.java:423)
    at com.sleepycat.je.log.LogManager.multiLog(LogManager.java:325)
    at com.sleepycat.je.log.LogManager.log(LogManager.java:273)
    at com.sleepycat.je.tree.LN.logInternal(LN.java:600)
    at com.sleepycat.je.tree.LN.log(LN.java:411)
    at com.sleepycat.je.cleaner.FileProcessor.processFoundLN(FileProcessor.java:1070)
    at com.sleepycat.je.cleaner.FileProcessor.processLN(FileProcessor.java:884)
    at com.sleepycat.je.cleaner.FileProcessor.processFile(FileProcessor.java:673)
    at com.sleepycat.je.cleaner.FileProcessor.doClean(FileProcessor.java:278)
    at com.sleepycat.je.cleaner.FileProcessor.onWakeup(FileProcessor.java:137)
    at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:148)
Caused by: java.nio.BufferOverflowException
    at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:189)
    at java.nio.ByteBuffer.put(ByteBuffer.java:859)
    at com.sleepycat.je.log.LogUtils.writeBytesNoLength(LogUtils.java:350)
    at com.sleepycat.je.log.entry.LNLogEntry.writeBaseLNEntry(LNLogEntry.java:371)
    at com.sleepycat.je.log.entry.LNLogEntry.writeEntry(LNLogEntry.java:333)
    at com.sleepycat.je.log.entry.BaseReplicableEntry.writeEntry(BaseReplicableEntry.java:48)
    at com.sleepycat.je.log.entry.LNLogEntry.writeEntry(LNLogEntry.java:52)
    at com.sleepycat.je.log.LogManager.marshallIntoBuffer(LogManager.java:751)
    at com.sleepycat.je.log.LogManager.serialLogWork(LogManager.java:617)
    at com.sleepycat.je.log.LogManager.serialLog(LogManager.java:465)

    [#23492] (6.2.3)


  10. Fixed a locking bug that causes a deadlock when no real deadlock exists. The bug shows up with cursors using read-committed isolation.

    Here is the specific scenario:

    1. Cursor C1 in thread T1 reads a record R using Transaction X1. C1 creates a ReadCommittedLocker L1, with X1 as its buddy. L1 locks R.
    2. Cursor C2 in thread T2 tries to write-lock R, using another Transaction X2. X2 waits for L1 (T2 waits for T1).
    3. Cursor C3 in thread T1 tries to read R using X1. C3 creates a ReadCommittedLocker L3, with X1 as its buddy. L3 tries to lock R. L1 and L3 are not recognized as buddies, so L3 waits for X2 (T1 waits for T2)

    [#23821] (6.2.4)


  11. The ant build (build.xml) has been updated so that the JUnit jar file is now downloaded from Maven Central when needed for running tests with the 'test' target. This jar is no longer needed for building a JE jar file with the 'jar' target. See installation.html for an updated description of how to build JE and run the unit tests. [#23669] (6.2.7)

Changes in 6.1.5

  1. Made an improvement to eviction for Oracle NoSQL DB users, and several improvements to the DbCacheSize utility.

    For Oracle NoSQL DB users only, record versions are now discarded using a separate eviction step. This means that the record versions can be discarded to free cache memory without discarding the entire BIN (bottom internal node). In general, this makes better use of memory and reduces IO for some workloads.

    The improvements to DbCacheSize are as follows.

    [#23550] (6.1.0)


  2. Fixes a bug which prevented serialization of ReplicaWriteException. Previously an attempt to serialize this exception could fail with the following characteristic stack trace when the StateChangeEvent object was encountered during serialization: 
     Caused by: java.io.NotSerializableException: com.sleepycat.je.rep.StateChangeEvent

    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1181)
    at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1541)
    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1506)
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1429)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1175)
    at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1541)
    at java.io.ObjectOutputStream.defaultWriteObject(ObjectOutputStream.java:439)
    at java.util.logging.LogRecord.writeObject(LogRecord.java:470)
    ...
    [#23578] (6.1.1)

  3. The JE HA replica replay mechanism now uses a separate thread to write replica acknowledgements and heartbeat responses to the network. This change results in two improvements:
    1. The replay of changes sent by the master can make progress even in the presence of brief network stalls, thus increasing replica replay throughput; improvements in the range of 5 to 10% have been observed in internal test scenarios.
    2. This new thread is also used to send spontaneous heartbeat response messages, making the heartbeat mechanism, used to detect node failures, more robust.
    [#23195] (6.1.1)

  4. Performance enhancement: executing a subset of CRUD and internal operations on memory-resident BIN-deltas.

    Before JE 6.0, BIN-deltas were used as a disk optimization only: to reduce the amount of bytes written to disk every time a new BIN version had to to be logged. BIN-deltas would never appear in the in-memory BTrees, and if the most recently logged version of a BIN was a delta, fetching that BIN into the in-memory tree required 2 disk reads: one for the delta and one for the most recent full-BIN version.

    Starting with JE 6.0, BIN-deltas can appear in the in-memory BTree. Specifically, if a full dirty BIN is selected for eviction, rather than evicting the whole BIN (and incurring a disk write), the BIN is converted to a delta that stays in the cache. If a subsequent operation needs the full BIN and the delta is still in the cache, only one disk read will be done.

    Further disk-read savings can be realized, because many operations can (under certain conditions) be performed directly on the BIN-delta, without the need for the full BIN. However, in 6.0, only a small subset of background operations were modified to exploit BIN-deltas. In JE 6.1, the set of operations that can be performed on BIN-deltas has been extended. Examples of such operations include key searches in BTrees, if the search key is found on a BIN delta and deletion or update of the record a cursor is located on, if the cursor is located on a BIN-delta. These changes affect both internal operations as well as the search, delete, and putCurrent methods of the Database and Cursor API classes.

    [#23428] (6.1.1)


  5. Performance enhancement: Reduced latch contention during BTree searches.

    Typically, thread synchronization during BTree searches is done via latch coupling: at most 2 tree nodes (a parent and a child) are latched at a time. Furthermore, a node is latched in shared (SH) mode, unless it is expected that it will be updated, in which case it is latched in exclusive (EX) mode. Finally, SH latches are not upgradeable to EX latches (to avoid deadlocks and reduce latching overhead).

    JE follows this general latch-coupling technique. However, it also has to deal with the JE-specific fact that fetching a missing child node into the cache requires that its memory-resident parent be updated (because the parent points to its children via direct Java object references). As a result, during a JE BTree search every node is potentially updated, which precludes the use of SH latches. To cope with this complication, JE has been using one of the following approaches during its various kinds of BTree searches: (a) use SH latches, but if a missing child needs to be fetched, release the SH latch on the parent and restart the search from the beginning, using EX latches on all nodes this time, (b) do grandparent latching: use SH latches but keep a latch on the grandparent node so that if we need to fetch a missing child of the parent node, the SH latch on the parent can be released, and then the parent can be relatched in EX mode, (c) do latch-coupling with EX latches only. Obviously, (c) is the worst choice, but all of the 3 approaches result in more and longer-held EX latches than necessary. As a result, some JE applications have experienced performance problems due to excessive latch contention during BTree searches.

    In JE 6.1, a new latching algorithm has been implemented to replace all of (a), (b), and (c) above. The new algorithm uses SH latches, but if a missing child needs to be fetched, it first "pins" the parent (to prevent its eviction), then releases the SH latch on the parent, and finally reads the child node from the log (without any latches held). After the child is fetched, it latches the remembered parent in EX mode, unpins it, and checks whether it is still the correct parent for the search and for the child node that was fetched. If so, the search continues down the tree. If not, it restarts the search from the beginning. Compared to approach (a) above, this new algorithm may restart a search multiple times, however the probability of even a single restart is less than (a), and each restart uses SH latches. Furthermore, no latches are held during the long random disk read done to fetch a missing child.

    [#18617] (6.1.1)


  6. Fixed a bug that could result in the following exception in a JE HA application: 
    com.sleepycat.je.EnvironmentFailureException: 
 Node5(5):... VLSN 3,182,883 should be held within this tracker.
    or 
    com.sleepycat.je.EnvironmentFailureException: 
 Node5(5):...end of last bucket should match end of range ...
    [#23491]

  7. Improved the Monitor's ability to discover group status changes, which should improve the robustness of notifications after the monitor is down or when it has lost network connectivity.

    [#23631] (6.1.2)


  8. A new implementation for Database.count() and a new variant of Database.count() that takes a memoryLimit as input.

    Counting the number of records in a database is now implemented using a disk-ordered-scan (DOS), similar to the one used by DiskOrderedCursor. DOS may consume a large amount of memory, and to avoid OutOfMemoryErrors, it requires that a limit on its memory consumption is provided. As a result, a new method, Database.count(long memoryLimit), has been implemented that takes this memory limit as a parameter. The existing Database.count() method is still available and uses an internally established limit.

    This change fixes two problems of the previous implementation (based on the SortedLSNTreeWalker class): 1. There was no upper bound on the memory consumption of the previous implementation and 2. It was buggy in the case where concurrent thread activity could cause full BINs to be mutated to deltas or vice versa.

    [#23646] (6.1.2)


  9. Fixed bug in DiskOrderedCursor.

    Iterating over the records of a database via a DiskOrderedCursor would cause a crash if a BIN delta was encountered in the in-memory BTree (because in this case a copy of the BIN delta was created and cached for later use, but the copy did not contain all the needed information from the original). This bug was introduced in JE 6.0.11.

    [#23646] (6.1.2)


  10. Fixed a bug in DiskOrderedCursor for DeferredWrite databases. An example of the stack trace when the bug occurs is below. Note that although the exception message indicates that a file is missing, actually the problem was transient and no file was missing. Upgrading to the current JE release will fix the problem without requiring data conversion or restoring from a backup. 
    com.sleepycat.je.EnvironmentFailureException:
(JE 5.0.97) Environment must be closed, caused by:
com.sleepycat.je.EnvironmentFailureException:
Environment invalid because of previous exception:
(JE 5.0.97) ... java.io.FileNotFoundException: ...\ffffffff.jdb
(The system cannot find the file specified) LOG_FILE_NOT_FOUND:
Log file missing, log is likely invalid.
Environment is invalid and must be closed.
    at com.sleepycat.je.EnvironmentFailureException.wrapSelf(EnvironmentFailureException.java:210)
    at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1594)
    at com.sleepycat.je.dbi.DiskOrderedCursorImpl.checkEnv(DiskOrderedCursorImpl.java:234)
    at com.sleepycat.je.DiskOrderedCursor.checkState(DiskOrderedCursor.java:367)
    at com.sleepycat.je.DiskOrderedCursor.getNext(DiskOrderedCursor.java:324)
    ...
    [#23676] (6.1.3)

  11. An API change requires application changes if write operations are performed on a non-replicated database in a replicated environment. A code change is necessary for applications with the following characteristics:

    In order to perform write operations in such cases, the application must now call TransactionConfig.setLocalWrite(true).

    In addition, it is no longer possible to use a single transaction to write to both a replicated and a non-replicated databases. IllegalOperationException will be thrown if this is attempted.

    These changes were necessary to prevent corruption when a transaction contains write operations for both replicated and non-replicated databases, and a failover occurs that causes a rollback of this transaction. The probability of corruption is low, but it can occur under the right conditions.

    For more information see the javadoc for TransactionConfig.setLocalWrite(true), and the "Non-replicated Databases in a Replicated Environment" section of the ReplicatedEnvironment class javadoc.

    [#23330] (6.1.3)


  12. Read-only transactions are now supported. A read-only transaction prohibits write operations, and more importantly in a replicated environment it automatically uses Durability.ReplicaAckPolicy.NONE. A read-only transaction on a Master will thus not be held up, or throw InsufficientReplicasException, if the Master is not in contact with a sufficient number of Replicas at the time the transaction is initiated. To configure a read-only transaction, call TransactionConfig.setReadOnly(true). See this method's javadoc for more information.

    Durability.READ_ONLY_TXN has been deprecated and TransactionConfig.setReadOnly should be used instead.

    [#23330] (6.1.3)


  13. Fixed a bug that could cause a NullPointerException, such as the one below, when a ReplicatedEnvironment is opened on an HA replica node. This prevents the environment from being opened.

    The conditions that cause the bug are:

    1. a replica has been restarted after an abnormal shutdown (ReplicatedEnvironment.close was not called),
    2. a transaction writing records in multiple databases was in progress at the time of the abnormal shutdown,
    3. one of the databases, but not all of them, is then removed or truncated, and finally
    4. another abnormal shutdown occurs.

    If this bug is encountered, it can be corrected by upgrading to the JE release containing this fix, and no data loss will occur.

    This bug is similar to another bug that was fixed in JE 5.0.70 [#22052]. This bug differs in that the transaction must write records in multiple databases, and at least one but not all of the databases must be removed or truncated between the two abnormal shutdowns. 

    com.sleepycat.je.EnvironmentFailureException: (JE 6.1.3) Node1(-1):...
last LSN=0x3/0x4427 LOG_INTEGRITY: Log information is incorrect, problem is
likely persistent. Environment is invalid and must be closed.
    at com.sleepycat.je.recovery.RecoveryManager.traceAndThrowException(RecoveryManager.java:3012)
    at com.sleepycat.je.recovery.RecoveryManager.undoLNs(RecoveryManager.java:1253)
    at com.sleepycat.je.recovery.RecoveryManager.buildTree(RecoveryManager.java:741)
    at com.sleepycat.je.recovery.RecoveryManager.recover(RecoveryManager.java:352)
    at com.sleepycat.je.dbi.EnvironmentImpl.finishInit(EnvironmentImpl.java:654)
    at com.sleepycat.je.dbi.DbEnvPool.getEnvironment(DbEnvPool.java:208)
    at com.sleepycat.je.Environment.makeEnvironmentImpl(Environment.java:252)
    at com.sleepycat.je.Environment.(Environment.java:232)
    at com.sleepycat.je.Environment.(Environment.java:188)
    at com.sleepycat.je.rep.ReplicatedEnvironment.(ReplicatedEnvironment.java:573)
    at com.sleepycat.je.rep.ReplicatedEnvironment.(ReplicatedEnvironment.java:443)
    ... [app creates a new ReplicatedEnvironment here] ...
Caused by: java.lang.NullPointerException
    at com.sleepycat.je.log.entry.LNLogEntry.postFetchInit(LNLogEntry.java:412)
    at com.sleepycat.je.txn.TxnChain.(TxnChain.java:133)
    at com.sleepycat.je.txn.TxnChain.(TxnChain.java:84)
    at com.sleepycat.je.recovery.RollbackTracker$RollbackPeriod.getChain(RollbackTracker.java:1009)
    at com.sleepycat.je.recovery.RollbackTracker$Scanner.rollback(RollbackTracker.java:483)
    at com.sleepycat.je.recovery.RecoveryManager.undoLNs(RecoveryManager.java:1182)
    ... 11 more
    [#22071] (6.1.3)

  14. Fixed a bug where a transaction configured for no-wait (using TransactionConfig.setNoWait(true)) behaved as a normal (wait) transction when the ReadCommitted isolation mode was also used. Due to this bug, a LockTimeoutException was thrown when a LockNotAvailableException should have been thrown instead, and the transaction was invalidated when it should not have been. [#23653] (6.1.4)

  15. Fixed eviction bug for shared-cache environments. The bug caused LRU corruption and potential memory leaks in certain cases. The bug was introduced in JE 6.0. Note that the bug has no impact for environments that are not using a shared cache (EnvironmentConfig.setSharedCache(true)). [#23696] (6.1.4)

Changes in 6.0.11

  1. Added support in JE HA for the new SECONDARY node type. SECONDARY nodes can only be replicas, not masters, and do not participate in either elections or durability decisions. SECONDARY nodes can be used to increase the available number of read replicas without changing the election or durability quorum of the group, and without requiring communication with the secondaries during master elections or transaction commits.

    Changes include adding the NodeType.SECONDARY enumeration constant, and the ReplicationGroup.getSecondaryNodes and ReplicationGroup.getDataNodes methods. [#22482] (6.0.1)


  2. Made improvements to internal latching to allow interrupting threads that are waiting on latches, cause a timeouts when a latch deadlock occurs, and enable latch instrumentation via system properties. Note that latching is not related to transactional locking and latches are intended to be held for very short periods. [#22993] (6.0.3)

  3. The following log cleaner configuration parameters in the EnvironmentConfig class have been deprecated and are no longer used. If configured, they will be silently ignored. Lazy and proactive migration are no longer supported due to negative impacts on eviction, checkpointing and Btree splits. If a persistent log cleaner backlog occurs, the recommended solution is to configure additional cleaner threads. [#23070] (6.0.3)

  4. When using secondary databases and DPL secondary indexes, the locking order for reads via a secondary has been changed to reduce the possibility of deadlocks. This optimization does not apply when the serializable isolation mode is used, and does not apply to the JoinCursor. [#22368] (6.0.4)

  5. Improved Btree cache usage by caching a BIN-delta -- the partial form of a BIN containing only the dirty entries -- in preference to logging it and then evicting the entire BIN. This reduces disk reads if CRUD operations are performed on the BIN before the entire BIN is evicted, because only one BIN fetch rather than two is needed. Disk writes are also reduced to some degree. The performance improvement applies only when BINs are being evicted from cache. The improvement is signficant when CRUD operations address a non-random subset of the keys in the data set.

    As part of the performance improvement work, the following statistics were added.

    In addition, the EnvironmentConfig.TREE_MAX_DELTA param has been deprecated. As of JE 5.0, the benefit from logging BIN-deltas is unrelated to the number of deltas that have been logged since the last full BIN. To configure BIN-delta logging, use EnvironmentConfig.TREE_BIN_DELTA.

    [#22662] (6.0.5)


  6. An optimization for Databases with sorted duplicates configured has been made to improve log cleaning performance. Records in duplicates databases need no longer be tracked or processed by the log cleaner, which reduces cleaning costs significantly when duplicates databases are used for a significant portion of a data set, for example, as secondary index databases.

    As described under 'Upgrading from JE 5.0 or earlier' at the top of this document, to support this cleaner optimization a change was made involving partial Btree and duplicate comparators. Partial comparators are an advanced feature that few applications use. As of JE 6.0, using partial comparators is not recommended. Applications that do use partial comparators must now change their comparator classes to implement the new PartialComparator tag interface, before running the application with JE 6. Failure to do so may cause incorrect behavior during transaction aborts. See the PartialComparator javadoc for more information.

    [#22864] (6.0.5)


  7. Fixed a bug that sometimes resulted in an uncommitted record deletion performed in one transaction to be visible (result in a NOTFOUND result) to an operation performed in another transaction. This bug applies to the use of Database.delete and PrimaryIndex.delete. It does not apply to the use of SecondaryDatabase.delete, SecondaryIndex.delete, or the use of a cursor to perform a deletion. Note that this problem is distinct from a similar bug that was fixed in JE 5.0.98 ([#22892]).

    [#23132] (6.0.5)


  8. Modified the algorithm that protects cleaned log files from deletion to consider the relative cost of replication replay versus network restore, as well as available disk space. When JE HA decides whether to delete cleaned log files, it uses information it stores about the progress of replication replay for each electable replica to retain useful log files even if the replicas are offline, subject to the ReplicationConfig.REP_STREAM_TIMEOUT parameter. The system does not store information about replication progress for secondary replicas, though, so a different approach has been added.

    The modified algorithm estimates the costs of replication replay and network restore, and protects log files from deletion that could be used for replay if there is sufficient disk space and replay would be less expensive than network restore. These computations apply to all replicas, but are particularly useful for secondary replicas, for which log files will not otherwise be retained if the replicas become temporarily unreachable. Note that disk space calculations are only performed when running with Java 7 or later.

    Two new ReplicationConfig parameters were added:

    [#22575] (6.0.5)


  9. An improvement was made to the calculation of log utilization to avoid under-cleaning or over-cleaning. For example, when log utilization was estimated to be lower than actual utilization, unnecessary over-cleaning would occur, which could reduce performance. Or when log utilization was estimated to be higher than actual utilization, under-cleaning would prevent reclaiming unused disk space.

    To prevent these problems, the size of each logged record is now stored in the Btree BINs (bottom internal nodes), so that utilization can be calculated correctly during record updates and deletions, while still avoiding a fetch of the old version of the record. With this change, the utilization adjustment facility in the log cleaner, which attempted to compensate for this problem by estimating utilization, is no longer needed by most applications.

    Therefore the EnvironmentConfig.CLEANER_ADJUST_UTILIZATION parameter is now false by default rather than true, and will be disabled completely in a future version of JE. For more information, see the javadoc for this parameter.

    [#22275] (6.0.7)


  10. The helper hosts parameter used in JE HA replication is now mutable. Accordingly, the set/getHelperHosts() methods and the HELPER_HOST definition in com.sleepycat.je.rep.ReplicationConfig have been moved to their parent class, ReplicationMutableConfig. The change is fully link and source compatible. [#22753] (6.0.7)

  11. Improved the performance of eviction by removing a bottleneck that was causing thread contention. Previously, for workloads with heavy eviction, threads were often waiting on a mutex in the TargetSelector.selectIN method. This impacted not only JE's dedicated background threads, but also application threads that were participating in critical eviction. A new approach is used that dramatically reduces thread contention and increases performance (compared to JE 5 and earlier) for such workloads.

    In addition, the new eviction approach implements a more accurate LRU which ensures that dirty nodes are evicted last and thereby reduces unnecessary logging.

    As part of this change, the following configuration parameters were deprecated and are ignored by JE: 

        EnvironmentConfig.EVICTOR_NODES_PER_SCAN
    EnvironmentConfig.EVICTOR_LRU_ONLY
    And the following configuration parameter was added: 
        EnvironmentConfig.EVICTOR_N_LRU_LISTS
    [#23063] (6.0.7)

  12. A change was made involving the charset for internal text (messages) that appear in the JE log (.jdb files). Previously, the default JVM charset was used. When dumping the log with DbPrintLog (e.g., for debugging purposes), if the default JVM charset was different than the one at the time the log was written, the text messages would be garbled. For example, this occurred when the log was written with an EBCDIC charset and then dumped with a UTF8 charset. This has been fixed by always writing and reading text in the UTF8 charset. [#15296] (6.0.8)

  13. A new HA configuration parameter: com.sleepycat.je.rep.ReplicationConfig.BIND_INADDR_ANY was added. This parameter permits binding of the port used by HA to all the local interfaces on the host. The javadoc associated with this configuration parameter provides further details. [#23437] (6.0.9)

  14. Fix a bug that could under rare conditions, primarily frequent failovers, cause the following exception in an HA environment. 
    Caused by: com.sleepycat.je.EnvironmentFailureException: (JE 5.0.97)
node2(2):foo\node2 Read invisible log entry at 0x0/0xcb776
hdr type="INS_LN_TX/8" vlsn v="19,373" isReplicated="1" isInvisible="1"
prev="0xcb74c" size="17" cksum="2626620732"
LOG_INTEGRITY: Log information is incorrect, problem is likely persistent.
fetchTarget of 0x0/0xcb776 parent IN=29 IN class=com.sleepycat.je.tree.BIN
lastFullVersion=0x0/0xf154c lastLoggedVersion=0x0/0xf588e
parent.getDirty()=true state=3
at com.sleepycat.je.log.LogManager.getLogEntryFromLogSource(LogManager.java:1054)
at com.sleepycat.je.log.LogManager.getLogEntry(LogManager.java:906)
at com.sleepycat.je.log.LogManager.getLogEntryAllowInvisibleAtRecovery(LogManager.java:867)
at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1427)
at com.sleepycat.je.tree.BIN.fetchTarget(BIN.java:1250)
at com.sleepycat.je.recovery.RecoveryManager.undo(RecoveryManager.java:2415)
at com.sleepycat.je.recovery.RecoveryManager.rollbackUndo(RecoveryManager.java:2268)
...
    [#22848] (6.0.10)

  15. EntityStore.close has been changed to fix a bug that caused a memory leak when the Database could not be closed, for example, if it had open cursors. The javadoc for this method was also updated to warn that it must be called to avoid memory leaks, even when the Environment is invalid. [#23462] (6.0.10)