SAP HANA Database Index Server crash due to missing LogSegment file.

Issue:

You are using HANA Reversion 49 and below. The indexserver of the SAP HANA database crashes or fail to starts with following message in crash stack dump:

Cannot open file "//logsegment_000_XXXXXXXX.dat", rc=2: No such file or directory (X=digits)

Reason and Prerequisites

You are using HANA Rev 49 and below.

You recently attempted to execute the following statement:

"ALTER SYSTEM RECLAIM LOG"

And during the execution process the statement is abnormally terminated with the following scenario:

•     I/O Failure
•     File System Corruption
•     Disk Full
•     Crash
•     Deadlock

This potentially created an inconsistency between internal metadata and actual log segment file on the disk. In certain case index server will attempt to look for certain non-existing log segments and crash.

This workaround is only intended if there is an inconsistency where log segment file is being marked as "freed" in internal metadata but the actual file got deleted in the file system. For any other logsegment corruption issue this note does not apply to you.

To verify this please login with adm account and run:

hdblogdiag -s and compare if the file listed here got deleted on the file system.

 
Solution

To apply this workaround:

        1. Created a dummy zero byte log segment file using the following command with adm user:

    touch //logsegment_000_XXXXXXXX.dat
    2. Restart the index server. Depends on the setting, you should notice that the dummy zero byte file was inflated to 1,073,741,824 bytes. If index server crash again but with different missing log segment file with increased serial number, repeat step 1 until no furthur crash occurs.

    3. DO NOT RUN ANY UPDATE/INSERT STATEMENT RIGHT NOW. The persistency layer will look for another non existing log segment file if too much change occured and indexserver will crash again.

    4. Run "ALTER SYSTEM RECLAIM LOG" query. HANA will rebuild/realign the internal meta data.

   5. Run the following query and verify all log segment files listed in the result does physically exist in the file system:

    SELECT * FROM M_LOG_SEGMENTS
Please note this issue has been resolved in Rev 50. With the fix a warning message will be printed in the trace file instead of crash if any of the log segments marked as "freed" is missing from the file system.





More Details refer KBA:1788692