SAP HANA system High CPU Utilization Caused by Multiple Threads in sse_icc_lib::mgetSearchi_SSE4impl

Symptom

You experience high CPU utilization. In kernel profiler trace or runteimdump you identify multiple threads in callstack:

sse_icc_lib::mgetSearchi_SSE4impl
sse_icc_lib::mgetSearchi_SSE4
TRexUtils::IndexVectorRef::mgetSearch
...
TRexUtils::JobParallelMgetSearch::run()
TRexUtils::Parallel::JobBase::runEx()
TRexUtils::Parallel::JobBase::run
...

or

sse_icc_lib::mgetSearchi_AVX2impl
sse_icc_lib::mgetSearchi_AVX2
TRexUtils::IndexVectorRef::mgetSearch
...
TRexUtils::JobParallelMgetSearch::run()'
TRexUtils::Parallel::JobBase::runEx()
TRexUtils::Parallel::JobBase::run
...

or

sse_icc_lib::mgetSearchi_AVX512impl
sse_icc_lib::mgetSearchi_AVX512
TRexUtils::IndexVectorRef::mgetSearch
...
TRexUtils::JobParallelMgetSearch::run()'
TRexUtils::Parallel::JobBase::runEx()
TRexUtils::Parallel::JobBase::run

Other Terms

SAP HANA, column store, high CPU, secondary index
Reason and Prerequisites

Reason:

Above callstack points to single attribute scan on a column store table.


Solution

This can be alleviated by creating a secondary index on the table and columns to be searched.

Creating the index will improve query performance significantly (factor 10 and more) and reduce CPU consumption, but will also increase memeory consumption as a concatenated attribute will be generated.



 More details : 2321573

SAP HANA sidadm user login error

Symptom

It is reported that when logging in with an adm, and the loging scripts initializing the users environment fail to run correctly and the Admin user cannot be used.

It is also reported that other adm accounts do not have the same problem, even for the same SID.

The following errors are seen when logging in:

:~ # su - adm
su: warning: cannot change directory to /usr/sap//home: Permission denied
-sh: grep: command not found
-sh: warning: setlocale: LC_MESSAGES: cannot change locale (en_US.UTF-8): Permission denied
-sh: warning: setlocale: LC_CTYPE: cannot change locale (en_US.UTF-8): Permission denied
-sh: warning: setlocale: LC_COLLATE: cannot change locale (en_US.UTF-8): Permission denied
-sh: warning: setlocale: LC_TIME: cannot change locale (en_US.UTF-8): Permission denied
-sh: warning: setlocale: LC_NUMERIC: cannot change locale (en_US.UTF-8): Permission denied
-sh: warning: setlocale: LC_COLLATE: cannot change locale (en_US.UTF-8)
-sh: warning: setlocale: LC_COLLATE: cannot change locale (en_US.UTF-8)
-sh: warning: setlocale: LC_CTYPE: cannot change locale (en_US.UTF-8)
-sh: warning: setlocale: LC_CTYPE: cannot change locale (en_US.UTF-8)
-sh: warning: setlocale: LC_MESSAGES: cannot change locale (en_US.UTF-8)
-sh: warning: setlocale: LC_MESSAGES: cannot change locale (en_US.UTF-8)
-sh: warning: setlocale: LC_NUMERIC: cannot change locale (en_US.UTF-8)
-sh: warning: setlocale: LC_NUMERIC: cannot change locale (en_US.UTF-8)
-sh: warning: setlocale: LC_TIME: cannot change locale (en_US.UTF-8)
-sh: warning: setlocale: LC_TIME: cannot change locale (en_US.UTF-8)
-sh: grep: command not found
-sh: grep: command not found
-sh: grep: command not found
-sh: grep: command not found
-sh: grep: command not found
-sh: grep: command not found
-sh: grep: command not found
-sh: sed: command not found
-sh: sed: command not found
-sh: /usr/sap//home/.profile: Permission denied
adm@:/root>

Environment

HANA DB 2.0
Reproducing the Issue

    Log in with the User Account adm (using ssh or su will give the same result).


Cause

The adm user has the following ACL permissions for the /usr directory:

# file: usr
# owner: root
# group: root
user::rwx
user:adm:r--
group::r-x
mask::r-x
other::r-x

Solution

Grant read/write persmissions to the "SID"adm user








More details :  2780462

SAP HANA startup fails with a crashdump in hdbdaemon

Symptom

You are starting HANA. During the startup, it fails and you find a crashdump in the daemon process with name similar to daemon.crashdump.044766.trc. Upon checking the crashdump file, you observe the [CRASH_STACK] session have the following content:

[CRASH_STACK] Stacktrace of crash: (no-section-time-available)
----> Pending exceptions (possible root cause) <---- br="">exception 1: no.1000000 (MultiDB/impl/MultiDBConfiguration.cpp:392)
 hdbmdcdispatcher has wrong permissions /usr/sap//HDB00/exe/mdc/hdbmdcdispatcher 33128 != 04550; $condition$=(info.st_mode & 07557) == 04550
NO exception throw location recorded. Stack generation system error.
Unknown pending exception
Throw location of the exception unknown

Environment

    SAP HANA Platform 1.0;
    Multitenant Database Containers is used (mode = multidb);
    Database Isolation Level is set to High (database_isolation = high);

Reproducing the Issue

Start HANA: "HDB Start" or "sapcontrol -nr -function StartSystem"
Cause

Prerequisites: You have HANA 1.0 running with MDC mode and database isolation level set to high.

The error message in the CRASH_STACK indicates that there are wrong permissions in the file /usr/sap//HDB00/exe/mdc/hdbmdcdispatcher.

You can check it by running the following command:

ls - ltrah /usr/sap//HDB00/exe/mdc/hdbmdcdispatcher

The default authorization and owner are below:

-r-sr-x--- 1 root sapsys 8.7M May 14 19:38 hdbmdcdispatcher
 

Resolution

For root cause purposes, you have to check with your OS team who/what changed the ownership and authorizations of the file.

To fix this, you need to run the following commands as root:

    "chown root:sapsys /usr/sap//HDB00/exe/mdc/hdbmdcdispatcher";
    "chmod 04550 /usr/sap//HDB00/exe/mdc/hdbmdcdispatcher";

You can now start the SAP system.





















More details 2792608

Hardware requirement for SAP HANA 2.0

You want to use SAP HANA 2.0 and need to know minimal HW requirements

Other Terms

CPU, Haswell, Ivybridge, Westmere, Broadwell, POWER8, POWER7+
Reason and Prerequisites

You start implementing SAP HANA 2.0 or you are upgrading a productive HANA system to SAP HANA 2.0

Solution

Hardware Requirements for SAP HANA 2.0 on Intel-Based Hardware Platforms

Starting with SAP HANA 2.0 computing nodes with at least Intel Haswell CPU or later are strongly recommended. However, SAP's requirement remains Intel Westmere CPU. Specifically, if you are running an upgrade of a productive HANA system from SAP HANA 1.0 to SAP HANA 2.0 you can use Intel Ivy Bridge or Westmere CPU based compute nodes.

Hardware Requirements for SAP HANA 2.0 on IBM Power Servers

Starting with SAP HANA 2.0 computing nodes with at least IBM POWER8 CPU or later are mandatory.

Error 1142 "Invalid OAM Page..." during boot-time recovery of saptempdb database in Sybase

Symptom

    Error 1142 is reported while starting the ASE server:

Error: 1142, Severity: 22, State: 4 Invalid OAM Page 0. Found pstat=0x0, object=0, database=6, indid=0

    An accompanying stacktrace is generated and refers to following functions during 'saptempdb' database recovery:

pg__objdeall_validate_obj()
pg_ptndeal()
closetable()
close_range()
s_estmt_loopend()
s_execute.fdpr.chunk()
sequencer()
execproc()
s_execute()

     As a result, database recovery fails since the database is not brought online.

Environment

SAP Adaptive Server Enterprise (ASE) 15.7 SP122 for Business Suite
Cause

    Access to saptempdb occurs prior to the database being online.
    This issue has been addressed by SAP CR 771382: Its description reads, "error 1142 was raised on a user defined tempdb (saptempdb) during boot time recovery against 15.7 sp122 ASE. Please use system procedure sp_dbrecovery_order to change database recovery order. It will let ASE rebuild user defined tempdb saptempdb first after recovery system databases, then recovery other user databases".

Resolution

Ensure that saptempdb recovers and is brought online, prior to the user database (SID) recovery. This can be done via the sp_dbrecovery_order stored procedure.

sp_dbrecovery_order saptempdb, 1, force, strict
If there is already an recovery order defined for 1, it needed to be removed with the command:  sp_dbrecovery_order saptempdb, -1







More details refer Snote: 2072361

What is TSK files in SAP ?

 What is TSK files in SAP ?


    These .TSK files are pure text files and contain an entry of SAP installtion entry.
TSK file contains with = T, D, P, I, V

T = Table Structure

D = Data

P = Primary Key

I = Index

V= View

and  entry in tsk file C, I, E.

C = Create
I =  Import
E = Export

 Every entry in the tsk file have status  such as

xeq = waiting
err  = Error
ok   = Completed


   When started for the first time, R3load reads a TSK file completely, closes the file, renames it in a TSK.bck (for example, SAPSDIC.TSK.bck) and opens the TSK file for writing. After each executed action an entry is written with the corresponding status.





   If all actions are processed, or if the unprocessed actions could be written back to the TSK file after an error, the TSK.bck file is deleted. If R3load also finds a TSK.bck file in addition to a TSK file during start-up, it terminates immediately with a corresponding error message, as this either means that an R3load process is running and is processing this request, or that a serious error has already occurred that may not be eliminated during the restart, but requires user intervention. When the error has been eliminated you must merge the TSK file and the TSK.bck file prior to or during the restart of the R3load.