The Application Server Java stopped - No appropriate logs can be found for investigation

 

The Application Server Java stopped - No appropriate logs can be found for investigation

Symptom:

AS Java terminates and no relevant log enries can be found in std_server log file.

AS Java terminates with exit code:

In dev_server files it can be seen that the system is terminated with for example exit code 1:

F [Thr 3342] Thu Jan 26 08:08:04 2017

F  [Thr 3342] *** LOG => SfCJavaVm: exit hook is called. (rc = 1)

F  ********************************************************************************

F  *** ERROR => Java node 'server1<x> terminated with exit code 1.

F  ***

F  *** Please see section 'Java program exit codes'

F  *** in SAP Note 1316652 for additional information and trouble shooting advice.

F  ********************************************************************************

F  [Thr 3342] *** LOG => exiting (exitcode 1, retcode 1).

M  [Thr 3342] CCMS: CCMS Monitoring Cleanup finished successfully.

Or for example with exit code 11112:

F [Thr 140265776588544] Tue Jan 10 09:55:42 2017

F  [Thr 140265776588544] *** LOG => SfCJavaVm: exit hook is called. (rc = -11112)

F  ********************************************************************************

F  *** ERROR => Java node 'server<x>' terminated with exit code -11112.

F  ***

F  *** Please see section 'Java program exit codes'

F  *** in SAP Note 1316652 for additional information and trouble shooting advice.

F  ********************************************************************************

F  [Thr 140265776588544] *** LOG => exiting (exitcode -11112, retcode 2).

M  [Thr 140265776588544] CCMS: CCMS Monitoring Cleanup finished successfully.

Shutdown occures due to expired time seen in dev_server log file:

F [Thr 139643987195744] *** LOG => Timer expired (120000 ms).

Or:

ERROR => Shutdown timer expired.

No further related entries can be found either in the instance work directory logs or in the default trace. After this AS Java can be started without any issue.

Resolution:

1. Open Configtool.

2. Choose the affected instance -> VM Parameters -> Additional -> click in New -> next to "Name" put "-XX:+DumpStackTraceOnSystemExit" -> press "OK".

3. Then you will see "-XX:+DumpStackTraceOnSystemExit" as a parameter under "VM Parameters". This parameter will be generating a thread dump from the shut down halt.

4. Restart the whole instance in order to validate changes.

5. Similar thread dumps can be seen in the latest dev_server log files after the next shutdown halt:

J Thu Jan  5 05:34:07 2017

 J  JVM_Halt called with exit code 1

 J  "Managed_Application_Thread_26" #160 prio=5 os_prio=0 cpu=320.00 [reset 40.00] ms elapsed=65.97 [reset 1.01] s allocated=35801568 B (34.14 MB) [reset 3417824 B (3.26 MB)] defined_classes=46

 J  io= file i/o: 72293/14580 B, net i/o: 1747986/573008 B, files opened:3, socks opened:0  [reset file i/o: 72293/14580 B, net i/o: 82198/44320 B, files opened:3, socks opened:0 ]

 J  user="XXXXX" application="ariba.com/ariba_JobBean" isapplicationthread="true" tid=0x0000000001e88800 nid=0x1f04e / 127054  pthread-id=139842005186304 runnable  [_thread_in_vm (_running), stack(0x00007f2f80eb3000,0x00007f2f810b4000)] [0x00007f2f810af000]

 J     java.lang.Thread.State: RUNNABLE

 J   at java.lang.Shutdown.halt0(I)V(Native Method)

 J   at java.lang.Shutdown.halt(I)V(Shutdown.java:95)

 J   - locked <0x0000000747e92e78> (a java.lang.Shutdown$Lock)

 J   at java.lang.Shutdown.exit(I)V(Shutdown.java:169)

 J   - locked <0x0000000747e04a98> (a java.lang.Class for java.lang.Shutdown)

 J   at java.lang.Runtime.exit(I)V(Runtime.java:90)

 J   at java.lang.System.exit(I)V(System.java:910)

 J   at ariba.filetransferclient.Util.zip(Ljava/io/File;Ljava/io/File;Ljava/io/FilenameFilter;)V(Util.java:390)

 J   at ariba.filetransferclient.Util.generateZip(Ljava/io/File;Ljava/io/File;Ljava/io/FilenameFilter;Lorg/apache/commons/logging/Log;)V(Util.java:489)

The above example shows that ariba.com/ariba_JobBean application somehow gave a sign for shutdown process. In this case an ARIBA expert should analyse the background of the issue further.

J Fri Jan 13 06:58:33 2017

 J  JVM_Halt called with exit code -11112

 J  "Thread-86" #2642 prio=5 os_prio=0 cpu=70.00 [reset 70.00] ms elapsed=20.09 [reset 20.09] s allocated=8080384 B (7.71 MB) [reset 8080384 B (7.71 MB)] defined_classes=6

 J  io= file i/o: 8472/3475 B, net i/o: 357/85161 B, files opened:5, socks opened:-9  [reset file i/o: 8472/3475 B, net i/o: 357/85161 B, files opened:5, socks opened:-9 ]

 J  user="Administrator" isapplicationthread="true" tid=0x00007ff874da9800 nid=0x6b5b / 27483  pthread-id=140705505916672 runnable  [_thread_in_vm (_running), stack(0x00007ff88d92b000,0x00007ff88db2c000)] [0x00007ff88db2a000]

 J     java.lang.Thread.State: RUNNABLE

 J   at java.lang.Shutdown.halt0(I)V(Native Method)

J   at java.lang.Shutdown.halt(I)V(Shutdown.java:139)

J   - locked <0x00000007563f8e40> (a java.lang.Shutdown$Lock)

J   at java.lang.Shutdown.exit(I)V(Shutdown.java:213)

 J   - locked <0x00000007563f4098> (a java.lang.Class for java.lang.Shutdown)

 J   at java.lang.Runtime.exit(I)V(Runtime.java:109)

 J   at java.lang.System.exit(I)V(System.java:977)

 J   at com.sap.engine.core.Framework.shutDown0(ILjava/util/Properties;)V(Framework.java:550)

 J   - locked <0x00000007583fab88> (a com.sap.engine.core.Framework)

According to the previous entries someone with Administrator user shut down the AS Java Systems with sapcontrol, jsmon/jcmon etc. As this user is Administrator, it is not possible to find out who and why had restarted the Systems. In this case it is recommended to change the password of the Administrator user.

These two thread dumps are only examples. Other scenario might happen in different cases. This method is a good way to identify the reason of the shutdown of AS java systems.

Reference Snote: 2457769

java.lang.ClassNotFoundException: com.sap.security.core.server.secstorefs.SecStoreFS: AS Java issue

java.lang.ClassNotFoundException: com.sap.security.core.server.secstorefs.SecStoreFS: AS Java issue

Symptom:

You are trying to start the AS Java system, but the dispatcher does not start and you see the following exception in the defaultTrace.X.trc file:

Caused by: java.lang.ClassNotFoundException: com.sap.security.core.server.secstorefs.SecStoreFS

This can happen more frequently during or after an AS Java upgrade.

Solution:

There are two options to solve this issue:

Restore the tc_sec_secstorefs.jar file from backup;

 

Copy the tc_sec_secstorefs.jar file from:

/usr/sap/<SID>/<instanceID>/j2ee/cluster/server0/bin/core_lib

to

/usr/sap/<SID>/<instanceID>/j2ee/cluster/dispatcher/bin/core_lib

After this you need restart the engine.

 

Reference Snote : 2176030

How to check if secure store keyphrase is correct

How to check if secure store keyphrase is correct

If you are not sure if the KeyPhrase you know is the correct KeyPhrase which was used to encrypt the secure store in Configtool.

If you have performed System Copy and Target System is not starting. You want to check if KeyPhrase you entered in SapInst (System Copy Tool) is the correct KeyPhrase used to encrypt Secure Store on Source System.

SAP NW Java System provides a tool to check if the entered KeyPhrase is correct or not. The tool takes input as:

Solution:

• A KeyPhrase string
• A SecStore.properties

Command on Windows Machine: <Drive>:\usr\sap\<SID>\SYS\global\sltools\checkKeyPhrase.bat -f <location of SecStore.properties file>

Command on UNIX/AIX Machine: <Drive>:\usr\sap\<SID>\SYS\global\sltools\checkKeyPhrase.sh -f <location of SecStore.properties file>

After executing the command, it prompts for KeyPhrase:

Enter the KeyPhrase and system suggests if entered KeyPhrase is correct or not:

 Reference Snote: 1895736

Additional software requirement during Install or applying SAP Java(ADS) NW750 SP/Patch on Windows Server

 

Additional software requirement during Install or applying SAP Java(ADS) NW750 SP/Patch on Windows Server.

Issue:

After applying ADS (Adobe Document Services) NW750 SP/Patch on Windows Server, XMLForm module terminate abnormally.

Default trace shows following exception from ADS:

Service XMLFormService: Native process (PID=0) <drive>:\\usr\\sap\\<SID>\\<Instance>\\j2ee\\os_libs\\adssap\\XMLFormService\\bin\\XMLForm.exe terminated abnormally with error code -1073741515

SAP Environment:

This issue happens from following ADS SP/Patch level and higher:

NW750 SP-11 Patch level 8

NW750 SP-14 Patch level 3

NW750 SP-15 Patch level 2

NW750 SP-16 Patch level 1

NW750 SP-17 and higher SP

Solution:

Make sure that following Microsoft Visual C++ runtime is installed at Windows Server.

Microsoft Visual C++ 2012 Redistributable (x86)

AND

Microsoft Visual C++ 2013 Redistributable (x86)

Remark: ADS requires 32bit version of Visual C++ runtime.

Source : 2741686

SAP BASIS Overview

 

What is SAP Basis ?

1.  The term “Basis” is a reference to the underlying technical architecture of an SAP application. SAP BASIS stands for Business Application Software Integrated Solution.  The term encompasses the server infrastructure , software and the server processes that run the SAP application.

Who are Basis consultants ?

1. SAP consultants who design, build and manage the SAP environment are termed as Basis consultants. They are the SAP Technical architects in the IT organization.

What is the scope of Basis consultants in an IT organization ?

1. SAP consultants possess the skills required to administer and configure SAP specific settings in an IT landscape. They are part of a larger organization which may run a landscape consisting of several SAP and Non-SAP applications.

What are the pre-requisites for starting a career in SAP Basis?

1.  Basic knowledge of operating system administration, network management and database administration.

SAP SUM: SAPControl path is not valid in windows environment

 

SAP SUM: SAPControl path is not valid in windows environment

 We have encountered an error (as below) while performing the update activity using the SAP Software Update Manager in windows environment.

Solution

As error referred, login to your operating system (in this example, it’s Windows) using the <SID>adm.

Go to the system properties --> Advanced--> Environment variable--> user variable section--> click Path --> edit -->make sure the path (kernel directory location) is correctly maintained. example (E:\usr\sap\SID\SYS\exe\uc\NTADM64)

How to Generate a System Information XML File (Stack XML) from SAP JAVA stack

SUM: Generating a System Information XML File (stack XML)from JAVA Stack for use with SAP Maintenance Planner

Proceed as follows:

1.Download and unpack a new SUM 1.0/2.0 instance as described the SUM user guide. To download the guide relevant for your system, go to https://support.sap.com/sltoolset - > System Maintenance -> Software Update Manager <version> -> Guides for for SUM <version>.

2.Navigate to /usr/sap/<SID>/SUM/sdt/param/ on Unix- based or IBM-I OS, or <DRIVE>:\usr\sap\<sapsid>\SUM\sdt\param on Windows, and open the startup.props file for editing.

3.Create a new line with the following parameter and setting: Scenario=SYS_INFO_EXPORT 

For dual-stack systems you also have to add ABAP=false.

4.Save and close the startup.props file.

5.If required, configure the SAP Host Agent as described in the SUM user guide.

6.Register SUM in the SAP Host Agent using the following command:

For Windows: <DRIVE>:\usr\sap\<sapsid>\SUM\STARTUP.BAT

For UNIX, user root mandatory: /usr/sap/<SID>/SUM/STARTUP <SID>

7.Start SUM and complete one full run. Depending on the security setup of your system, use the following URL:

https://<hostname>:1129/lmsl/sumjava/<SID>/index.html

Note that the dialog about choosing a stack.xml-based scenario or a manually prepared directory is automatically skipped.

8.To acquire the system info XML file, navigate to /usr/sap/<SID>/SUM/sdt/htdoc/SysInfoExport.xml on Unix- based or IBM-I OS or <DRIVE>:\usr\sap\<sapsid>\SUM\sdt\htdoc\SysInfoExport.xml on Windows.

Altermatively, open a browser tab, go to http://<sum-hostname>:1128/lmsl/sumjava/<SID>/SysInfoExport.xml, and copy the information from there.

Reference note: 2293050

How to reset Sybase sapsso/sapsa password for SAP ASE for Business Suite

 

How to reset Sybase sapsso/sapsa password for SAP ASE 15/16 for Business Suite.

Follow the below steps to reset the password.

(if you forget the password of sapsa use the below steps to reset the password.

1.Shut down ASE, if it is running

2.Edit $SYBASE/$SYBASE_ASE/install/RUN_<SID> file, and add a new line with -psapsso at the end of the file.

For example:

/prd/rel157/ASE-15_0/bin/dataserver \

-d/prd/rel157/data/master.dat \

-e/prd/rel157/ASE-15_0/install/prd.log \

-c/prd/rel157/ASE-15_0/prd.cfg \

-M/prd/rel157/ASE-15_0 \

-sprd \

-psapsso \ 

3.Start ASE. (Please make sure there is no error reported during the start of ASE like max memory not high enough, please fix this in CFG file(/sybase/SID/SID.cfg) and start again before proceeding)

4.A new password will be printed into terminal screen right after the last system database online information.

(Note: This new password is only printed on the terminal screen. It is NOT written to the errorlog. Do not close this session until after using the password. Once the session is killed/closed, the password will no longer be valid. )

It looks like this (this is only an example, you will see a different set of character than 'ddsoiadknnygf4'):

 New SSO password for sapsso: ddsoiadknnygf4

5.Please immediately login to ASE using isql and change the passwords of users 'sapsso':

isql -Usapsso -P ddsoiadknnygf4 -S prd

use master

go

alter login sapsso with password "<sapsso-password>" modify password "<newpassword>"

go

After changing the passwords of database logins 'sapsso', you should immediately change the passwords for the database users in the Secure Storage. The passwords in the ABAP/Java Secure Storage are used for the connection to the ABAP/Java database schema by the application servers.

6.Edit RUN_server file and remove -psapsso entry (this is important. if you start the server again with -psapsso still in place, the sapsso password will get reset with another temporary value).

Reference from SNote: 2008256