Node Sync Issues

1. When the node agent fails to trust the DM.

Logs indicated:
.
[3/10/12 17:50:21:780 CDT] 00000027 RoleBasedAuth A SECJ0305I: The role-based authorization check failed for admin-authz operation StatusCache:placeReport:com.ibm.ws.management.status.StatusReport. The user UNAUTHENTICATED (unique ID: unauthenticated) was not granted any of the following required roles: operator, administrator. ( All of IBM infocenter instructions just confused me more, google was not helpful either)

Tried a simple approach:

1.stop all of the nodeagents since all of them where out of synch.

 

2.Manually sync the node with syncNode.sh "./syncNode.sh pwas6cons 10045" pointing to the DMs SOAP port

3. Start the node agent

Logs look much clean and beautiful now.

Nodeagent Logs:
[3/10/12 18:19:35:690 CDT] 00000028 NodeSyncTask A ADMS0003I: The configuration synchronization completed successfully.
[3/10/1218:20:35:694 CDT] 00000029 NodeSyncTask A ADMS0003I: The configuration synchronization completed successfully.

2.You have included this fix in this email, but the symptoms of this issue is when you see cluster application updates reflecting different timestamps for each instance and when you log into the deployment manager and synchronize 2 -3 minutes later when you refresh the status it shows as not being synchronized.

you would clearn up : */tmp, */config/temp and */wstemp areas

Incase cluster application updates not working as intended then please follow the steps to prevent the issue from happening again, if this works we then will come up with a script to clean up */tmp, */config/temp and */wstemp areas.
                        
1) Stop the dmgr                                                      
2) on dmgr side delete the contents under wstemp, temp and config/temp
 folder  from <profile_root>                                        
3) start the dmgr.                                                    
4) Stop the Node using stopNode command from the <profile_root>/bin of
 AppServer                                                          
5)synchronizing the node by running syncNode.sh from  <profile_root>/bin
if security is able then please run following command                 
syncNode.sh <DMgr_hostName><SOAP_PORT_of_DMGR> -username <username>  -password <password>                                                 
6)Start the node and server.

  

 

LTPA Token Corruption, EPOCH Refresh

BackGround: For successful synchronization we would need a valid LTPA token, looking at the nodeagent logs it looks like the EPOCH got refreshed and synch operation is failing after, which means the LTPA token we have may have become corrupt.  

 ADMS0005E: The system is unable to generate synchronization request: javax

.management.JMRuntimeException: ADMN0022E: Access is denied for the getRepositoryEpoch operation on ConfigRepository MBean bec

ause of insufficient or empty credentials.

In Administrative console:

1.      SSL certificate and key management -> Key set groups ->Select Key set group name -> uncheck the box for "Automatically generate keys"
Make sure to clear the Automatically generate keys option

2.      From the Key set groups -> check key set Group name and hit Generated Keys tab.

3.      Click OK and Save to save the changes to the master configuration.

4.      Stop the dmgr

5.      For the DM delete 1) wstemp, 2) temp and $config/temp -> Make SURE you only delete temp and NOT config folder under DM profile root directory

6.      Start the dmgr

7.      Stop the Node using stopNode command.

8.      Manually synchronize

9.      Start Node

10.  Verify if synchronization operation again.

We may have to restart the application server JVM as well, but we cannot do that ourselves and if it comes down to it we will have to have the application group schedule a restart request when it is convenient for them to do so

AIX commands used in the WAS/WebLogic

Vmstat	Gives: i.      Traps ii.      Virtual memory iii.      Paging iv.      CPU v.      Number of interrupts per second vi.      Kernel threads
Iostat	1.    Gives: i.      CPU usage ii.      i/o of disk, adapter, ttys iii.      i/o subsystem
topas	i.      logical partition information (# topas -L) ii.      processes(# topas -P) iii.      file system(# topas -F) iv.      disks(# topas -D)
For CPU Monitoring	Admin can make use of: 1.    netpmon 2.    sar (sar -u) 3.    topas
For memory monitoring	Admin can make use of: 1.    svmon 2.    netpmon 3.    filemon Svmon and filemon are perfagent tools
For I/O subsystem,	1.    fileplace 2.    filemon
For Network	1.    tcpdump 2.    netpmon
For processors and threads,	1.    svmon 2.    kdb 3.    fuser 4.    prof 5.    truss nmon will give entire OS performance information
COMMANDS	Mpstat: this command displays performance statistics of all logical CPU in system. Lparstat: reports LPAR related information and statistics. SAR: System Activity Records: it collects reports and saves system activity information.
Traced based commands	CPU Monitoring: tprof, trace, trcrpt Memory: trace, trcrpt i/o subsystem: trace, trcrpt network: iptrace, trace, trcrpt processes and threads: tprof, trace, trcrpt

 

Difference between Websphere 5.1, 6.1 and 7.0

Profiles
WebSphere 5.1:No Concepts of profile ,there are 4 types of Installation -Express,Base ,Network Deployment and Enterprise.
Websphere 6.1:Cell Profile,Deployment Manager profile,Application Server profile,Custom Profile
Websphere 7.0 Cell(DeploymentManager and managed node),Management,Application Server,Custom profile,Secure Proxy.
Note:Under Management there are three types of profiles available :Administrative agent

Deployment Manager

Job Manager

Note:The Main use of Job Manager is to queue jobs to application server in a flexible management environment

Managing Profiles
WebSphere 5.1 :Websphere multiple installation instance can be created using wsinstance script

WebSphere 6.1:There are two ways of managing a profile
1.Profile Management Tool(GUI)

2.Manage profiles(Command interface for managing profiles )
WebSphere 7.0: same as 6.1

Security Roles
WAS 5.1:Administrator,operator,configurator
WAS 6.1:Administrator,operator,configurator,Deployer,Admin Security Manager,ISC Admin
WAS 7.0:Administrator,operator,configurator,Deployer,Admin Security Manager,ISC Admin,Auditor

WebServers supported
WAS 5.1:Apache HttpServer,Domino Server,IHS,Microsoft IIS,Sun Java System Web Server,HTTP Server for iseries
WAS 6.1:Apache HttpServer,Domino Server,IHS,Microsoft IIS,Sun Java System Web Server
WAS 7.0:HTTPServer for Z/Os and all web servers supported in 6.1

User Registries/Repositries
WAS 5.1:Local Operating System,Standalone LDAP registry,Standalone Custom registry
WAS 6.1:Federated repositories,Local Operating System,Standalone LDAP registry,Standalone Custom registry or file based registry
WAS 7.0:Same as 6.1

lOGGING AND TRACING
WAS 5.1Diagnostic trace
JVM logs
Process logs
IBM Service logs
WAS 6.1
Apart from the logs available in 5.1 there is a Change log detail levels which will enable the Message level and trace level of the JVM
WAS 7.0Same as V 6.1

Managing WebServers
WAS 5.1:Web Servers cannot be managed through Websphere Admin Console
WAS 6.1:WebServers can be Administered using the Websphere Admin Console (Stopping, Starting, Generation and propagation of Plug-in can be done). Web Servers can be created in Managed node or in Unmanaged node
WAS 7.0 same AS V 6.1

JMS
WAS 5.1:JMS Fail Over Support and scalability is not available
WAS 6.1:JMS Fail over support and scalability is available.SIB(Service Integration Bus Concept is being introduced)
WAS 7.0:Same as V 6.1

Monitoring
WAS 5.1:N/A
WAS 6.1:TPF(Tivoli Performance Viewer) is embedded in the Websphere Admin Console for monitoring WebSphere Objects
WAS 7.0same as V 6.1

Intelligent Run Time provisioning
WAS 5.1N/A
WAS 6.1N/A
WAS 7.0Intelligent run time provisioning is a new concept introduced in V7.0 At run time the server uses the activation plan to start only those components that are required inside the application server

Components like Web Container , EJB Container , Web Service and SIP Container are dynamically activated

SIP and Portlet Container
WAS 5.1:N/A
WAS 6.1SIP(Session Initiation Protocol) extends the application server to allow to run SIP applications written to JSR 116 Specification

The Portlet applications can deployed which is compliant with JSR 168

WAS 7.0same as V 6.1

wsadmin scripts
WAS 5.1:JACL is the scripting language which is used
WAS 6.1:JACL will be deprecated from 6.1 and Jython scripting will be used.
WAS 7.0:Same as V 6.1

 

Websphere Response file in AIX enviornment

1.License Acceptance-OPT silentInstallLicenseAcceptance="true"
2. Operating System Prerequisite Checking
# If you want to disable operating system prerequisite checking, uncomment
# the following line. This will notify the installer to continue with
# the installation and log the warnings even though the prerequisite checking
# has failed.
#-OPT disableOSPrereqChecking="true"
3. Non-blocking Prerequisite Checking
# If you want to disable non-blocking prerequisite checking, uncomment
# the following line. This will notify the installer to continue with
# the installation and log the warnings even though the prerequisite checking
# has failed.
-OPT disableNonBlockingPrereqChecking="true"
4. # Install a New Copy
-OPT installType="installNew"
5. samplesSelected - Indicates that the feature is selected for installation.
# noFeature - Indicates that the feature is not selected for installation,
# this is the default option
-OPT feature="noFeature"
6. Install Location-OPT installLocation="/usr/IBM/WebSphere/AppServer"
7. Profile Creation Selection
-OPT profileType="cell"
8. # Valid Values:
# true - Administrative security is enabled, user name and
# password required.
# false - Administrative security is not enabled.
-OPT PROF_enableAdminSecurity="false"
9. Deployment Manager Profile name
-OPT PROF_dmgrProfileName=Dmgr
10. Application Server Profile name
OPT PROF_appServerProfileName=AppSrv
11. Host name
-OPT PROF_hostName=pt
12.profile path and default path
13. Deployment Manager Node name
-OPT PROF_nodeName=ITSOCellManager
14. Application Server Node name
OPT PROF_appServerNodeName=PTNode
15. Cell name
-OPT PROF_cellName=ITSOCell
16.default port
-OPT PROF_defaultPorts="true"
17. Validate Ports
-OPT PROF_validatePorts="true"
18. WebServer Check
true - enable the creation of a webserver definition.
# false - do not enable the creation of a webserver definition.
#
-OPT PROF_webServerCheck="true"
19. WebServer Type
-OPT PROF_webServerType=HIS
20. WebSer
-OPT PROF_webServerName=webserver1ver Name
21. WebServer Hostname
-OPT PROF_webServerHostname=br
22. WebServer Port
OPT PROF_webServerPort=80
Update Installer: response.txt
License Acceptance
-OPT silentInstallLicenseAcceptance="true"
2. Operating System Prerequisite Checking
-OPT disableOSPrereqChecking="true"
3. Existing Installation Checking
-OPT disableEarlyPrereqChecking="true"
4, Install Location
#-OPT installLocation="C:\Program Files\IBM\WebSphere\UpdateInstaller"

-OPT installLocation="/usr/IBM/WebSphere/UpdateInstaller

Key points from Websphere Application Server

• What is the default URL of the admin console: https://$hostname:9043/ibm/console.
• What are the default ports: HTTP: 9080, HTTPS: 9443.
• How to locate the logs: Logs can be found under $install_root/profiles/$profile_name/logs/$server_name. The default profile name is AppSrv01 and the default server name is server1. Example:/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/server1. SystemOut.log is the file containing everything that was logged to standard out. Logs can also be viewed from the admin console by navigating to Troubleshooting/Logging and Tracing/server_name/Runtime.
• How to start/stop a server: If you're dealing with a "Network Deployment" type of installation (multiple application servers running under the control of the "deployment manager"), your can start/stop a server from the console (Server/Server Types/WebSphere application servers). Otherwise you have to do it from command line. Go to install_root/bin and run ./startServer.sh server_name, e.g., ./startServer.sh server1 (this assumes that your installation has only one profile defined, otherwise you may need to "cd" to the profile_name/bin directory). Make sure that you run all commands using the appropriate system account. To stop the server, run ./stopServer.sh server_name -username user_name -password password. user_name and password is the credentials of an admin account, typically the same one you use to login to the console.
• How to deploy an application: In admin console, navigate to Applications/Application Types/WebSphere enterprise applications, click on "Install new application", select "Fast path", accept all the defaults except that on "step 2" make sure that you targeted correct servers (if you have multiple servers/clusters in your environment). Note that you can deploy a WAR file directly, you don't have to build an EAR. In this case, make sure that you set a context root on "step 4" screen of the wizard.
• How to change context root of a Web application: Go to Applications/Application Types/WebSphere enterprise applications/application_name/Context Root For Web Modules in the console. Re-start the application after the change.
• How to change the order of classloaders: If you're getting a ClassNotFoundException when you're starting the app, changing the order of classloaders is the first thing you may want to try. Go to Applications/Application Types/WebSphere enterprise applications/application_name/Manage Modules/module_name and make the appropriate selection in the "Class loader order" drop-down (this assumes you're doing it for a WAR module).
• How to enable dynamic class reloading: If you need to frequently update your deployed application (e.g., you use a local WAS installation for development), enabling dynamic reloading could be a huge time saver. Go to your application in the console, "Class loading and update detection", set "Override class reloading settings ..." and set polling interval to 2 seconds. See this post for more details on how to configure your development environment to support class reloading.
• How to find a host name and a port of the server: Go to Server/Server Types/WebSphere application servers. You'll find the host name in the Host Name column. To find a port, click on your server, and expand Ports. WC_defaulthost is the HTTP port and WC_defaulthost_secure is the HTTPS port.
• How to kill a JVM: If the normal "stop" routine failed to stop the server in a reasonable amount of time, you may need to kill it. In a "Network Deployment" environment, simply navigate to the list of servers, select the server and click "Terminate". A node agent will kill the JVM for you. To achieve the same from command line (the only option if you're running standalone), cd to install_root/profiles/profile_name/logs/server_name, and kill the process ID contained in the file server_name.pid. On Unix, you can simply do kill -9 `cat server1.pid` (assuming server1 is your server name). Use task manager or taskkill /PID on Windows.
• How to browse JMS messages: Go to Buses/Your bus name/Destinations/Your destination/Queue points/Your queue point/Runtime/Messages.
• Where to find configuration files: WAS has many configuration files, most of them are in XML/XMI format. The files are located under $install_root/profiles/$profile_name/config/cells/$cell_name.

 

Web Container errors

Generally we got errors like

• “HTTP 404 errors”
• HTTP 500 errors” on
• Incorrect information displayed on Web pages.
  If you do not know the type of error, continue with“Collect and analyze the Webserver log:
  We need check acces log files in web servers:
  _ From the access.log:
  127.0.0.1 - - [28/Mar/2007:19:52:31 -0400] "GET url HTTP/1.1" 404 304
  127.0.0.1 - - [28/Mar/2007:20:03:48 -0400] "GET urlHTTP/1.1"500 5348
  Note the URL the Web server was trying to serve when the error occurred.
  Indicate error:
  [Wed Mar 28 19:52:31 2007] [error] [client 127.0.0.1] File does
  not exist: file_location the error:

1. HTTP 404 errors:

• External factors, such as a problem in the Web server
•  Configuration problems, such as an incorrect Web server plug-in or virtual
  host configuration.
• Runtime problems, such as an application or application server not started
• User or application problems, such as an incorrectly specified URL.

Errors displayed by the browser:

• The page cannot be displayed
• JSP error or JSF error
• Failed to find resource
• File not found
• WebGroup/virtual host not defined.

JSP error or JSF error:

• incorrectly specified URL.
• is not available on the server.
•  Web server plug-in configuration problem.
• Verify that the Web server plug-in is working correctly.
• Verify that the Web server is responding
• Verify that the application is running

WebGroup / Virtual Host has not been defined:

• Verify that the virtual host configuration is correct(set the host aliases, each
• consisting of a host name and port number)

Verify that the URL causing the error is specified correctly

• Verify that the application is running

2. HTTP 500 errors

• HTTP 500 errors indicate that an internal server problem has occurred during theprocess of serving the requested page. Examples of the types of problems that can cause this in WebSphere Application Server include:
• _ JSP processor errors
• _ Application code (JSP, JSF, servlet) errors
• _ Session errors
  To verify that an application server is available to serve the file:
• 1. Determine the server or cluster that hosts the application.
• 2. Check the status of the application server.
• 3. Start the application server if it is not running
  Root causes
  Some root causes for HTTP 500 errors are:
• _ Code error
• _ Invalid session object
• _ Response generation errors

Code error

• The root cause of this problem is usually a programming error.
• Invalid session object
   
  Check the servlet or JSP code that threw the exception to make sure that the
  session is not being invalidated too early. If that is not the case, check the
  session timeout interval to ensure that it is not too short.
• The session manager component uses the HttpSession interface to create a session between an HTTP client and the server. When a new session object is
  created, a unique session ID is assigned to it.
• Session tracking is use to maintain state and user information for
  multiple request

3. A WEB GROUP / VIRTUAL HOST PROBLEM

 

SRVE0255E : URI not working when i hit first time . when i refresh browser again its working
SRVE0255E  A WebGroup/Virtual Host to handle /<contextroot> has not been defined

Resolution :

1. server –>Enterprise Applications–><your application>/Resource references –>click on Browse and choose the correct one
2.Update the plugin and propagate
3.restart the web/application server
4. Then try again URL.

 

4. Error 500 : Internal Server error : ERROR: lib_stream: openStream: Failed in r_gsk_secure_soc_init: GSK_ERROR_BAD_CERT(gsk rc = 414)

 

Cause:

The above error indicates  WAS SSL certificate was not trusted by the WAS plugin configured for IHS
 
Resolution:

Extract the default Personal Certificate:
1. Login to the WebSphere Application Server Administrative Console.

2. Select Security > SSL certificate and key management > Key Stores and certificates.

3. Select NodeDefaultKeyStore for a stand-alone deployment or CellDefaultKeyStore for a Network deployment.

4. Click Personal Certificates, select the default check box, and then click Extract.

5. Give the extracted file a path and name, such as: /root/defaultCert.arm. Note: The convention is to give the file a .arm extension.

6. Leave encoding set to Base64.

7. Click OK.

Locate your *.kdb file:

1. In the httpd.conf file, find the directory in which the plugin-cfg.xml file is stored by searching for the WebSpherePluginConfig line

example:

WebSpherePluginConfig “/opt/IBM/HTTPServer/Plugins1/config/webserver1/plugin-cfg.xml”
2. Find the directory in which the key database file (*.kdb) is stored by searching for the term “keyring” in the plugin-cfg.xml file.
For example:
<PropertyName=”keyring” Value=”/opt/IBM/HTTPServer/Plugins1/config/webserver1/plugin-key.kdb”/>
Note this location as you will need to use it later

4. ERROR : checkOptions: combination of options specified is correct (67)Address already in use: make_sock: could not bind to address 10.xx.xx.xx:10021 no listening sockets available, shutting down Unable to open logs

Cause : The port already in use by another process

Resolution
1. Check the port is running or not : netstart -an |grep <port number>
2. kill -9 pid <old pind which port is running>
3. To change the new port number
4. restart web server : /usr/local/apache /bin --- ./apachectl -k start.

5. StaleConnectionException
This exception (com.ibm.websphere.ce.cm.StaleConnectionException) indicates that the connection currently being held is no longer valid.
This can occur for numerous reasons, including the following:
1 . The application tries to get a connection and fails, as when the database is not started.
2 A connection is no longer usable due to a database failure. When an application tries to use a connection it has previously obtained, the connection is no longer valid. In this case, all connections currently in use by an application could get this error when they try to use the connection.
3 The application using the connection has already called close() and then tries to use the connection again.
4 The connection has been orphaned because the application had not used it in at most two times the orphan timeout; then the application tries to use the orphaned connection.

6.