• OAC has been working fine for….180 days….and now won’t start!
• Service appears to come up OK, but no services accessible & reboots don’t fix it.
• The short answer – DBCS “as provisioned” has a 180 day expiry on all schemas – and officially there’s no way to unexpire the accounts without changing the passwords, which’ll break “everything”.
• This post shows how to identify & debug the problem, fix it quickly and simply, and then prevent it happening again!

Identify the Problem

It could of course be a number of issues – so let’s diagnose it first.

Initial check

Check is Essbase is running, or if it’s just the UI, by SSHing onto the server and attempting to run a local MAXL connection:

sudo -u oracle /u01/data/domains/esscs/esstools/bin/startMAXL.sh

No Game! For the sake of this post, we’ll assume essbase, smartview & the UI are all non-functional. If Essbase is running, but the UI isn’t then check CPU utilisation in top – maybe the machine is still busy starting up. Wait another few minutes if the CPUs are busy (or use the Qubix startup automation that checks this for you)

Finding the logs

We need to find the respective log files to see what errors occurred in the startup. All interesting logs are in /u01/data/logs:

In this case, we looked in /u01/data/logs/AdminServer_Logs/AdminServer.log and found the following error:

java.sql.SQLException: ORA-28001: the password has expired

Validating the Error

Checking in the database, we looked at the ESSD01 accounts (this is the prefix name we used by this OAC instance):

select username, account_status, expiry_date 
from dba_users 
where username like 'ESSD%'
order by 1 asc;

We can see that the accounts have all expired – this matches the error log we saw above.

Fixing it

The official way of unexpiring an account is to set a new password.

alter user Jon identified by Chips1234;

• Note you may, in some instances, also need to unlock the account.
• Unfortunately, this won’t work for us – we need the accounts to be unexpired without changing the password since the passwords are all embedded and encrypted in the OAC environment.
• There are *some* password reset scripts on the environment to reset the password in weblogic which we could use as a fallback, but we want the lowest risk, least effort solution.
• Instead, we used the following script to extract the encrypted password and reset the password using the same password – still encrypted. We never get sight of the actual password!

SELECT 'ALTER USER '|| name ||' IDENTIFIED BY VALUES '''|| spare4 ||';'|| password ||''';' FROM sys.user$ WHERE name like 'ESSD%';

This produces output something like:

• Run the produced SQL statements to unexpire the passwords without changing them.
• (Re)Start the OAC service

Preventing it happening again

• Consider changing the password policy, otherwise these passwords will expire again in another 180 days!
• You most likely want “user” passwords to adhere to the default policy, but allow service accounts where the password is not known/used etc to persist.
• Specifically, create an alternate password policy for these service accounts that adjusts the password expiry policy, but leaves everything else as per the default.

create profile oacserviceprofile limit password_life_time unlimited;
select 'alter user '||username||' profile oacserviceprofile;'
from dba_users 
where username like 'ESS%';

And run the output to change the service account profiles:

alter user ESSD01_BIPLATFORM profile oacserviceprofile;
alter user ESSD01_WLS profile oacserviceprofile;
alter user ESSD01_MDS profile oacserviceprofile;
alter user ESSD01_STB profile oacserviceprofile;
alter user ESSD01_IAU_VIEWER profile oacserviceprofile;
alter user ESSD01_IAU profile oacserviceprofile;
alter user ESSD01_IAU_APPEND profile oacserviceprofile;
alter user ESSD01_OPSS profile oacserviceprofile;
alter user ESSD01_WLS_RUNTIME profile oacserviceprofile;

Finally, check the users are now showing a null expiry_date and all accounts are open:

select username, account_status, expiry_date from dba_users where username like 'ESS%';