Stackstorm UI being really slow and does not show certain options

[zsmanjot]

Hi Team,

Our production stackstorm UI is not working fine from past couple of days (being a month now exactly) and it does not populate any data . Snippet below:

Even when it shows the data , it does not show some options as highlighted in the screenshot.

Can some one help on this?

[arm4b]

Did you take a look for any errors in the web browser developer console?

[zsmanjot]

Hi @armab

Thanks for replying.

I could see below errors:

But when i specifically click on actions and open any action and then look into execution , only then it shows the history. What could be the reason that it does not load on its own?

[zsmanjot]

Hi

Could anyone help with this? It is the same and we are still struggling.

++ @arm4b

[fdrab]

What's your setup? the UI just calls the API to retrieve the results. Do you have the same issues if querying executions using postman?

[daymickcorr]

what does the command "st2 trace list" output
did you have a look at the log folder /var/log/st2

[zsmanjot]

I am unaware of what to check in logs. Could anyone help here? With my new setup also , i am unable to access the UI and gives me gateway 504 error.

[daymickcorr]

I don't know either what to look for, error 504 means stackstorm is never sending information to nginx. I can try to guess what you should look for, based on when this issue happens or what you were trying to do.

I'm pretty sure you are using the default install.

3 components of stackstorm interact with nginx, st2web uses st2auth to login then the rest is st2api.

The fact that you seem to be authenticated, indicates to me you should not look in st2auth, and probably look st2api.

essentially currently all you know is that st2api is not answering back to nginx and you want to find out why.

Look for an error at the end of the st2api log as soon as the error happens. Look at the time the log entry is created, try to find a message that seems to tell you why st2api is not responding.

[zsmanjot]

I am getting nothing in the logs as well for the errors. It just logs the query when i login to the UI.

[daymickcorr]

can you confirm you have looked at /var/log/st2/at2api.log

[zsmanjot]

All the log files are checked. Even the new ST2 boxes shows the same issue that even does not have any workflow running. @daymickcorr

[zsmanjot]

This is what i get in chrome console.

Tried on firefox as well. The same issue is there but no 504 gateway errors though. HISTORY page is always blank.

[daymickcorr]

504 gateway time-out means nginx answers but nginx cannot talk to stackstorm.

Can you look at those logs /var/log/st2/st2api.log

[zsmanjot]

Sorry , it has been long and i was moved to a different project but circling back here.

The issue remains same.

Anyone who could help here?

I dont see anything in /var/log/st2/st2api.log when i login to UI.

[daymickcorr]

The issue is between nginx and st2api, if stackstorm has no logs then look at nginx logs, i think zscaller is causing the issue

Nginx logs are at
/var/log/nginx/st2webui.*
/var/log/nginx/ssl-st2webui*

Usually what happens is st2 crashes so it does not respond to nginx, since the logs are empty it would mean that when nginx tries to reach http://127.0.0.1:9101 it does not seem to be able to communicate with it.