Error: Driver output queue to EV manager exceeded (size 10000), starting over writing of Values with same DPID

[scripto]

Hello,

On our system (PVSS II ver 3.9, Linux Redhat), all stations via coupled IEC 104. The system is stable for over a month, then comes as a surprise following error:

PVSS00iec (1) 05/20/2014 20:09:04.345, SYS, SEVERE, 145, Driver output queue to EV manager exceeded (size 10000), starting over writing of Values with same DPID

The system freezes and does not accept more data. Only switching to a redundancy system revived the system. After restarting the IEC-driver everything goes back to normal.

My question: how can I display the current size of the queue utilization of IEC-driver?

The upper limit is 10000. I know that you can increase this value, but I do not know if it would solve the problem.
So I want observe the current load and then decide what to do.
someone knows a way to visualization of this value?

Greetings

[leoknipp]

Hello,

the log-message describes that the event-manager does not handle data sent by the driver fast enough.
There is no possibility to get the current queue size. If an overload situation at the driver is detected and internal datapoint-element _DriverX(_2).OV is set (X is substituted with the driver number).

In your description you've mentioned that the system freezes.
This might be the interesting point in this situation and this should be investigated. The log-message written by the driver is probably only a consecutive fault.

If you need further assistance please get in contact with the vendor for your project or with the official WinCC OA support.

Best Regards
Leopold Knipp
Senior Support Specialist

[scripto]

This error occurs at very long intervals (1-2 months) and takes place without pre-alarm. I would like to monitor the system to find the cause of the error and correct it. Sure, I can increase the value "maxOutputQueueSize". But I do not know what new problems I create when I double this value to 20000. Are there rules?
regards

[leoknipp]

Hello,

as described in the previous mail this might be only an consecutive fault.
You have to look in the PVSS_II.log-file if some errors occured before the driver has reached the limit for the output queue, e.g. a blocking manager.

Best Regards
Leopold Knipp
Senior Support Specialist

[mkoller]

you can run a shellscript periodically and send the event manager a report signal, e.g.
/opt/WinCC_OA/3.12/bin/killdbg PID -report MSGQUEUE
PID is the process id of the event manager, and the PVSS_II environment variable needs to point to your project's config file, e.g.
setenv PVSS_II /home/projekte/test312/config/config

The output you get is something like:

WCCILevent0:eventmanager message queues:
WCCILevent0: [ (SYS: 1 Data -num 0 CONN: 1) ] current: 0 max: 53 lastMsgID: 1479 remMsgCount: 0 throughput in [msg/sec]: 0 throughput out [msg/sec]: 0
WCCILevent0: [ (SYS: 1 Event -num 0 CONN: 1) ] current: 0 max: 45 lastMsgID: 714 remMsgCount: 0 throughput in [msg/sec]: 0 throughput out [msg/sec]: 0
WCCILevent0: [ (SYS: 1 Driver -num 1 CONN: 1) ] current: 0 max: 23 lastMsgID: 61 remMsgCount: 0 throughput in [msg/sec]: 0 throughput out [msg/sec]: 0
WCCILevent0:redundancy message buffer size: 0
WCCILevent0:valuechange answerqueue size : 0
WCCILevent0:connect answerqueue sizes : 0 / 0
WCCILevent0:request answerqueue size : 0

[leoknipp]

Hello,

at the Lab-section there is also a sample panel available which can be used to set a debug-flag in a defined interval:
https://portal.etm.at/index.php?option= ... &Itemid=79

To see the current buffer size in send and receive direction you can set the debug-flag "-report dispatch" for the driver.

Best Regards
Leopold Knipp
Senior Support Specialist

[scripto]

Hello,
thank you for answers. "Cyclic Report Flags" is available only from version 3.10? We use version 3.9. Returns the tool for it also correct results?
Thank you

[leoknipp]

Hello,

I'm not sure if the panel and the script also will work in 3.9.
The panel version should be OK for the usage in 3.9. You have to test it on your own if it works.

Best Regards
Leopold Knipp
Senior Support Specialist

[shotdunyun]

Hi, I am having the same problem right now with this driver overload.
It's causing the client (running on Windows) to run very slowly after 8 hours or so.
Server is running RedHat 6.5.
Before receiving the error "Driver output queue to EV manager exceeded...", I am getting this:

"Unexpected state, MsgQueueContainer, appendMsg, MSG Limit exceeded - Over the limit 200000 for connection (SYS: 1 Ctrl -num 1 CONN: 1)! Current queuesize: 200001"

I have tried the following configs:

[event]
maxInputMsgCount=200000

[driver]
maxOutputQueueSize=12000

It's still having the same problem.
As such, I have to temporarily schedule a crontab to restart winccoa process on the server.

The project is relatively small. Running three modbus drivers, with the following DPE quantities:
driver 1 - 1143
driver 2 - 333
driver 3 - 1

Would appreciate any help on this.

Thank you.

[leoknipp]

Hello,

the log-message refers to the connection to a CTRL-manager and not for one of the drivers.
Please have a look at the following FAQ
https://portal.etm.at/index.php?view=it ... &Itemid=54

If the queue for the connection driver --> event is exceeding the limit several times you have to check the load for the complete system (CPU, memory).
Maybe the driver is sending too much data, e.g. no smoothing configured or GQs are started in a short interval.

Best Regards
Leopold Knipp
Senior Support Specialist