19 Replies Latest reply on Nov 10, 2016 12:04 PM by alexslv

    Emails disappear after being processed by Alert Central

    mcgeg

      I have noticed that the majority of emails sent to the mailbox for Alert Central are just being deleted.

       

      In particular all of our VMware alarms are being deleted. I have deleted all Alert Sources and it still deletes the emails.

       

      Are there any logs to show how the emails are being processed?

        • Re: Emails disappear after being processed by Alert Central
          nate.mellendorf

          Wait, I think I know what's happening here.

           

          Alert Central process e-mails received, and then deletes them. I believe that's by design.

          It takes the alert, runs it against the sources I've configured, and then deletes it.

           

          If you've configured Alert Central to listen for your VMware alarms but you’re not getting any, you should post a screen capture of your rule and how it's configured.

          I bet we could figure out what's going on, and why it's not working.

           

          But yes, captured or not~ I'm almost positive AC deletes all e-mails received.

          Keep us posted!

            • Re: Emails disappear after being processed by Alert Central
              terrancekennedy

              I can confirm that AC does delete all emails that it processes and this is by design.

              As nate.mellendorf stated it may be how your processing the emails.

               

              A few thing to be aware of:

              • It can be an issue with your "Matching Rules" or an issue with your "Route To Group".
              • If your "Matching Rules" does not match something in the email it will not process it and give you a big red exclamation point in the mail alerts tab.
                • Also at this time you can start processing the alert from the email (this is helpful)
              • The Default configuration for "Route To Group" will send it to trash and not assign it.

               

              Hope this info helps

                • Re: Emails disappear after being processed by Alert Central
                  jcampore

                  we are seeing the same issue in which case alerts are coming to the mailbox designated to Solarwinds Alert central, emails are being deleted effectively, I can see when they come in and a second later deleted but these are not being triggereted by Alert Central. Not even with an Exclamation mark. Basically email comes in and gets deleted and it is happening with mostly Oracle DB email alerts being sent by our servers to the Solarwinds Alert Central mailbox. This makes me think like other posting have, that in fact Alert Central discards emails even if they dont have attachment but by some other keyword on the Body of the email. In addition, to clarify, I had created a back up of Alert Central, deleted all my emails configured email sources, and tried sending the same email to the mailbox and not triggering is happening at the ALert Central end. Therfore, there isn't a "Route" that could be trasshing these

              • Re: Emails disappear after being processed by Alert Central
                mberkhan

                I'm having the same thing happen.  Emails are being removed by Alert Central, but it's not actually generating any alerts for them.

                 

                I've been testing with three sources.  A camera, a UPS, and another program.  All 3 are sending emails out, all the emails are being received, and after a moment they are all removed by Alert Central.  Only the emails from the program are being turned into alerts; the camera and UPS are being ignored.  I've tried routing them through different SMTP gateways, putting different from addresses on them.  Nothing works.  Alert Central just won't pay attention to emails form some sources.

                 

                Are there log files anywhere we can look at to see why Alert Central isn't doing anything with these messages?

                 

                _matt

                  • Re: Emails disappear after being processed by Alert Central
                    Tomas.Saghy

                    Email processing in Alert Central is implemented in this way:

                    • Alert Central is periodically checking emails in the configured mail box: reads and deletes emails
                    • For received emails it goes thru email alert source definitions (in the same order as are defined in list of email alert sources - higher in list means higher priority) and evaluates matching rules
                      • when match is found, the email is assigned to that email alert source
                      • otherwise (no email alert source can be matched to the email) new alert in triggered state is created (summary = "Unable to determine the source of this alert", source = "not assigned") - you can use this email to create/refine your email source rules
                    • For emails matched/assigned to an email alert source:
                      • summary, object and severity is captured according to specified rules
                      • according to "route to group" settings/rules the alert is:
                        • assigned to a group (new alert in triggered state is created at this point) - and the groups escalation policy starts to execute for the new alert
                        • the email is trashed (no alert is created)

                     

                    By default the "route to group" rule is set to trash the alert - you need to explicitly change this to use some of your groups.

                    You can use the test tab on the editing email alert source window to verify what happens with your email (whether it matches; to which group the alert is routed).

                     

                    If you can't fix your issue after this explanation then please attach screenshots of you email alert source definition ("matching rule" and "route to group" tab).

                      • Re: Emails disappear after being processed by Alert Central
                        mberkhan

                        I understand what should be happening.  It's just not.

                         

                        Currently I have 0 email sources configured.  When emails come in from some sources (a monitoring program) 'Unable to determine' alerts are created.  For other sources (a camera and a UPS) no alert is created even when using the same outgoing information that worked for the monitoring program. 

                         

                        I created a source to catch the emails that are being ignored from the camera and tried the test.  It passed, but the system still deletes actual incoming info from those sources without creating alerts.

                         

                        _matt

                    • Re: Emails disappear after being processed by Alert Central
                      Sydney

                      Any ideas for this.  I have the same tinkg happening.

                      1. E mails generated by a SQL job are received from the SQL server and appear in the inbox and disappear.  They never show up in the alerts window.

                      2. Test E mails generated from the same SQL server using a vb app.  show up in AC with the red icon showing the source is not configured.

                       

                      I beleive something in the format of the E Mail is causing it not to be displayed.

                       

                      Any Ideas?

                       

                      Update:  I have reconfigured the SQL job to send the simplest E MAil with nothing in it  and not attachment.

                      From:  serveraddress

                      To: Alert Central Mail box

                      Subject  "Test"

                       

                      It never shows up in alert central.

                       

                       

                      Help

                        • Re: Emails disappear after being processed by Alert Central
                          Tomas.Saghy
                          • body of the message is taken form "text/*" message (or "text/*" part from a multipart message).
                          • there is a duplicated message detection in the code - based on the message id (the "Message-ID" header) - latest 5 message ids are considered
                          • there is a bounce/delivery status notification detection in the code - no alert is created from these mails (when such email is detected, an INFO message is written to the application log)

                          Please check the application log (/opt/oncall/logs/oncall.log file on the appliance - you can use Putty to connect to linux console; or use WinSCP to download the file; or you can bundle logs in administration console, download it and check the file in the bundle). If bounce/delivery status notification email was detected, it is logged.

                           

                          If it doesn't help then please attach the source code of the email sent by your SQL server (especially with all headers, mime parts; the text itself is not important).

                          • Re: Emails disappear after being processed by Alert Central
                            Sydney


                            Another update:  Looking at the header of both e mails the difference is in the encoding.  The E mail that works has

                            Content-Type: text/plain; charset="us-ascii"

                            Content-Transfer-Encoding: quoted-printable

                             

                            The E mail that does not work has this

                            Content-Type: text/plain; charset="utf-8"

                            Content-Transfer-Encoding: base64

                             

                            any ideas?

                              • Re: Emails disappear after being processed by Alert Central
                                mberkhan

                                Checked the log file after sending an ignored email.  Getting an error for the encoding.

                                 

                                01:39:37.343 [EmailPoller-1]  WARN c.s.oncall.email.EmailPollerImpl - Failed to process email

                                java.io.IOException: Unknown encoding:

                                        at javax.mail.internet.MimePartDataSource.getInputStream(MimePartDataSource.java:113) ~[mail-1.4.5.jar:1.4.5]

                                        at com.sun.mail.handlers.text_plain.getContent(text_plain.java:107) ~[mail-1.4.5.jar:1.4.5]

                                        at javax.activation.DataSourceDataContentHandler.getContent(DataHandler.java:790) ~[na:1.7.0_04]

                                        at javax.activation.DataHandler.getContent(DataHandler.java:537) ~[na:1.7.0_04]

                                        at javax.mail.internet.MimeBodyPart.getContent(MimeBodyPart.java:637) ~[mail-1.4.5.jar:1.4.5]

                                        at com.solarwinds.oncall.email.utils.EmailParserUtils.extractMailData(EmailParserUtils.java:77) ~[oncall-email-1.1.6-SNAPSHOT.jar:na]

                                        at com.solarwinds.oncall.email.utils.EmailParserUtils.extractMailData(EmailParserUtils.java:91) ~[oncall-email-1.1.6-SNAPSHOT.jar:na]

                                        at com.solarwinds.oncall.email.EmailPollerImpl.generateAndPublishInboundMessages(EmailPollerImpl.java:487) [oncall-email-1.1.6-SNAPSHOT.jar:na]

                                        at com.solarwinds.oncall.email.EmailPollerImpl.runEmailFetching(EmailPollerImpl.java:372) [oncall-email-1.1.6-SNAPSHOT.jar:na]

                                        at com.solarwinds.oncall.email.EmailPollerImpl.access$600(EmailPollerImpl.java:72) [oncall-email-1.1.6-SNAPSHOT.jar:na]

                                        at com.solarwinds.oncall.email.EmailPollerImpl$FetchEmail.run(EmailPollerImpl.java:308) [oncall-email-1.1.6-SNAPSHOT.jar:na]

                                        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_04]

                                        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) [na:1.7.0_04]

                                        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) [na:1.7.0_04]

                                        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_04]

                                        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_04]

                                        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_04]

                                        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_04]

                                        at java.lang.Thread.run(Thread.java:722) [na:1.7.0_04]

                                 

                                 

                                For the three systems I was testing the two that fail have this:

                                 

                                Content-Type: text/plain; charset=utf-8;

                                Content-Transfer-Encoding:

                                 

                                Content-Type: text/plain; charset=

                                Content-Transfer-Encoding: quoted-printable

                                 

                                The one that succeeds doesn't have either of those content lines.

                                 

                                _matt

                                  • Re: Emails disappear after being processed by Alert Central
                                    Sydney


                                    Thanks for the reply.  Which log file are you talking about?

                                     

                                    MIke

                                      • Re: Emails disappear after being processed by Alert Central
                                        Sydney


                                        After some more testing I have found the apparent issue.  If the e mail is sent using 64bit encoding it fails.  I send two EMails that are the same except for the encoding and the 64bit fails the 'Content-Transfer-Encoding: quoted-printable' succeeds. Now I need to get Alert Central to accept the 64bit message or have SQL send it as Content-Transfer-Encoding: quoted-printable.

                                        1 of 1 people found this helpful
                                          • Re: Emails disappear after being processed by Alert Central
                                            Tomas.Saghy

                                            Can you put here the error from the log file when you send the base64 encoded email (in the /opt/oncall/logs/oncall.log file on the appliance). Thanks

                                              • Re: Emails disappear after being processed by Alert Central
                                                Sydney

                                                00:00:00.001 [pool-6-thread-1]  INFO c.s.o.l.update.UpdateCheckScheduler - UpdateCheck via scheduled cron

                                                15:25:22.515 [EmailQueue Listener-4]  WARN c.s.o.e.p.MessageParserEngineImpl - The message doesn't match any alertSource. Assigning to default hidden alertSource.

                                                15:25:22.525 [EmailQueue Listener-1]  WARN c.s.o.e.p.MessageParserEngineImpl - The message doesn't match any alertSource. Assigning to default hidden alertSource.

                                                15:26:46.973 [EmailPoller-1]  WARN c.s.oncall.email.EmailPollerImpl - Failed to process email

                                                javax.mail.MessagingException: Problem getting the content of the message

                                                at com.solarwinds.mail.ews.EWSMessage.getContent(EWSMessage.java:66) ~[ews-provider-1.0.jar:na]

                                                at com.solarwinds.oncall.email.EmailPollerImpl.generateAndPublishInboundMessages(EmailPollerImpl.java:577) [oncall-email-1.1.0-SNAPSHOT.jar:na]

                                                at com.solarwinds.oncall.email.EmailPollerImpl.runEmailFetching(EmailPollerImpl.java:345) [oncall-email-1.1.0-SNAPSHOT.jar:na]

                                                at com.solarwinds.oncall.email.EmailPollerImpl.access$600(EmailPollerImpl.java:74) [oncall-email-1.1.0-SNAPSHOT.jar:na]

                                                at com.solarwinds.oncall.email.EmailPollerImpl$FetchEmail.run(EmailPollerImpl.java:281) [oncall-email-1.1.0-SNAPSHOT.jar:na]

                                                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_04]

                                                at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) [na:1.7.0_04]

                                                at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) [na:1.7.0_04]

                                                at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_04]

                                                at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_04]

                                                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_04]

                                                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_04]

                                                at java.lang.Thread.run(Thread.java:722) [na:1.7.0_04]

                                                Caused by: com.sun.mail.util.DecodingException: BASE64Decoder: Error in encoded stream: needed at least 2 valid base64 characters, but only got 0 before padding character (=), the 10 most recent characters were: "/rBui8khA="

                                                at com.sun.mail.util.BASE64DecoderStream.decode(BASE64DecoderStream.java:266) ~[mail-1.4.5.jar:1.4.5]

                                                at com.sun.mail.util.BASE64DecoderStream.read(BASE64DecoderStream.java:144) ~[mail-1.4.5.jar:1.4.5]

                                                at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283) ~[na:1.7.0_04]

                                                at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325) ~[na:1.7.0_04]

                                                at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177) ~[na:1.7.0_04]

                                                at java.io.InputStreamReader.read(InputStreamReader.java:184) ~[na:1.7.0_04]

                                                at com.sun.mail.handlers.text_plain.getContent(text_plain.java:125) ~[mail-1.4.5.jar:1.4.5]

                                                at javax.activation.DataSourceDataContentHandler.getContent(DataHandler.java:790) ~[na:1.7.0_04]

                                                at javax.activation.DataHandler.getContent(DataHandler.java:537) ~[na:1.7.0_04]

                                                at javax.mail.internet.MimeMessage.getContent(MimeMessage.java:1420) ~[mail-1.4.5.jar:1.4.5]

                                                at com.solarwinds.mail.ews.EWSMessage.getContent(EWSMessage.java:61) ~[ews-provider-1.0.jar:na]

                                                ... 12 common frames omitted

                                                15:26:46.978 [EmailPoller-1] ERROR c.s.oncall.email.EmailPollerImpl - email message not set to delete, forcing delete to clear all messages in mail box

                                                17:36:32.749 [localhost-startStop-1]  INFO c.s.o.c.OncallPropertyPlaceholderConfigurer$$EnhancerByCGLIB$$ad4a71fa - Loading properties file from URL [file:/etc/oncall/ocam-config.properties]

                                                17:36:33.088 [PerfDataHandler]  INFO c.s.o.perf.PerformanceTrackerImpl - PerfDataHandler thread started

                                                17:36:39.655 [localhost-startStop-1]  INFO o.a.activemq.broker.BrokerService - Using Persistence Adapter: AMQPersistenceAdapter(/opt/oncall/data/activemq-data-email)

                                                17:36:39.665 [localhost-startStop-1]  INFO o.a.a.s.amq.AMQPersistenceAdapter - AMQStore starting using directory: /opt/oncall/data/activemq-data-email

                                                17:36:39.710 [localhost-startStop-1]  INFO o.a.activemq.kaha.impl.KahaStore - Kaha Store using data directory /opt/oncall/data/activemq-data-email/kr-store/state

                                                17:36:39.758 [localhost-startStop-1]  INFO o.a.a.s.amq.AMQPersistenceAdapter - Active data files: []

                                                17:36:39.765 [localhost-startStop-1]  WARN o.a.a.s.amq.AMQPersistenceAdapter - The ReferenceStore is not valid - recovering ...

                                                17:36:39.765 [localhost-startStop-1]  INFO o.a.activemq.kaha.impl.KahaStore - Kaha Store successfully deleted data directory /opt/oncall/data/activemq-data-email/kr-store/data

                                                17:36:39.766 [localhost-startStop-1]  INFO o.a.a.s.amq.AMQPersistenceAdapter - Journal Recovery Started from: DataManager:(data-)

                                                17:36:39.781 [localhost-startStop-1]  INFO o.a.activemq.kaha.impl.KahaStore - Kaha Store using data directory /opt/oncall/data/activemq-data-email/kr-store/data

                                                17:36:40.473 [localhost-startStop-1]  INFO o.a.a.s.amq.AMQPersistenceAdapter - Recovered 2642 operations from redo log in 0.707 seconds.

                                                17:36:40.473 [localhost-startStop-1]  INFO o.a.a.s.amq.AMQPersistenceAdapter - Finished recovering the ReferenceStore

                                                17:36:40.475 [localhost-startStop-1]  INFO o.a.activemq.broker.BrokerService - ActiveMQ 5.6.0 JMS Message Broker (alertCentralEmail) is starting

                                                17:36:40.475 [localhost-startStop-1]  INFO o.a.activemq.broker.BrokerService - For help or more information please see: http://activemq.apache.org/

                                                17:36:40.652 [localhost-startStop-1]  INFO o.a.a.store.kahadb.plist.PListStore - PListStore:[/opt/oncall/data/activemq-data-email/alertCentralEmail/tmp_storage ] started

                                                17:36:40.678 [localhost-startStop-1]  INFO o.a.a.broker.TransportConnector - Connector vm://alertCentralEmail Started

                                                17:36:40.681 [localhost-startStop-1]  INFO o.a.activemq.broker.BrokerService - ActiveMQ JMS Message Broker (alertCentralEmail, ID:opsvc2as06.gomez.com-35361-1409074600486-0:1) started

                                                17:36:40.844 [localhost-startStop-1]  INFO o.a.activemq.broker.BrokerService - Using Persistence Adapter: AMQPersistenceAdapter(/opt/oncall/data/activemq-data-wf)

                                                17:36:40.844 [localhost-startStop-1]  INFO o.a.a.s.amq.AMQPersistenceAdapter - AMQStore starting using directory: /opt/oncall/data/activemq-data-wf

                                                17:36:40.853 [localhost-startStop-1]  INFO o.a.activemq.kaha.impl.KahaStore - Kaha Store using data directory /opt/oncall/data/activemq-data-wf/kr-store/state

                                                17:36:40.892 [localhost-startStop-1]  INFO o.a.a.s.amq.AMQPersistenceAdapter - Active data files: []

                                                17:36:40.892 [localhost-startStop-1]  WARN o.a.a.s.amq.AMQPersistenceAdapter - The ReferenceStore is not valid - recovering ...

                                                17:36:40.893 [localhost-startStop-1]  INFO o.a.activemq.kaha.impl.KahaStore - Kaha Store successfully deleted data directory /opt/oncall/data/activemq-data-wf/kr-store/data

                                                17:36:40.893 [localhost-startStop-1]  INFO o.a.a.s.amq.AMQPersistenceAdapter - Journal Recovery Started from: DataManager:(data-)

                                                17:36:40.905 [localhost-startStop-1]  INFO o.a.activemq.kaha.impl.KahaStore - Kaha Store using data directory /opt/oncall/data/activemq-data-wf/kr-store/data

                                                17:36:41.942 [localhost-startStop-1]  INFO o.a.a.s.amq.AMQPersistenceAdapter - Recovered 6764 operations from redo log in 1.049 seconds.

                                                17:36:41.942 [localhost-startStop-1]  INFO o.a.a.s.amq.AMQPersistenceAdapter - Finished recovering the ReferenceStore

                                                17:36:41.942 [localhost-startStop-1]  INFO o.a.activemq.broker.BrokerService - ActiveMQ 5.6.0 JMS Message Broker (alertCentralWorkflow) is starting

                                                17:36:41.943 [localhost-startStop-1]  INFO o.a.activemq.broker.BrokerService - For help or more information please see: http://activemq.apache.org/

                                                17:36:41.951 [localhost-startStop-1]  INFO o.a.a.store.kahadb.plist.PListStore - PListStore:[/opt/oncall/data/activemq-data-wf/alertCentralWorkflow/tmp_storage ] started

                                                17:36:41.956 [localhost-startStop-1]  INFO o.a.a.broker.TransportConnector - Connector vm://alertCentralWorkflow Started

                                                17:36:41.956 [localhost-startStop-1]  INFO o.a.activemq.broker.BrokerService - ActiveMQ JMS Message Broker (alertCentralWorkflow, ID:opsvc2as06.gomez.com-35361-1409074600486-0:2) started

                                                17:36:41.987 [localhost-startStop-1]  INFO o.a.activemq.broker.BrokerService - Using Persistence Adapter: AMQPersistenceAdapter(/opt/oncall/data/activemq-data-notif)

                                                17:36:41.988 [localhost-startStop-1]  INFO o.a.a.s.amq.AMQPersistenceAdapter - AMQStore starting using directory: /opt/oncall/data/activemq-data-notif

                                                17:36:41.995 [localhost-startStop-1]  INFO o.a.activemq.kaha.impl.KahaStore - Kaha Store using data directory /opt/oncall/data/activemq-data-notif/kr-store/state

                                                17:36:42.022 [localhost-startStop-1]  INFO o.a.a.s.amq.AMQPersistenceAdapter - Active data files: []

                                                17:36:42.022 [localhost-startStop-1]  WARN o.a.a.s.amq.AMQPersistenceAdapter - The ReferenceStore is not valid - recovering ...

                                                17:36:42.023 [localhost-startStop-1]  INFO o.a.activemq.kaha.impl.KahaStore - Kaha Store successfully deleted data directory /opt/oncall/data/activemq-data-notif/kr-store/data

                                                17:36:42.023 [localhost-startStop-1]  INFO o.a.a.s.amq.AMQPersistenceAdapter - Journal Recovery Started from: DataManager:(data-)

                                                17:36:42.035 [localhost-startStop-1]  INFO o.a.activemq.kaha.impl.KahaStore - Kaha Store using data directory /opt/oncall/data/activemq-data-notif/kr-store/data

                                                17:36:42.638 [localhost-startStop-1]  INFO o.a.a.s.amq.AMQPersistenceAdapter - Recovered 3406 operations from redo log in 0.615 seconds.

                                                17:36:42.638 [localhost-startStop-1]  INFO o.a.a.s.amq.AMQPersistenceAdapter - Finished recovering the ReferenceStore

                                                17:36:42.638 [localhost-startStop-1]  INFO o.a.activemq.broker.BrokerService - ActiveMQ 5.6.0 JMS Message Broker (alertCentralNotification) is starting

                                                17:36:42.639 [localhost-startStop-1]  INFO o.a.activemq.broker.BrokerService - For help or more information please see: http://activemq.apache.org/

                                                17:36:42.647 [localhost-startStop-1]  INFO o.a.a.store.kahadb.plist.PListStore - PListStore:[/opt/oncall/data/activemq-data-notif/alertCentralNotification/tmp_storage ] started

                                                17:36:42.651 [localhost-startStop-1]  INFO o.a.a.broker.TransportConnector - Connector vm://alertCentralNotification Started

                                                17:36:42.652 [localhost-startStop-1]  INFO o.a.activemq.broker.BrokerService - ActiveMQ JMS Message Broker (alertCentralNotification, ID:opsvc2as06.gomez.com-35361-1409074600486-0:3) started

                                                17:36:43.827 [localhost-startStop-1]  INFO com.solarwinds.oncall.config.Startup - Prepare application start...

                                                17:36:44.069 [localhost-startStop-1]  INFO com.solarwinds.oncall.config.Startup - Appliance version: 1.1.0.537

                                                17:36:44.622 [localhost-startStop-1]  INFO c.s.oncall.email.EmailPollerImpl - started new email poller thread

                                                17:36:44.626 [localhost-startStop-1]  INFO com.solarwinds.oncall.config.Startup - Existing Appliance UUID= 6e543abe-1255-4093-9ee7-6239efc73eec

                                                17:36:44.644 [Thread-15]  INFO c.s.o.w.o.BeforeOncallNotificationScanner - Starting before on-call scheduller ...

                                                17:36:44.647 [localhost-startStop-1]  INFO com.solarwinds.oncall.config.Startup - Application started

                                                17:36:45.431 [localhost-startStop-1]  INFO c.s.oncall.jmx.CustomMBeanExporter - Registering beans for JMX exposure on startup

                                                17:36:45.463 [localhost-startStop-1]  INFO c.s.oncall.jmx.CustomMBeanExporter - Bean with name 'emailJmx' has been autodetected for JMX exposure

                                                17:36:45.465 [localhost-startStop-1]  INFO c.s.oncall.jmx.CustomMBeanExporter - Bean with name 'orionJmx' has been autodetected for JMX exposure

                                                17:36:45.491 [localhost-startStop-1]  INFO c.s.oncall.jmx.CustomMBeanExporter - Bean with name 'notificationJmx' has been autodetected for JMX exposure

                                                17:36:45.522 [localhost-startStop-1]  INFO c.s.oncall.jmx.CustomMBeanExporter - Bean with name 'JSwisJmx' has been autodetected for JMX exposure

                                                17:36:45.546 [localhost-startStop-1]  INFO c.s.oncall.jmx.CustomMBeanExporter - Bean with name 'msgParserJmx' has been autodetected for JMX exposure

                                                17:36:45.565 [localhost-startStop-1]  INFO c.s.oncall.jmx.CustomMBeanExporter - Bean with name 'databaseJmx' has been autodetected for JMX exposure

                                                17:36:45.579 [localhost-startStop-1]  INFO c.s.oncall.jmx.CustomMBeanExporter - Bean with name 'updateCheckJmx' has been autodetected for JMX exposure

                                                17:36:47.414 [localhost-startStop-1]  INFO c.s.oncall.jmx.CustomMBeanExporter - Bean with name 'workflowJmx' has been autodetected for JMX exposure

                                                17:36:47.428 [localhost-startStop-1]  INFO c.s.oncall.jmx.CustomMBeanExporter - Located managed bean 'notificationJmx': registering with JMX server as MBean [Alert Central:name=Notification]

                                                17:36:47.493 [localhost-startStop-1]  INFO c.s.oncall.jmx.CustomMBeanExporter - Located managed bean 'msgParserJmx': registering with JMX server as MBean [Alert Central:name=Message parser]

                                                17:36:47.516 [localhost-startStop-1]  INFO c.s.oncall.jmx.CustomMBeanExporter - Located managed bean 'databaseJmx': registering with JMX server as MBean [Alert Central:name=Database]

                                                17:36:47.556 [localhost-startStop-1]  INFO c.s.oncall.jmx.CustomMBeanExporter - Located managed bean 'JSwisJmx': registering with JMX server as MBean [Alert Central:name=jSWIS]

                                                17:36:47.607 [localhost-startStop-1]  INFO c.s.oncall.jmx.CustomMBeanExporter - Located managed bean 'workflowJmx': registering with JMX server as MBean [Alert Central:name=Workflow]

                                                17:36:47.683 [localhost-startStop-1]  INFO c.s.oncall.jmx.CustomMBeanExporter - Located managed bean 'emailJmx': registering with JMX server as MBean [Alert Central:name=Email]

                                                17:36:47.805 [localhost-startStop-1]  INFO c.s.oncall.jmx.CustomMBeanExporter - Located managed bean 'orionJmx': registering with JMX server as MBean [Alert Central:name=Orion,id=Global]

                                                17:36:47.825 [localhost-startStop-1]  INFO c.s.oncall.jmx.CustomMBeanExporter - Located managed bean 'updateCheckJmx': registering with JMX server as MBean [Alert Central:name=General Update Availablity]

                                                17:36:47.826 [localhost-startStop-1]  WARN c.s.oncall.jmx.CustomMBeanExporter - Bean with key 'updateCheckJmx' has been registered as an MBean but has no exposed attributes or operations

                                                17:36:48.248 [localhost-startStop-1]  INFO c.s.o.c.impl.VelocityEngineInit - file.resource.loader.path : ./, /

                                                17:36:48.249 [localhost-startStop-1]  INFO c.s.o.c.impl.VelocityEngineInit - file.resource.loader.class : org.apache.velocity.runtime.resource.loader.FileResourceLoader

                                                17:36:48.249 [localhost-startStop-1]  INFO c.s.o.c.impl.VelocityEngineInit - resource.loader : file

                                                17:36:48.637 [localhost-startStop-1]  INFO c.s.oncall.config.NotificationConfig - class.resource.loader.class : org.apache.velocity.runtime.resource.loader.ClasspathResourceLoader

                                                17:36:48.637 [localhost-startStop-1]  INFO c.s.oncall.config.NotificationConfig - resource.loader : class

                                                17:36:49.494 [OrionSync #2]  INFO c.s.o.o.l.i.AlertCentralToOrionSyncThread - AlertCentralToOrionSyncThread #2 started

                                                17:36:49.499 [OrionSync #5]  INFO c.s.o.o.l.i.AlertCentralToOrionSyncThread - AlertCentralToOrionSyncThread #5 started

                                                17:36:49.704 [EmailPoller-1]  INFO com.solarwinds.mail.ews.EWSStore - Trying to connect by using EWS: Exchange2007_SP1

                                                17:36:52.203 [localhost-startStop-1]  INFO com.solarwinds.oncall.web.MdcFilter - Alert Central Integration Build: 1.1.6.1796

                                                17:38:02.516 [Thread-17] ERROR com.jolbox.bonecp.ConnectionHandle - Database access problem. Killing off all remaining connections in the connection pool. SQL State = 57P01

                                                17:38:02.519 [Thread-17] ERROR o.h.e.jdbc.spi.SqlExceptionHelper - FATAL: terminating connection due to administrator command

                                                17:38:02.519 [Thread-17] ERROR o.h.e.jdbc.spi.SqlExceptionHelper - An I/O error occured while sending to the backend.

                                                17:38:02.526 [BoneCP-pool-watch-thread-main] ERROR c.j.b.hooks.AbstractConnectionHook - Failed to acquire connection Sleeping for 7000ms and trying again. Attempts left: 5. Exception: java.net.ConnectException: Connection refused

                                                17:38:02.531 [Thread-17] ERROR o.s.t.support.TransactionTemplate - Application exception overridden by rollback exception

                                                org.apache.ibatis.exceptions.PersistenceException:

                                                ### Error querying database.  Cause: org.hibernate.exception.GenericJDBCException: FATAL: terminating connection due to administrator command

                                                ### The error may exist in org/activiti/db/mapping/entity/Job.xml

                                                ### The error may involve org.activiti.engine.impl.persistence.entity.JobEntity.selectNextJobsToExecute-Inline

                                                ### The error occurred while setting parameters

                                                ### SQL: select       RES.*            from ACT_RU_JOB RES          LEFT OUTER JOIN ACT_RU_EXECUTION PI ON PI.ID_ = RES.PROCESS_INSTANCE_ID_     where (RES.RETRIES_ > 0)       and (RES.DUEDATE_ is null or RES.DUEDATE_ < ?)       and (RES.LOCK_OWNER_ is null or RES.LOCK_EXP_TIME_ < ?)    and (         (RES.EXECUTION_ID_ is null)      or       (PI.SUSPENSION_STATE_ = 1)       )       LIMIT ? OFFSET ?

                                                ### Cause: org.hibernate.exception.GenericJDBCException: FATAL: terminating connection due to administrator command

                                                at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:23) ~[mybatis-3.1.1.jar:3.1.1]

                                                at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:104) ~[mybatis-3.1.1.jar:3.1.1]

                                                at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:95) ~[mybatis-3.1.1.jar:3.1.1]

                                                at org.activiti.engine.impl.db.DbSqlSession.selectListWithRawParameter(DbSqlSession.java:301) ~[activiti-engine-5.11.jar:5.11]

                                                at org.activiti.engine.impl.db.DbSqlSession.selectList(DbSqlSession.java:292) ~[activiti-engine-5.11.jar:5.11]

                                                at org.activiti.engine.impl.db.DbSqlSession.selectList(DbSqlSession.java:287) ~[activiti-engine-5.11.jar:5.11]

                                                at org.activiti.engine.impl.db.DbSqlSession.selectList(DbSqlSession.java:274) ~[activiti-engine-5.11.jar:5.11]

                                                at org.activiti.engine.impl.persistence.entity.JobManager.findNextJobsToExecute(JobManager.java:105) ~[activiti-engine-5.11.jar:5.11]

                                                at org.activiti.engine.impl.cmd.AcquireJobsCmd.execute(AcquireJobsCmd.java:50) ~[activiti-engine-5.11.jar:5.11]

                                                at org.activiti.engine.impl.cmd.AcquireJobsCmd.execute(AcquireJobsCmd.java:33) ~[activiti-engine-5.11.jar:5.11]

                                                at org.activiti.engine.impl.interceptor.CommandExecutorImpl.execute(CommandExecutorImpl.java:24) ~[activiti-engine-5.11.jar:5.11]

                                                at org.activiti.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:60) ~[activiti-engine-5.11.jar:5.11]

                                                at org.activiti.spring.SpringTransactionInterceptor$1.doInTransaction(SpringTransactionInterceptor.java:42) ~[activiti-spring-5.11.jar:na]

                                                at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130) ~[spring-tx-3.1.1.RELEASE.jar:3.1.1.RELEASE]

                                                at org.activiti.spring.SpringTransactionInterceptor.execute(SpringTransactionInterceptor.java:40) [activiti-spring-5.11.jar:na]

                                                at com.solarwinds.oncall.workflow.activiti.custom.CustomActivitiCommandInterceptor.execute(CustomActivitiCommandInterceptor.java:80) [oncall-workflow-1.1.6-SNAPSHOT.jar:na]

                                                at org.activiti.engine.impl.jobexecutor.AcquireJobsRunnable.run(AcquireJobsRunnable.java:61) [activiti-engine-5.11.jar:5.11]

                                                at java.lang.Thread.run(Thread.java:722) [na:1.7.0_04]

                                                Caused by: org.hibernate.exception.GenericJDBCException: FATAL: terminating connection due to administrator command

                                                at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:54) ~[hibernate-core-4.1.2.jar:4.1.2]

                                                at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:125) ~[hibernate-core-4.1.2.jar:4.1.2]

                                                at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:110) ~[hibernate-core-4.1.2.jar:4.1.2]

                                                at org.hibernate.engine.jdbc.internal.proxy.AbstractStatementProxyHandler.continueInvocation(AbstractStatementProxyHandler.java:129) ~[hibernate-core-4.1.2.jar:4.1.2]

                                                at org.hibernate.engine.jdbc.internal.proxy.AbstractProxyHandler.invoke(AbstractProxyHandler.java:81) ~[hibernate-core-4.1.2.jar:4.1.2]

                                                at $Proxy91.execute(Unknown Source) ~[na:na]

                                                at org.apache.ibatis.executor.statement.PreparedStatementHandler.query(PreparedStatementHandler.java:56) ~[mybatis-3.1.1.jar:3.1.1]

                                                at org.apache.ibatis.executor.statement.RoutingStatementHandler.query(RoutingStatementHandler.java:70) ~[mybatis-3.1.1.jar:3.1.1]

                                                at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:57) ~[mybatis-3.1.1.jar:3.1.1]

                                                at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:267) ~[mybatis-3.1.1.jar:3.1.1]

                                                at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:141) ~[mybatis-3.1.1.jar:3.1.1]

                                                at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:105) ~[mybatis-3.1.1.jar:3.1.1]

                                                at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:81) ~[mybatis-3.1.1.jar:3.1.1]

                                                at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:101) ~[mybatis-3.1.1.jar:3.1.1]

                                                ... 16 common frames omitted

                                                Caused by: org.postgresql.util.PSQLException: FATAL: terminating connection due to administrator command

                                                at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2102) ~[postgresql-9.1-901.jdbc4.jar:na]

                                                at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1835) ~[postgresql-9.1-901.jdbc4.jar:na]

                                                at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257) ~[postgresql-9.1-901.jdbc4.jar:na]

                                                at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500) ~[postgresql-9.1-901.jdbc4.jar:na]

                                                at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388) ~[postgresql-9.1-901.jdbc4.jar:na]

                                                at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:381) ~[postgresql-9.1-901.jdbc4.jar:na]

                                                at com.jolbox.bonecp.PreparedStatementHandle.execute(PreparedStatementHandle.java:138) ~[bonecp-0.7.1.RELEASE.jar:0.7.1.RELEASE]

                                                at sun.reflect.GeneratedMethodAccessor116.invoke(Unknown Source) ~[na:na]

                                                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_04]

                                                at java.lang.reflect.Method.invoke(Method.java:601) ~[na:1.7.0_04]

                                                at org.hibernate.engine.jdbc.internal.proxy.AbstractStatementProxyHandler.continueInvocation(AbstractStatementProxyHandler.java:122) ~[hibernate-core-4.1.2.jar:4.1.2]

                                                ... 26 common frames omitted

                                                17:38:03.742 [ActiveMQ ShutdownHook]  INFO o.a.activemq.broker.BrokerService - ActiveMQ Message Broker (alertCentralNotification, ID:opsvc2as06.gomez.com-35361-1409074600486-0:3) is shutting down

                                                17:38:03.744 [ActiveMQ ShutdownHook]  INFO o.a.activemq.broker.BrokerService - ActiveMQ Message Broker (alertCentralWorkflow, ID:opsvc2as06.gomez.com-35361-1409074600486-0:2) is shutting down

                                                17:38:03.745 [ActiveMQ ShutdownHook]  INFO o.a.activemq.broker.BrokerService - ActiveMQ Message Broker (alertCentralEmail, ID:opsvc2as06.gomez.com-35361-1409074600486-0:1) is shutting down

                                                17:38:03.765 [ActiveMQ ShutdownHook]  INFO o.a.a.broker.TransportConnector - Connector vm://alertCentralNotification Stopped

                                                17:38:03.765 [ActiveMQ ShutdownHook]  INFO o.a.a.broker.TransportConnector - Connector vm://alertCentralWorkflow Stopped

                                                17:38:03.766 [ActiveMQ ShutdownHook]  INFO o.a.a.broker.TransportConnector - Connector vm://alertCentralEmail Stopped

                                                17:38:03.768 [ActiveMQ ShutdownHook]  INFO o.a.a.store.kahadb.plist.PListStore - PListStore:[/opt/oncall/data/activemq-data-email/alertCentralEmail/tmp_storage ] stopped

                                                17:38:03.768 [ActiveMQ ShutdownHook]  INFO o.a.a.store.kahadb.plist.PListStore - PListStore:[/opt/oncall/data/activemq-data-notif/alertCentralNotification/tmp_storage ] stopped

                                                17:38:03.768 [ActiveMQ ShutdownHook]  INFO o.a.a.store.kahadb.plist.PListStore - PListStore:[/opt/oncall/data/activemq-data-wf/alertCentralWorkflow/tmp_storage ] stopped

                                                17:38:03.838 [ActiveMQ ShutdownHook]  INFO o.a.activemq.broker.BrokerService - ActiveMQ JMS Message Broker (alertCentralWorkflow, ID:opsvc2as06.gomez.com-35361-1409074600486-0:2) stopped

                                                17:38:03.838 [ActiveMQ ShutdownHook]  INFO o.a.activemq.broker.BrokerService - ActiveMQ JMS Message Broker (alertCentralNotification, ID:opsvc2as06.gomez.com-35361-1409074600486-0:3) stopped

                                                17:38:03.841 [ActiveMQ ShutdownHook]  INFO o.a.activemq.broker.BrokerService - ActiveMQ JMS Message Broker (alertCentralEmail, ID:opsvc2as06.gomez.com-35361-1409074600486-0:1) stopped

                                                17:38:47.097 [localhost-startStop-1]  INFO c.s.o.c.OncallPropertyPlaceholderConfigurer$$EnhancerByCGLIB$$224bd883 - Loading properties file from URL [file:/etc/oncall/ocam-config.properties]

                                                17:38:47.434 [PerfDataHandler]  INFO c.s.o.perf.PerformanceTrackerImpl - PerfDataHandler thread started

                                                17:38:52.522 [localhost-startStop-1]  INFO o.a.activemq.broker.BrokerService - Using Persistence Adapter: AMQPersistenceAdapter(/opt/oncall/data/activemq-data-email)

                                                17:38:52.534 [localhost-startStop-1]  INFO o.a.a.s.amq.AMQPersistenceAdapter - AMQStore starting using directory: /opt/oncall/data/activemq-data-email

                                                17:38:52.580 [localhost-startStop-1]  INFO o.a.activemq.kaha.impl.KahaStore - Kaha Store using data directory /opt/oncall/data/activemq-data-email/kr-store/state

                                                17:38:52.629 [localhost-startStop-1]  INFO o.a.a.s.amq.AMQPersistenceAdapter - Active data files: []

                                                17:38:52.642 [localhost-startStop-1]  INFO o.a.activemq.broker.BrokerService - ActiveMQ 5.6.0 JMS Message Broker (alertCentralEmail) is starting

                                                17:38:52.642 [localhost-startStop-1]  INFO o.a.activemq.broker.BrokerService - For help or more information please see: http://activemq.apache.org/

                                                17:38:52.802 [localhost-startStop-1]  INFO o.a.activemq.kaha.impl.KahaStore - Kaha Store using data directory /opt/oncall/data/activemq-data-email/kr-store/data

                                                17:38:52.867 [localhost-startStop-1]  INFO o.a.a.store.kahadb.plist.PListStore - PListStore:[/opt/oncall/data/activemq-data-email/alertCentralEmail/tmp_storage ] started

                                                17:38:52.907 [localhost-startStop-1]  INFO o.a.a.broker.TransportConnector - Connector vm://alertCentralEmail Started

                                                17:38:52.910 [localhost-startStop-1]  INFO o.a.activemq.broker.BrokerService - ActiveMQ JMS Message Broker (alertCentralEmail, ID:opsvc2as06.gomez.com-33077-1409074732665-0:1) started

                                                17:38:53.117 [localhost-startStop-1]  INFO o.a.activemq.broker.BrokerService - Using Persistence Adapter: AMQPersistenceAdapter(/opt/oncall/data/activemq-data-wf)

                                                17:38:53.118 [localhost-startStop-1]  INFO o.a.a.s.amq.AMQPersistenceAdapter - AMQStore starting using directory: /opt/oncall/data/activemq-data-wf

                                                17:38:53.126 [localhost-startStop-1]  INFO o.a.activemq.kaha.impl.KahaStore - Kaha Store using data directory /opt/oncall/data/activemq-data-wf/kr-store/state

                                                17:38:53.130 [localhost-startStop-1]  INFO o.a.a.s.amq.AMQPersistenceAdapter - Active data files: []

                                                17:38:53.147 [localhost-startStop-1]  INFO o.a.activemq.broker.BrokerService - ActiveMQ 5.6.0 JMS Message Broker (alertCentralWorkflow) is starting

                                                17:38:53.147 [localhost-startStop-1]  INFO o.a.activemq.broker.BrokerService - For help or more information please see: http://activemq.apache.org/

                                                17:38:53.147 [localhost-startStop-1]  INFO o.a.activemq.kaha.impl.KahaStore - Kaha Store using data directory /opt/oncall/data/activemq-data-wf/kr-store/data

                                                17:38:53.157 [localhost-startStop-1]  INFO o.a.a.store.kahadb.plist.PListStore - PListStore:[/opt/oncall/data/activemq-data-wf/alertCentralWorkflow/tmp_storage ] started

                                                17:38:53.163 [localhost-startStop-1]  INFO o.a.a.broker.TransportConnector - Connector vm://alertCentralWorkflow Started

                                                17:38:53.163 [localhost-startStop-1]  INFO o.a.activemq.broker.BrokerService - ActiveMQ JMS Message Broker (alertCentralWorkflow, ID:opsvc2as06.gomez.com-33077-1409074732665-0:2) started

                                                17:38:53.208 [localhost-startStop-1]  INFO o.a.activemq.broker.BrokerService - Using Persistence Adapter: AMQPersistenceAdapter(/opt/oncall/data/activemq-data-notif)

                                                17:38:53.215 [localhost-startStop-1]  INFO o.a.a.s.amq.AMQPersistenceAdapter - AMQStore starting using directory: /opt/oncall/data/activemq-data-notif

                                                17:38:53.226 [localhost-startStop-1]  INFO o.a.activemq.kaha.impl.KahaStore - Kaha Store using data directory /opt/oncall/data/activemq-data-notif/kr-store/state

                                                17:38:53.234 [localhost-startStop-1]  INFO o.a.a.s.amq.AMQPersistenceAdapter - Active data files: []

                                                17:38:53.251 [localhost-startStop-1]  INFO o.a.activemq.broker.BrokerService - ActiveMQ 5.6.0 JMS Message Broker (alertCentralNotification) is starting

                                                17:38:53.251 [localhost-startStop-1]  INFO o.a.activemq.broker.BrokerService - For help or more information please see: http://activemq.apache.org/

                                                17:38:53.252 [localhost-startStop-1]  INFO o.a.activemq.kaha.impl.KahaStore - Kaha Store using data directory /opt/oncall/data/activemq-data-notif/kr-store/data

                                                17:38:53.261 [localhost-startStop-1]  INFO o.a.a.store.kahadb.plist.PListStore - PListStore:[/opt/oncall/data/activemq-data-notif/alertCentralNotification/tmp_storage ] started

                                                17:38:53.271 [localhost-startStop-1]  INFO o.a.a.broker.TransportConnector - Connector vm://alertCentralNotification Started

                                                17:38:53.271 [localhost-startStop-1]  INFO o.a.activemq.broker.BrokerService - ActiveMQ JMS Message Broker (alertCentralNotification, ID:opsvc2as06.gomez.com-33077-1409074732665-0:3) started

                                                17:38:54.353 [localhost-startStop-1]  INFO com.solarwinds.oncall.config.Startup - Prepare application start...

                                                17:38:54.480 [localhost-startStop-1]  INFO com.solarwinds.oncall.config.Startup - Appliance version: 1.1.6.693

                                                17:38:54.796 [localhost-startStop-1]  INFO c.s.oncall.email.EmailPollerImpl - started new email poller thread

                                                17:38:54.799 [localhost-startStop-1]  INFO com.solarwinds.oncall.config.Startup - Existing Appliance UUID= 6e543abe-1255-4093-9ee7-6239efc73eec

                                                17:38:54.815 [Thread-15]  INFO c.s.o.w.o.BeforeOncallNotificationScanner - Starting before on-call scheduller ...

                                                17:38:54.815 [localhost-startStop-1]  INFO com.solarwinds.oncall.config.Startup - Application started

                                                17:38:55.289 [localhost-startStop-1]  INFO c.s.oncall.jmx.CustomMBeanExporter - Registering beans for JMX exposure on startup

                                                17:38:55.309 [localhost-startStop-1]  INFO c.s.oncall.jmx.CustomMBeanExporter - Bean with name 'emailJmx' has been autodetected for JMX exposure

                                                17:38:55.311 [localhost-startStop-1]  INFO c.s.oncall.jmx.CustomMBeanExporter - Bean with name 'orionJmx' has been autodetected for JMX exposure

                                                17:38:55.327 [localhost-startStop-1]  INFO c.s.oncall.jmx.CustomMBeanExporter - Bean with name 'notificationJmx' has been autodetected for JMX exposure

                                                17:38:55.338 [localhost-startStop-1]  INFO c.s.oncall.jmx.CustomMBeanExporter - Bean with name 'JSwisJmx' has been autodetected for JMX exposure

                                                17:38:55.347 [localhost-startStop-1]  INFO c.s.oncall.jmx.CustomMBeanExporter - Bean with name 'msgParserJmx' has been autodetected for JMX exposure

                                                17:38:55.355 [localhost-startStop-1]  INFO c.s.oncall.jmx.CustomMBeanExporter - Bean with name 'databaseJmx' has been autodetected for JMX exposure

                                                17:38:55.360 [localhost-startStop-1]  INFO c.s.oncall.jmx.CustomMBeanExporter - Bean with name 'updateCheckJmx' has been autodetected for JMX exposure

                                                17:38:56.648 [localhost-startStop-1]  INFO c.s.oncall.jmx.CustomMBeanExporter - Bean with name 'workflowJmx' has been autodetected for JMX exposure

                                                17:38:56.654 [localhost-startStop-1]  INFO c.s.oncall.jmx.CustomMBeanExporter - Located managed bean 'notificationJmx': registering with JMX server as MBean [Alert Central:name=Notification]

                                                17:38:56.685 [localhost-startStop-1]  INFO c.s.oncall.jmx.CustomMBeanExporter - Located managed bean 'msgParserJmx': registering with JMX server as MBean [Alert Central:name=Message parser]

                                                17:38:56.696 [localhost-startStop-1]  INFO c.s.oncall.jmx.CustomMBeanExporter - Located managed bean 'databaseJmx': registering with JMX server as MBean [Alert Central:name=Database]

                                                17:38:56.711 [localhost-startStop-1]  INFO c.s.oncall.jmx.CustomMBeanExporter - Located managed bean 'JSwisJmx': registering with JMX server as MBean [Alert Central:name=jSWIS]

                                                17:38:56.730 [localhost-startStop-1]  INFO c.s.oncall.jmx.CustomMBeanExporter - Located managed bean 'workflowJmx': registering with JMX server as MBean [Alert Central:name=Workflow]

                                                17:38:56.757 [localhost-startStop-1]  INFO c.s.oncall.jmx.CustomMBeanExporter - Located managed bean 'emailJmx': registering with JMX server as MBean [Alert Central:name=Email]

                                                17:38:56.783 [localhost-startStop-1]  INFO c.s.oncall.jmx.CustomMBeanExporter - Located managed bean 'orionJmx': registering with JMX server as MBean [Alert Central:name=Orion,id=Global]

                                                17:38:56.791 [localhost-startStop-1]  INFO c.s.oncall.jmx.CustomMBeanExporter - Located managed bean 'updateCheckJmx': registering with JMX server as MBean [Alert Central:name=General Update Availablity]

                                                17:38:56.792 [localhost-startStop-1]  WARN c.s.oncall.jmx.CustomMBeanExporter - Bean with key 'updateCheckJmx' has been registered as an MBean but has no exposed attributes or operations

                                                17:38:57.134 [localhost-startStop-1]  INFO c.s.o.c.impl.VelocityEngineInit - file.resource.loader.path : ./, /

                                                17:38:57.135 [localhost-startStop-1]  INFO c.s.o.c.impl.VelocityEngineInit - file.resource.loader.class : org.apache.velocity.runtime.resource.loader.FileResourceLoader

                                                17:38:57.135 [localhost-startStop-1]  INFO c.s.o.c.impl.VelocityEngineInit - resource.loader : file

                                                17:38:57.491 [localhost-startStop-1]  INFO c.s.oncall.config.NotificationConfig - class.resource.loader.class : org.apache.velocity.runtime.resource.loader.ClasspathResourceLoader

                                                17:38:57.491 [localhost-startStop-1]  INFO c.s.oncall.config.NotificationConfig - resource.loader : class

                                                17:38:58.216 [OrionSync #2]  INFO c.s.o.o.l.i.AlertCentralToOrionSyncThread - AlertCentralToOrionSyncThread #2 started

                                                17:38:58.223 [OrionSync #5]  INFO c.s.o.o.l.i.AlertCentralToOrionSyncThread - AlertCentralToOrionSyncThread #5 started

                                                17:38:59.977 [EmailPoller-1]  INFO com.solarwinds.mail.ews.EWSStore - Trying to connect by using EWS: Exchange2007_SP1

                                                17:39:00.085 [localhost-startStop-1]  INFO com.solarwinds.oncall.web.MdcFilter - Alert Central Integration Build: 1.1.6.1796

                                                17:42:22.233 [http-bio-8080-exec-7] ERROR c.s.o.logic.update.UpdateCheckLogic - UpdateCheckVAMI.loadVersionjava.io.FileNotFoundException: /opt/vmware/etc/UpdateVami.txt (No such file or directory)

                                                17:44:31.694 [EmailPoller-1]  WARN c.s.oncall.email.EmailPollerImpl - Failed to process email

                                                javax.mail.MessagingException: Problem getting the content of the message

                                                at com.solarwinds.mail.ews.EWSMessage.getContent(EWSMessage.java:66) ~[ews-provider-1.1.5.jar:na]

                                                at com.solarwinds.oncall.email.utils.EmailParserUtils.extractMailData(EmailParserUtils.java:77) ~[oncall-email-1.1.6-SNAPSHOT.jar:na]

                                                at com.solarwinds.oncall.email.EmailPollerImpl.generateAndPublishInboundMessages(EmailPollerImpl.java:487) [oncall-email-1.1.6-SNAPSHOT.jar:na]

                                                at com.solarwinds.oncall.email.EmailPollerImpl.runEmailFetching(EmailPollerImpl.java:372) [oncall-email-1.1.6-SNAPSHOT.jar:na]

                                                at com.solarwinds.oncall.email.EmailPollerImpl.access$600(EmailPollerImpl.java:72) [oncall-email-1.1.6-SNAPSHOT.jar:na]

                                                at com.solarwinds.oncall.email.EmailPollerImpl$FetchEmail.run(EmailPollerImpl.java:308) [oncall-email-1.1.6-SNAPSHOT.jar:na]

                                                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_04]

                                                at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) [na:1.7.0_04]

                                                at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) [na:1.7.0_04]

                                                at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_04]

                                                at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_04]

                                                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_04]

                                                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_04]

                                                at java.lang.Thread.run(Thread.java:722) [na:1.7.0_04]

                                                Caused by: com.sun.mail.util.DecodingException: BASE64Decoder: Error in encoded stream: needed at least 2 valid base64 characters, but only got 0 before padding character (=), the 10 most recent characters were: "ZGgRUK2RQ="

                                                at com.sun.mail.util.BASE64DecoderStream.decode(BASE64DecoderStream.java:266) ~[mail-1.4.5.jar:1.4.5]

                                                at com.sun.mail.util.BASE64DecoderStream.read(BASE64DecoderStream.java:144) ~[mail-1.4.5.jar:1.4.5]

                                                at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283) ~[na:1.7.0_04]

                                                at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325) ~[na:1.7.0_04]

                                                at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177) ~[na:1.7.0_04]

                                                at java.io.InputStreamReader.read(InputStreamReader.java:184) ~[na:1.7.0_04]

                                                at com.sun.mail.handlers.text_plain.getContent(text_plain.java:125) ~[mail-1.4.5.jar:1.4.5]

                                                at javax.activation.DataSourceDataContentHandler.getContent(DataHandler.java:790) ~[na:1.7.0_04]

                                                at javax.activation.DataHandler.getContent(DataHandler.java:537) ~[na:1.7.0_04]

                                                at javax.mail.internet.MimeMessage.getContent(MimeMessage.java:1420) ~[mail-1.4.5.jar:1.4.5]

                                                at com.solarwinds.mail.ews.EWSMessage.getContent(EWSMessage.java:61) ~[ews-provider-1.1.5.jar:na]

                                                ... 13 common frames omitted

                                                17:44:31.695 [EmailPoller-1] ERROR c.s.oncall.email.EmailPollerImpl - email message not set to delete, forcing delete to clear all messages in mail box

                                                17:45:55.822 [EmailPoller-1] ERROR c.s.oncall.email.EmailPollerImpl - Duplicate e-mail message

                                                17:47:14.989 [EmailPoller-1] ERROR c.s.oncall.email.EmailPollerImpl - Duplicate e-mail message

                                                17:47:35.494 [http-bio-8080-exec-12] ERROR s.data.serialization.JsonSerializer - Error serializing query results

                                                java.lang.ArrayIndexOutOfBoundsException: null

                                                at java.lang.System.arraycopy(Native Method) ~[na:1.7.0_04]

                                                at org.codehaus.jackson.util.TextBuffer.buildResultArray(TextBuffer.java:699) ~[jackson-core-asl-1.9.6.jar:1.9.6]

                                                at org.codehaus.jackson.util.TextBuffer.contentsAsArray(TextBuffer.java:374) ~[jackson-core-asl-1.9.6.jar:1.9.6]

                                                at org.codehaus.jackson.io.JsonStringEncoder.quoteAsString(JsonStringEncoder.java:150) ~[jackson-core-asl-1.9.6.jar:1.9.6]

                                                at solarwinds.data.serialization.JsonValue.quote(JsonValue.java:54) ~[information-service-1.4.jar:na]

                                                at solarwinds.data.serialization.JsonObject.writeToSingleLine(JsonObject.java:39) ~[information-service-1.4.jar:na]

                                                at solarwinds.data.serialization.JsonObject.writeTo(JsonObject.java:24) ~[information-service-1.4.jar:na]

                                                at solarwinds.data.serialization.JsonSerializer.serialize(JsonSerializer.java:48) ~[information-service-1.4.jar:na]

                                                at solarwinds.informationservice.service.JSONView.render(JSONView.java:38) [information-service-ws-1.4.jar:na]

                                                at com.solarwinds.oncall.swis.PerfTrackingSwisQueryView.render(PerfTrackingSwisQueryView.java:65) [oncall-swis-module-1.1.6-SNAPSHOT.jar:na]

                                                at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1180) [spring-webmvc-3.1.1.RELEASE.jar:3.1.1.RELEASE]

                                                at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:950) [spring-webmvc-3.1.1.RELEASE.jar:3.1.1.RELEASE]

                                                at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852) [spring-webmvc-3.1.1.RELEASE.jar:3.1.1.RELEASE]

                                                at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882) [spring-webmvc-3.1.1.RELEASE.jar:3.1.1.RELEASE]

                                                at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:789) [spring-webmvc-3.1.1.RELEASE.jar:3.1.1.RELEASE]

                                                at javax.servlet.http.HttpServlet.service(HttpServlet.java:647) [servlet-api.jar:na]

                                                at javax.servlet.http.HttpServlet.service(HttpServlet.java:728) [servlet-api.jar:na]

                                                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) [catalina.jar:7.0.33]

                                                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) [catalina.jar:7.0.33]

                                                at com.solarwinds.oncall.web.MdcFilter.doFilter(MdcFilter.java:102) [MdcFilter.class:na]

                                                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) [catalina.jar:7.0.33]

                                                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) [catalina.jar:7.0.33]

                                                at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:311) [spring-security-web-3.1.0.RELEASE.jar:3.1.0.RELEASE]

                                                at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116) [spring-security-web-3.1.0.RELEASE.jar:3.1.0.RELEASE]

                                                at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83) [spring-security-web-3.1.0.RELEASE.jar:3.1.0.RELEASE]

                                                at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) [spring-security-web-3.1.0.RELEASE.jar:3.1.0.RELEASE]

                                                at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) [spring-security-web-3.1.0.RELEASE.jar:3.1.0.RELEASE]

                                                at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) [spring-security-web-3.1.0.RELEASE.jar:3.1.0.RELEASE]

                                                at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:101) [spring-security-web-3.1.0.RELEASE.jar:3.1.0.RELEASE]

                                                at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) [spring-security-web-3.1.0.RELEASE.jar:3.1.0.RELEASE]

                                                at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) [spring-security-web-3.1.0.RELEASE.jar:3.1.0.RELEASE]

                                                at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) [spring-security-web-3.1.0.RELEASE.jar:3.1.0.RELEASE]

                                                at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:146) [spring-security-web-3.1.0.RELEASE.jar:3.1.0.RELEASE]

                                                at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) [spring-security-web-3.1.0.RELEASE.jar:3.1.0.RELEASE]

                                                at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54) [spring-security-web-3.1.0.RELEASE.jar:3.1.0.RELEASE]

                                                at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) [spring-security-web-3.1.0.RELEASE.jar:3.1.0.RELEASE]

                                                at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45) [spring-security-web-3.1.0.RELEASE.jar:3.1.0.RELEASE]

                                                at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) [spring-security-web-3.1.0.RELEASE.jar:3.1.0.RELEASE]

                                                at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:182) [spring-security-web-3.1.0.RELEASE.jar:3.1.0.RELEASE]

                                                at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) [spring-security-web-3.1.0.RELEASE.jar:3.1.0.RELEASE]

                                                at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105) [spring-security-web-3.1.0.RELEASE.jar:3.1.0.RELEASE]

                                                at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) [spring-security-web-3.1.0.RELEASE.jar:3.1.0.RELEASE]

                                                at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) [spring-security-web-3.1.0.RELEASE.jar:3.1.0.RELEASE]

                                                at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) [spring-security-web-3.1.0.RELEASE.jar:3.1.0.RELEASE]

                                                at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:173) [spring-security-web-3.1.0.RELEASE.jar:3.1.0.RELEASE]

                                                at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346) [spring-web-3.1.1.RELEASE.jar:3.1.1.RELEASE]

                                                at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:259) [spring-web-3.1.1.RELEASE.jar:3.1.1.RELEASE]

                                                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) [catalina.jar:7.0.33]

                                                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) [catalina.jar:7.0.33]

                                                at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88) [spring-web-3.1.1.RELEASE.jar:3.1.1.RELEASE]

                                                at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) [spring-web-3.1.1.RELEASE.jar:3.1.1.RELEASE]

                                                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) [catalina.jar:7.0.33]

                                                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) [catalina.jar:7.0.33]

                                                at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) [catalina.jar:7.0.33]

                                                at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) [catalina.jar:7.0.33]

                                                at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) [catalina.jar:7.0.33]

                                                at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99) [catalina.jar:7.0.33]

                                                at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:931) [catalina.jar:7.0.33]

                                                at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) [catalina.jar:7.0.33]

                                                at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407) [catalina.jar:7.0.33]

                                                at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004) [tomcat-coyote.jar:7.0.33]

                                                at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589) [tomcat-coyote.jar:7.0.33]

                                                at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312) [tomcat-coyote.jar:7.0.33]

                                                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_04]

                                                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_04]

                                                at java.lang.Thread.run(Thread.java:722) [na:1.7.0_04]

                                                17:47:40.688 [http-bio-8080-exec-7] ERROR c.s.o.logic.update.UpdateCheckLogic - UpdateCheckVAMI.loadVersionjava.io.FileNotFoundException: /opt/vmware/etc/UpdateVami.txt (No such file or directory)

                                                  • Re: Emails disappear after being processed by Alert Central
                                                    Tomas.Saghy

                                                    It seems that there is an issue when using Exchange protocol to get mail and with base64 encoded messages. Also it seems that you are receiving multiple emails with the same "Message-ID" header (or without any Message-ID header), and it is detected as duplicate.

                                                     

                                                    Can you make a try and use IMAP or POP3 to fetch emails from the Exchange server - with the very same email?

                                                     

                                                    Can you attach the source code of the message, which causes the failure? (for example you can send the message to a gmail account and gmail has possibility to show the email source code with all the headers and in original encoding).

                                                      • Re: Emails disappear after being processed by Alert Central
                                                        Sydney

                                                        Changes to IMAP sent test messages then changed back to EWS and sent them again.  The 64 bit failed.

                                                         

                                                        00:00:00.002 [beforeOnCallNotificationScheduler-1]  INFO c.s.o.l.update.UpdateCheckScheduler - UpdateCheck via scheduled cron

                                                        00:00:00.054 [beforeOnCallNotificationScheduler-1]  INFO c.s.o.logic.update.UpdateCheckLogic - Upgrade type: NO_UPGRADE Upgrade status:NONE Appliance IP: ******

                                                        07:56:03.208 [EmailPoller-1] ERROR c.s.oncall.email.EmailPollerImpl - Fetching emails failed

                                                        javax.mail.MessagingException: Problem getting folder

                                                        at com.solarwinds.mail.ews.EWSStore.getFolder(EWSStore.java:175) ~[ews-provider-1.1.5.jar:na]

                                                        at com.solarwinds.oncall.email.EmailPollerImpl.runEmailFetching(EmailPollerImpl.java:343) [oncall-email-1.1.6-SNAPSHOT.jar:na]

                                                        at com.solarwinds.oncall.email.EmailPollerImpl.access$600(EmailPollerImpl.java:72) [oncall-email-1.1.6-SNAPSHOT.jar:na]

                                                        at com.solarwinds.oncall.email.EmailPollerImpl$FetchEmail.run(EmailPollerImpl.java:308) [oncall-email-1.1.6-SNAPSHOT.jar:na]

                                                        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_04]

                                                        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) [na:1.7.0_04]

                                                        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) [na:1.7.0_04]

                                                        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_04]

                                                        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_04]

                                                        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_04]

                                                        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_04]

                                                        at java.lang.Thread.run(Thread.java:722) [na:1.7.0_04]

                                                        Caused by: java.lang.RuntimeException: Connection not established

                                                        at com.solarwinds.mail.ews.EWSFolder.<init>(EWSFolder.java:75) ~[ews-provider-1.1.5.jar:na]

                                                        at com.solarwinds.mail.ews.EWSStore.getDefaultFolder(EWSStore.java:156) ~[ews-provider-1.1.5.jar:na]

                                                        at com.solarwinds.mail.ews.EWSStore.getFolder(EWSStore.java:165) ~[ews-provider-1.1.5.jar:na]

                                                        ... 11 common frames omitted

                                                        Caused by: microsoft.exchange.webservices.data.EWSHttpException: Connection not established

                                                        at microsoft.exchange.webservices.data.HttpClientWebRequest.throwIfConnIsNull(HttpClientWebRequest.java:394) ~[ews-provider-1.1.5.jar:na]

                                                        at microsoft.exchange.webservices.data.HttpClientWebRequest.getResponseHeaders(HttpClientWebRequest.java:280) ~[ews-provider-1.1.5.jar:na]

                                                        at microsoft.exchange.webservices.data.ExchangeServiceBase.processHttpResponseHeaders(ExchangeServiceBase.java:1045) ~[ews-provider-1.1.5.jar:na]

                                                        at microsoft.exchange.webservices.data.SimpleServiceRequestBase.internalExecute(SimpleServiceRequestBase.java:58) ~[ews-provider-1.1.5.jar:na]

                                                        at microsoft.exchange.webservices.data.MultiResponseServiceRequest.execute(MultiResponseServiceRequest.java:142) ~[ews-provider-1.1.5.jar:na]

                                                        at microsoft.exchange.webservices.data.ExchangeService.bindToFolder(ExchangeService.java:349) ~[ews-provider-1.1.5.jar:na]

                                                        at microsoft.exchange.webservices.data.ExchangeService.bindToFolder(ExchangeService.java:374) ~[ews-provider-1.1.5.jar:na]

                                                        at microsoft.exchange.webservices.data.Folder.bind(Folder.java:50) ~[ews-provider-1.1.5.jar:na]

                                                        at microsoft.exchange.webservices.data.Folder.bind(Folder.java:69) ~[ews-provider-1.1.5.jar:na]

                                                        at com.solarwinds.mail.ews.EWSFolder.<init>(EWSFolder.java:72) ~[ews-provider-1.1.5.jar:na]

                                                        ... 13 common frames omitted

                                                        07:56:08.220 [EmailPoller-1]  INFO com.solarwinds.mail.ews.EWSStore - Trying to connect by using EWS: Exchange2010

                                                        20:03:21.016 [EmailPoller-1]  WARN c.s.oncall.email.EmailPollerImpl - Failed to process email

                                                        javax.mail.MessagingException: Problem getting the content of the message

                                                        at com.solarwinds.mail.ews.EWSMessage.getContent(EWSMessage.java:66) ~[ews-provider-1.1.5.jar:na]

                                                        at com.solarwinds.oncall.email.utils.EmailParserUtils.extractMailData(EmailParserUtils.java:77) ~[oncall-email-1.1.6-SNAPSHOT.jar:na]

                                                        at com.solarwinds.oncall.email.EmailPollerImpl.generateAndPublishInboundMessages(EmailPollerImpl.java:487) [oncall-email-1.1.6-SNAPSHOT.jar:na]

                                                        at com.solarwinds.oncall.email.EmailPollerImpl.runEmailFetching(EmailPollerImpl.java:372) [oncall-email-1.1.6-SNAPSHOT.jar:na]

                                                        at com.solarwinds.oncall.email.EmailPollerImpl.access$600(EmailPollerImpl.java:72) [oncall-email-1.1.6-SNAPSHOT.jar:na]

                                                        at com.solarwinds.oncall.email.EmailPollerImpl$FetchEmail.run(EmailPollerImpl.java:308) [oncall-email-1.1.6-SNAPSHOT.jar:na]

                                                        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_04]

                                                        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) [na:1.7.0_04]

                                                        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) [na:1.7.0_04]

                                                        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_04]

                                                        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_04]

                                                        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_04]

                                                        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_04]

                                                        at java.lang.Thread.run(Thread.java:722) [na:1.7.0_04]

                                                        Caused by: com.sun.mail.util.DecodingException: BASE64Decoder: Error in encoded stream: needed at least 2 valid base64 characters, but only got 1 before padding character (=), the 10 most recent characters were: "; charset="

                                                        at com.sun.mail.util.BASE64DecoderStream.decode(BASE64DecoderStream.java:266) ~[mail-1.4.5.jar:1.4.5]

                                                        at com.sun.mail.util.BASE64DecoderStream.read(BASE64DecoderStream.java:144) ~[mail-1.4.5.jar:1.4.5]

                                                        at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283) ~[na:1.7.0_04]

                                                        at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325) ~[na:1.7.0_04]

                                                        at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177) ~[na:1.7.0_04]

                                                        at java.io.InputStreamReader.read(InputStreamReader.java:184) ~[na:1.7.0_04]

                                                        at com.sun.mail.handlers.text_plain.getContent(text_plain.java:125) ~[mail-1.4.5.jar:1.4.5]

                                                        at javax.activation.DataSourceDataContentHandler.getContent(DataHandler.java:790) ~[na:1.7.0_04]

                                                        at javax.activation.DataHandler.getContent(DataHandler.java:537) ~[na:1.7.0_04]

                                                        at javax.mail.internet.MimeMessage.getContent(MimeMessage.java:1420) ~[mail-1.4.5.jar:1.4.5]

                                                        at com.solarwinds.mail.ews.EWSMessage.getContent(EWSMessage.java:61) ~[ews-provider-1.1.5.jar:na]

                                                        ... 13 common frames omitted

                                                        20:03:21.054 [EmailPoller-1] ERROR c.s.oncall.email.EmailPollerImpl - email message not set to delete, forcing delete to clear all messages in mail box

                                                        20:04:20.641 [http-bio-8080-exec-290]  INFO c.s.oncall.email.EmailPollerImpl - stopped email polling thread...there are 0 outstanding tasks.

                                                        20:04:20.781 [http-bio-8080-exec-290]  INFO c.s.oncall.email.EmailPollerImpl - started new email poller thread

                                                        20:04:26.144 [EmailPoller-1] ERROR c.s.oncall.email.EmailPollerImpl - Email Poller Failure

                                                        javax.mail.MessagingException: outlook.office365.com

                                                        at com.sun.mail.imap.IMAPStore.protocolConnect(IMAPStore.java:670) ~[mail-1.4.5.jar:1.4.5]

                                                        at javax.mail.Service.connect(Service.java:295) ~[mail-1.4.5.jar:1.4.5]

                                                        at com.solarwinds.oncall.email.EmailPollerImpl.getNewStoreAndConnect(EmailPollerImpl.java:648) ~[oncall-email-1.1.6-SNAPSHOT.jar:na]

                                                        at com.solarwinds.oncall.email.EmailPollerImpl.access$500(EmailPollerImpl.java:72) ~[oncall-email-1.1.6-SNAPSHOT.jar:na]

                                                        at com.solarwinds.oncall.email.EmailPollerImpl$FetchEmail.run(EmailPollerImpl.java:307) ~[oncall-email-1.1.6-SNAPSHOT.jar:na]

                                                        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_04]

                                                        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) [na:1.7.0_04]

                                                        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) [na:1.7.0_04]

                                                        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_04]

                                                        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_04]

                                                        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_04]

                                                        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_04]

                                                        at java.lang.Thread.run(Thread.java:722) [na:1.7.0_04]

                                                        Caused by: java.net.UnknownHostException: outlook.office365.com

                                                        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:178) ~[na:1.7.0_04]

                                                        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391) ~[na:1.7.0_04]

                                                        at java.net.Socket.connect(Socket.java:579) ~[na:1.7.0_04]

                                                        at java.net.Socket.connect(Socket.java:528) ~[na:1.7.0_04]

                                                        at com.sun.mail.util.SocketFetcher.createSocket(SocketFetcher.java:319) ~[mail-1.4.5.jar:1.4.5]

                                                        at com.sun.mail.util.SocketFetcher.getSocket(SocketFetcher.java:233) ~[mail-1.4.5.jar:1.4.5]

                                                        at com.sun.mail.iap.Protocol.<init>(Protocol.java:113) ~[mail-1.4.5.jar:1.4.5]

                                                        at com.sun.mail.imap.protocol.IMAPProtocol.<init>(IMAPProtocol.java:111) ~[mail-1.4.5.jar:1.4.5]

                                                        at com.sun.mail.imap.IMAPStore.protocolConnect(IMAPStore.java:637) ~[mail-1.4.5.jar:1.4.5]

                                                        ... 12 common frames omitted

                                                        20:04:28.203 [http-bio-8080-exec-293]  INFO c.s.oncall.email.EmailPollerImpl - stopped email polling thread...there are 1 outstanding tasks.

                                                        20:04:28.203 [http-bio-8080-exec-293]  INFO c.s.oncall.email.EmailPollerImpl - started new email poller thread

                                                        20:04:33.209 [EmailPoller-1] ERROR c.s.oncall.email.EmailPollerImpl - Email Poller Failure

                                                        javax.mail.MessagingException: outlook.office365.com

                                                        at com.sun.mail.imap.IMAPStore.protocolConnect(IMAPStore.java:670) ~[mail-1.4.5.jar:1.4.5]

                                                        at javax.mail.Service.connect(Service.java:295) ~[mail-1.4.5.jar:1.4.5]

                                                        at com.solarwinds.oncall.email.EmailPollerImpl.getNewStoreAndConnect(EmailPollerImpl.java:648) ~[oncall-email-1.1.6-SNAPSHOT.jar:na]

                                                        at com.solarwinds.oncall.email.EmailPollerImpl.access$500(EmailPollerImpl.java:72) ~[oncall-email-1.1.6-SNAPSHOT.jar:na]

                                                        at com.solarwinds.oncall.email.EmailPollerImpl$FetchEmail.run(EmailPollerImpl.java:307) ~[oncall-email-1.1.6-SNAPSHOT.jar:na]

                                                        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_04]

                                                        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) [na:1.7.0_04]

                                                        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) [na:1.7.0_04]

                                                        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_04]

                                                        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_04]

                                                        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_04]

                                                        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_04]

                                                        at java.lang.Thread.run(Thread.java:722) [na:1.7.0_04]

                                                        Caused by: java.net.UnknownHostException: outlook.office365.com

                                                        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:178) ~[na:1.7.0_04]

                                                        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391) ~[na:1.7.0_04]

                                                        at java.net.Socket.connect(Socket.java:579) ~[na:1.7.0_04]

                                                        at java.net.Socket.connect(Socket.java:528) ~[na:1.7.0_04]

                                                        at com.sun.mail.util.SocketFetcher.createSocket(SocketFetcher.java:319) ~[mail-1.4.5.jar:1.4.5]

                                                        at com.sun.mail.util.SocketFetcher.getSocket(SocketFetcher.java:233) ~[mail-1.4.5.jar:1.4.5]

                                                        at com.sun.mail.iap.Protocol.<init>(Protocol.java:113) ~[mail-1.4.5.jar:1.4.5]

                                                        at com.sun.mail.imap.protocol.IMAPProtocol.<init>(IMAPProtocol.java:111) ~[mail-1.4.5.jar:1.4.5]

                                                        at com.sun.mail.imap.IMAPStore.protocolConnect(IMAPStore.java:637) ~[mail-1.4.5.jar:1.4.5]

                                                        ... 12 common frames omitted

                                                        20:04:43.222 [EmailPoller-1] ERROR c.s.oncall.email.EmailPollerImpl - Email Poller Failure, last exception repeated 2 times

                                                        20:04:53.233 [EmailPoller-1] ERROR c.s.oncall.email.EmailPollerImpl - Email Poller Failure, last exception repeated 4 times

                                                        20:05:13.254 [EmailPoller-1] ERROR c.s.oncall.email.EmailPollerImpl - Email Poller Failure, last exception repeated 8 times

                                                        20:05:53.297 [EmailPoller-1] ERROR c.s.oncall.email.EmailPollerImpl - Email Poller Failure, last exception repeated 16 times

                                                        20:05:55.710 [http-bio-8080-exec-290]  INFO c.s.oncall.email.EmailPollerImpl - stopped email polling thread...there are 1 outstanding tasks.

                                                        20:05:55.711 [http-bio-8080-exec-290]  INFO c.s.oncall.email.EmailPollerImpl - started new email poller thread

                                                        20:06:00.656 [http-bio-8080-exec-282]  INFO c.s.oncall.email.EmailPollerImpl - stopped email polling thread...there are 1 outstanding tasks.

                                                        20:06:00.657 [http-bio-8080-exec-282]  INFO c.s.oncall.email.EmailPollerImpl - started new email poller thread

                                                        20:06:05.661 [EmailPoller-1]  INFO com.solarwinds.mail.ews.EWSStore - Trying to connect by using EWS: Exchange2010

                                                        20:06:14.175 [EmailPoller-1]  WARN c.s.oncall.email.EmailPollerImpl - Failed to process email

                                                        javax.mail.MessagingException: Problem getting the content of the message

                                                        at com.solarwinds.mail.ews.EWSMessage.getContent(EWSMessage.java:66) ~[ews-provider-1.1.5.jar:na]

                                                        at com.solarwinds.oncall.email.utils.EmailParserUtils.extractMailData(EmailParserUtils.java:77) ~[oncall-email-1.1.6-SNAPSHOT.jar:na]

                                                        at com.solarwinds.oncall.email.EmailPollerImpl.generateAndPublishInboundMessages(EmailPollerImpl.java:487) [oncall-email-1.1.6-SNAPSHOT.jar:na]

                                                        at com.solarwinds.oncall.email.EmailPollerImpl.runEmailFetching(EmailPollerImpl.java:372) [oncall-email-1.1.6-SNAPSHOT.jar:na]

                                                        at com.solarwinds.oncall.email.EmailPollerImpl.access$600(EmailPollerImpl.java:72) [oncall-email-1.1.6-SNAPSHOT.jar:na]

                                                        at com.solarwinds.oncall.email.EmailPollerImpl$FetchEmail.run(EmailPollerImpl.java:308) [oncall-email-1.1.6-SNAPSHOT.jar:na]

                                                        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_04]

                                                        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) [na:1.7.0_04]

                                                        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) [na:1.7.0_04]

                                                        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_04]

                                                        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_04]

                                                        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_04]

                                                        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_04]

                                                        at java.lang.Thread.run(Thread.java:722) [na:1.7.0_04]

                                                        Caused by: com.sun.mail.util.DecodingException: BASE64Decoder: Error in encoded stream: needed at least 2 valid base64 characters, but only got 0 before padding character (=), the 10 most recent characters were: "; charset="

                                                        at com.sun.mail.util.BASE64DecoderStream.decode(BASE64DecoderStream.java:266) ~[mail-1.4.5.jar:1.4.5]

                                                        at com.sun.mail.util.BASE64DecoderStream.read(BASE64DecoderStream.java:144) ~[mail-1.4.5.jar:1.4.5]

                                                        at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283) ~[na:1.7.0_04]

                                                        at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325) ~[na:1.7.0_04]

                                                        at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177) ~[na:1.7.0_04]

                                                        at java.io.InputStreamReader.read(InputStreamReader.java:184) ~[na:1.7.0_04]

                                                        at com.sun.mail.handlers.text_plain.getContent(text_plain.java:125) ~[mail-1.4.5.jar:1.4.5]

                                                        at javax.activation.DataSourceDataContentHandler.getContent(DataHandler.java:790) ~[na:1.7.0_04]

                                                        at javax.activation.DataHandler.getContent(DataHandler.java:537) ~[na:1.7.0_04]

                                                        at javax.mail.internet.MimeMessage.getContent(MimeMessage.java:1420) ~[mail-1.4.5.jar:1.4.5]

                                                        at com.solarwinds.mail.ews.EWSMessage.getContent(EWSMessage.java:61) ~[ews-provider-1.1.5.jar:na]

                                                        ... 13 common frames omitted

                                                        20:06:14.191 [EmailPoller-1] ERROR c.s.oncall.email.EmailPollerImpl - email message not set to delete, forcing delete to clear all messages in mail box

                                                        20:06:17.976 [EmailPoller-1]  WARN c.s.oncall.email.EmailPollerImpl - Failed to process email

                                                        javax.mail.MessagingException: Problem getting the content of the message

                                                        at com.solarwinds.mail.ews.EWSMessage.getContent(EWSMessage.java:66) ~[ews-provider-1.1.5.jar:na]

                                                        at com.solarwinds.oncall.email.utils.EmailParserUtils.extractMailData(EmailParserUtils.java:77) ~[oncall-email-1.1.6-SNAPSHOT.jar:na]

                                                        at com.solarwinds.oncall.email.EmailPollerImpl.generateAndPublishInboundMessages(EmailPollerImpl.java:487) [oncall-email-1.1.6-SNAPSHOT.jar:na]

                                                        at com.solarwinds.oncall.email.EmailPollerImpl.runEmailFetching(EmailPollerImpl.java:372) [oncall-email-1.1.6-SNAPSHOT.jar:na]

                                                        at com.solarwinds.oncall.email.EmailPollerImpl.access$600(EmailPollerImpl.java:72) [oncall-email-1.1.6-SNAPSHOT.jar:na]

                                                        at com.solarwinds.oncall.email.EmailPollerImpl$FetchEmail.run(EmailPollerImpl.java:308) [oncall-email-1.1.6-SNAPSHOT.jar:na]

                                                        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_04]

                                                        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) [na:1.7.0_04]

                                                        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) [na:1.7.0_04]

                                                        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_04]

                                                        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_04]

                                                        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_04]

                                                        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_04]

                                                        at java.lang.Thread.run(Thread.java:722) [na:1.7.0_04]

                                                        Caused by: com.sun.mail.util.DecodingException: BASE64Decoder: Error in encoded stream: needed at least 2 valid base64 characters, but only got 1 before padding character (=), the 10 most recent characters were: "CBBbGVydA="

                                                        at com.sun.mail.util.BASE64DecoderStream.decode(BASE64DecoderStream.java:266) ~[mail-1.4.5.jar:1.4.5]

                                                        at com.sun.mail.util.BASE64DecoderStream.read(BASE64DecoderStream.java:144) ~[mail-1.4.5.jar:1.4.5]

                                                        at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283) ~[na:1.7.0_04]

                                                        at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325) ~[na:1.7.0_04]

                                                        at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177) ~[na:1.7.0_04]

                                                        at java.io.InputStreamReader.read(InputStreamReader.java:184) ~[na:1.7.0_04]

                                                        at com.sun.mail.handlers.text_plain.getContent(text_plain.java:125) ~[mail-1.4.5.jar:1.4.5]

                                                        at javax.activation.DataSourceDataContentHandler.getContent(DataHandler.java:790) ~[na:1.7.0_04]

                                                        at javax.activation.DataHandler.getContent(DataHandler.java:537) ~[na:1.7.0_04]

                                                        at javax.mail.internet.MimeMessage.getContent(MimeMessage.java:1420) ~[mail-1.4.5.jar:1.4.5]

                                                        at com.solarwinds.mail.ews.EWSMessage.getContent(EWSMessage.java:61) ~[ews-provider-1.1.5.jar:na]

                                                        ... 13 common frames omitted

                                                        20:06:17.976 [EmailPoller-1] ERROR c.s.oncall.email.EmailPollerImpl - email message not set to delete, forcing delete to clear all messages in mail box

                                                        20:06:32.982 [EmailPoller-1]  WARN c.s.oncall.email.EmailPollerImpl - Failed to process email

                                                        javax.mail.MessagingException: Problem getting the content of the message

                                                        at com.solarwinds.mail.ews.EWSMessage.getContent(EWSMessage.java:66) ~[ews-provider-1.1.5.jar:na]

                                                        at com.solarwinds.oncall.email.utils.EmailParserUtils.extractMailData(EmailParserUtils.java:77) ~[oncall-email-1.1.6-SNAPSHOT.jar:na]

                                                        at com.solarwinds.oncall.email.EmailPollerImpl.generateAndPublishInboundMessages(EmailPollerImpl.java:487) [oncall-email-1.1.6-SNAPSHOT.jar:na]

                                                        at com.solarwinds.oncall.email.EmailPollerImpl.runEmailFetching(EmailPollerImpl.java:372) [oncall-email-1.1.6-SNAPSHOT.jar:na]

                                                        at com.solarwinds.oncall.email.EmailPollerImpl.access$600(EmailPollerImpl.java:72) [oncall-email-1.1.6-SNAPSHOT.jar:na]

                                                        at com.solarwinds.oncall.email.EmailPollerImpl$FetchEmail.run(EmailPollerImpl.java:308) [oncall-email-1.1.6-SNAPSHOT.jar:na]

                                                        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_04]

                                                        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) [na:1.7.0_04]

                                                        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) [na:1.7.0_04]

                                                        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_04]

                                                        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_04]

                                                        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_04]

                                                        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_04]

                                                        at java.lang.Thread.run(Thread.java:722) [na:1.7.0_04]

                                                        Caused by: com.sun.mail.util.DecodingException: BASE64Decoder: Error in encoded stream: needed at least 2 valid base64 characters, but only got 0 before padding character (=), the 10 most recent characters were: "; charset="

                                                        at com.sun.mail.util.BASE64DecoderStream.decode(BASE64DecoderStream.java:266) ~[mail-1.4.5.jar:1.4.5]

                                                        at com.sun.mail.util.BASE64DecoderStream.read(BASE64DecoderStream.java:144) ~[mail-1.4.5.jar:1.4.5]

                                                        at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283) ~[na:1.7.0_04]

                                                        at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325) ~[na:1.7.0_04]

                                                        at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177) ~[na:1.7.0_04]

                                                        at java.io.InputStreamReader.read(InputStreamReader.java:184) ~[na:1.7.0_04]

                                                        at com.sun.mail.handlers.text_plain.getContent(text_plain.java:125) ~[mail-1.4.5.jar:1.4.5]

                                                        at javax.activation.DataSourceDataContentHandler.getContent(DataHandler.java:790) ~[na:1.7.0_04]

                                                        at javax.activation.DataHandler.getContent(DataHandler.java:537) ~[na:1.7.0_04]

                                                        at javax.mail.internet.MimeMessage.getContent(MimeMessage.java:1420) ~[mail-1.4.5.jar:1.4.5]

                                                        at com.solarwinds.mail.ews.EWSMessage.getContent(EWSMessage.java:61) ~[ews-provider-1.1.5.jar:na]

                                                        ... 13 common frames omitted

                                                        20:06:32.983 [EmailPoller-1] ERROR c.s.oncall.email.EmailPollerImpl - email message not set to delete, forcing delete to clear all messages in mail box

                                                        20:06:35.340 [EmailPoller-1]  WARN c.s.oncall.email.EmailPollerImpl - Failed to process email

                                                        javax.mail.MessagingException: Problem getting the content of the message

                                                        at com.solarwinds.mail.ews.EWSMessage.getContent(EWSMessage.java:66) ~[ews-provider-1.1.5.jar:na]

                                                        at com.solarwinds.oncall.email.utils.EmailParserUtils.extractMailData(EmailParserUtils.java:77) ~[oncall-email-1.1.6-SNAPSHOT.jar:na]

                                                        at com.solarwinds.oncall.email.EmailPollerImpl.generateAndPublishInboundMessages(EmailPollerImpl.java:487) [oncall-email-1.1.6-SNAPSHOT.jar:na]

                                                        at com.solarwinds.oncall.email.EmailPollerImpl.runEmailFetching(EmailPollerImpl.java:372) [oncall-email-1.1.6-SNAPSHOT.jar:na]

                                                        at com.solarwinds.oncall.email.EmailPollerImpl.access$600(EmailPollerImpl.java:72) [oncall-email-1.1.6-SNAPSHOT.jar:na]

                                                        at com.solarwinds.oncall.email.EmailPollerImpl$FetchEmail.run(EmailPollerImpl.java:308) [oncall-email-1.1.6-SNAPSHOT.jar:na]

                                                        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_04]

                                                        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) [na:1.7.0_04]

                                                        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) [na:1.7.0_04]

                                                        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_04]

                                                        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_04]

                                                        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_04]

                                                        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_04]

                                                        at java.lang.Thread.run(Thread.java:722) [na:1.7.0_04]

                                                        Caused by: com.sun.mail.util.DecodingException: BASE64Decoder: Error in encoded stream: needed at least 2 valid base64 characters, but only got 1 before padding character (=), the 10 most recent characters were: "CBBbGVydA="

                                                        at com.sun.mail.util.BASE64DecoderStream.decode(BASE64DecoderStream.java:266) ~[mail-1.4.5.jar:1.4.5]

                                                        at com.sun.mail.util.BASE64DecoderStream.read(BASE64DecoderStream.java:144) ~[mail-1.4.5.jar:1.4.5]

                                                        at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283) ~[na:1.7.0_04]

                                                        at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325) ~[na:1.7.0_04]

                                                        at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177) ~[na:1.7.0_04]

                                                        at java.io.InputStreamReader.read(InputStreamReader.java:184) ~[na:1.7.0_04]

                                                        at com.sun.mail.handlers.text_plain.getContent(text_plain.java:125) ~[mail-1.4.5.jar:1.4.5]

                                                        at javax.activation.DataSourceDataContentHandler.getContent(DataHandler.java:790) ~[na:1.7.0_04]

                                                        at javax.activation.DataHandler.getContent(DataHandler.java:537) ~[na:1.7.0_04]

                                                        at javax.mail.internet.MimeMessage.getContent(MimeMessage.java:1420) ~[mail-1.4.5.jar:1.4.5]

                                                        at com.solarwinds.mail.ews.EWSMessage.getContent(EWSMessage.java:61) ~[ews-provider-1.1.5.jar:na]

                                                        ... 13 common frames omitted

                                                        20:06:35.341 [EmailPoller-1] ERROR c.s.oncall.email.EmailPollerImpl - email message not set to delete, forcing delete to clear all messages in mail box

                                        • Re: Emails disappear after being processed by Alert Central
                                          mkaviani

                                          I am wondering why this issue is not resoled from 6 month ago, is there any solution to solve it?