tag:status.punchh.com,2005:/historyPunchh Status - Incident History2024-03-28T20:55:11-07:00Punchhtag:status.punchh.com,2005:Incident/203651352024-04-11T00:30:00-07:002024-03-26T13:03:28-07:00Planned Infrastructure Maintenance - Downtime Expected<p><strong>THIS IS A SCHEDULED EVENT Apr <var data-var='date'>11</var>, <var data-var='time'>00:30</var> - <var data-var='time'>03:30</var> PDT</strong></p><p><small>Mar <var data-var='date'>26</var>, <var data-var='time'>13:03</var> PDT</small><br><strong>Scheduled</strong> - We want to notify you about an upcoming scheduled maintenance activity On Thursday April 11 at 00:30 Pacific time aimed at upgrading infrastructure components and services. This upgrade is a crucial step toward ensuring improved performance, scalability, and reliability of our systems.<br /><br />We anticipate needing shut down all services for a period of 3 hours in order to complete the upgradations. We will endeavour to keep the interruption as brief as possible.</p>tag:status.punchh.com,2005:Incident/203651142024-04-04T00:30:00-07:002024-03-26T13:01:38-07:00Planned Infrastructure Maintenance - Downtime Expected.<p><strong>THIS IS A SCHEDULED EVENT Apr <var data-var='date'>4</var>, <var data-var='time'>00:30</var> - <var data-var='time'>02:30</var> PDT</strong></p><p><small>Mar <var data-var='date'>26</var>, <var data-var='time'>13:01</var> PDT</small><br><strong>Scheduled</strong> - We want to notify you about an upcoming scheduled maintenance activity On Thursday April 04 at 00:30 Pacific time aimed at upgrading infrastructure components and services. This upgrade is a crucial step toward ensuring improved performance, scalability, and reliability of our systems.<br /><br />We anticipate needing shut down all services for a period of 2 hours in order to complete the upgradations. We will endeavour to keep the interruption as brief as possible.<br /><br />We appreciate your patience and thank you for your cooperation.!</p>tag:status.punchh.com,2005:Incident/203272512024-03-22T04:58:19-07:002024-03-22T05:17:42-07:00Punchh Dashboard Reports Are Not Functional<p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>04:58</var> PDT</small><br><strong>Resolved</strong> - The Tableau Server is stable & the reports section continues to work. We're working on the RCA.</p><p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>04:31</var> PDT</small><br><strong>Monitoring</strong> - The reports section is now functional. We continue to monitor the same.</p><p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>04:30</var> PDT</small><br><strong>Identified</strong> - It seems that the Tableau server has a licensing issue. We're trying to fix the same.</p><p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>04:10</var> PDT</small><br><strong>Investigating</strong> - The tableau server which handles the reports section in Punchh Dashboard is down. Hence, no reports are not visible. We're investigating the Tableau Failure.</p>tag:status.punchh.com,2005:Incident/202246152024-03-21T03:15:56-07:002024-03-21T03:15:56-07:00Planned Database Maintenance - (Partial Downtime in data pipelines)<p><small>Mar <var data-var='date'>21</var>, <var data-var='time'>03:15</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>21</var>, <var data-var='time'>00:15</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>05:42</var> PDT</small><br><strong>Scheduled</strong> - We would like to notify you that scheduled maintenance for our database systems is planned for Thursday, March 14th, at 00:15 AM Pacific time. During this maintenance window, we anticipate the need to temporarily shut down all data pipelines 3 hours.<br /><br />Please note that no downtime is expected for Applications, APIs, and Campaigns, etc. <br /><br />We will endeavour to keep the interruption as brief as possible. We appreciate your patience and thank you for your cooperation.!!</p>tag:status.punchh.com,2005:Incident/203074372024-03-20T02:00:56-07:002024-03-20T02:00:56-07:00Punchh Platform Hotfix Release Deployment - No Downtime<p><small>Mar <var data-var='date'>20</var>, <var data-var='time'>02:00</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>20</var>, <var data-var='time'>01:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>19</var>, <var data-var='time'>23:02</var> PDT</small><br><strong>Scheduled</strong> - Punchh Ops team will be deploying a new Hotfix release that will address functional and stability issues. This release will not cause any downtime during deployment.</p>tag:status.punchh.com,2005:Incident/202380522024-03-16T14:00:56-07:002024-03-16T14:00:56-07:00Third Party Service Maintenance - AWS Databricks<p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>14:00</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>11:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>15:30</var> PDT</small><br><strong>Scheduled</strong> - AWS Databricks has a scheduled outage during this period which will delay downstream data processing. We will update this page when the service is available again.</p>tag:status.punchh.com,2005:Incident/202730382024-03-15T15:10:56-07:002024-03-15T15:10:56-07:00Punchh Platform Hotfix Release Deployment - No Downtime<p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>15:10</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>14:10</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>14:09</var> PDT</small><br><strong>Scheduled</strong> - Punchh Ops team will be deploying a new Hotfix release that will address functional and stability issues. This release will not cause any downtime during deployment.</p>tag:status.punchh.com,2005:Incident/202407662024-03-14T02:25:56-07:002024-03-14T02:25:56-07:00Unplanned Infrastructure Maintenance (No Downtime)<p><small>Mar <var data-var='date'>14</var>, <var data-var='time'>02:25</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>14</var>, <var data-var='time'>00:25</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>14</var>, <var data-var='time'>00:20</var> PDT</small><br><strong>Scheduled</strong> - We would like to notify you that scheduled maintenance for our database systems is planned for Thursday, March 14th, at 00:25 AM Pacific time. <br /><br />During this maintenance window, we would be enhancing our Databases storage throughput, we don't anticipate any downtime during this maintenance.<br /><br />We will endeavour to keep the interruption as brief as possible. We appreciate your patience and thank you for your cooperation.!!</p>tag:status.punchh.com,2005:Incident/202290932024-03-14T02:00:57-07:002024-03-14T02:00:57-07:00Planned Infrastructure Maintenance - No Downtime Expected<p><small>Mar <var data-var='date'>14</var>, <var data-var='time'>02:00</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>14</var>, <var data-var='time'>01:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>16:04</var> PDT</small><br><strong>Scheduled</strong> - We are doing software upgrades on our application cluster.</p>tag:status.punchh.com,2005:Incident/202379082024-03-14T01:14:03-07:002024-03-14T01:14:03-07:00Data Pipeline interrupted<p><small>Mar <var data-var='date'>14</var>, <var data-var='time'>01:14</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>16:36</var> PDT</small><br><strong>Update</strong> - We have identified the issue and performed mitigation. It may take some time for the accumulated data to catch up.</p><p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>12:00</var> PDT</small><br><strong>Investigating</strong> - We are experiencing delayed data synchronization to our Data Lake as a result of an issue with one of our third-party vendors. As a result, campaign segmentation and campaign sending will be delayed, and other Punchh services that rely on the data lake will show increased data lag. We are working on a fix and will provide updates regularly.</p>tag:status.punchh.com,2005:Incident/201979762024-03-12T02:00:56-07:002024-03-12T02:00:56-07:00Planned Infrastructure Maintenance - No Downtime Expected<p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>02:00</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>01:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'> 8</var>, <var data-var='time'>16:54</var> PST</small><br><strong>Scheduled</strong> - We are doing software upgrades on our application cluster.</p>tag:status.punchh.com,2005:Incident/201757072024-03-06T05:10:56-08:002024-03-06T05:10:56-08:00Punchh Platform Hotfix Release Deployment - No Downtime<p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>05:10</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>04:50</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>04:49</var> PST</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.punchh.com,2005:Incident/201546772024-03-06T01:00:58-08:002024-03-06T01:00:58-08:00Punchh Platform Scheduled Release - No Downtime<p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>01:00</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>00:00</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'> 3</var>, <var data-var='time'>23:33</var> PST</small><br><strong>Scheduled</strong> - We will be undergoing scheduled planned release with No Downtime expected.</p>tag:status.punchh.com,2005:Incident/201001602024-02-28T16:45:57-08:002024-02-28T16:45:57-08:00Databricks Cluster Incident<p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>16:45</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>11:31</var> PST</small><br><strong>Identified</strong> - We are currently experiencing an outage of Databricks in the US East-1 Region. <br /><br />This outage is impacting all Databricks customers and is not specific to Punchh. <br /><br />As a result of this outage, we are experiencing campaign failures and missing dashboard statistics. We are reviewing mitigation options while we await further information from Databricks. <br /><br />The Databricks outage can be tracked in more detail here:<br />https://status.databricks.com/pages/incident/5cf02dde58a00904bda41926/65df815ede5f4405308cd082</p>tag:status.punchh.com,2005:Incident/200240812024-02-19T16:07:58-08:002024-02-19T16:07:58-08:00API Processing Delays<p><small>Feb <var data-var='date'>19</var>, <var data-var='time'>16:07</var> PST</small><br><strong>Resolved</strong> - API processing has been restored to normal service levels</p><p><small>Feb <var data-var='date'>19</var>, <var data-var='time'>15:03</var> PST</small><br><strong>Monitoring</strong> - Issue has been identified and remediated - API processing is returning to normal levels, we are currently monitoring.</p><p><small>Feb <var data-var='date'>19</var>, <var data-var='time'>11:45</var> PST</small><br><strong>Investigating</strong> - We are currently experiencing API processing delays. We are investigating the issue and will it resolved in the timely manner.</p>tag:status.punchh.com,2005:Incident/199820122024-02-16T03:15:58-08:002024-02-16T03:15:58-08:00Planned Database Maintenance - (Partial Downtime in data pipelines)<p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>03:15</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>00:15</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>00:47</var> PST</small><br><strong>Scheduled</strong> - We would like to notify you that scheduled maintenance for our database systems is planned for Friday, February 16th, at 00:15 AM Pacific time. During this maintenance window, we anticipate the need to temporarily shut down all legacy data pipelines (v 1.5) for 3 hours. <br /><br />Please note that no downtime is expected for Applications, APIs, and Campaigns, etc. <br /><br />We will endeavour to keep the interruption as brief as possible. We appreciate your patience and thank you for your cooperation.!!</p>tag:status.punchh.com,2005:Incident/199649002024-02-12T01:06:46-08:002024-02-12T01:06:46-08:00Punchh Platform Hotfix Release Deployment - No Downtime<p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>01:06</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>01:01</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>00:28</var> PST</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.punchh.com,2005:Incident/199240372024-02-07T00:30:56-08:002024-02-07T00:30:56-08:00Punchh Platform Scheduled Release - Downtime Expected<p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>00:30</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>23:30</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>10:31</var> PST</small><br><strong>Scheduled</strong> - On Tuesday 6 February at 11:30PM Pacific time we are deploying a new release of the Punchh software. This release contains important upgrades to improve our platform. Please see https://support.punchh.com/s/article/Punchh-Platform-Release-Notes-6-February-2024-Deployment for release notes.<br />We anticipate needing shut down all services for a period of 30-45 minutes in order to complete the release. We will endeavor to keep the interruption as brief as possible.<br />We appreciate your patience and thank you for your cooperation.</p>tag:status.punchh.com,2005:Incident/199214442024-02-06T05:11:33-08:002024-02-06T05:17:50-08:00Mothership Outage due DB Lock<p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>05:11</var> PST</small><br><strong>Resolved</strong> - We had an outage on Mothership's API. The issue occurred due to the Database Lock applied on RDS.<br />Time Duration: 06 Feb 2024 03:46 AM to 03:49 AM PDT</p>tag:status.punchh.com,2005:Incident/196883792024-02-06T03:30:01-08:002024-02-06T03:30:01-08:00Planned Database Maintenance - No Downtime Expected<p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>03:30</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>00:30</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>05:56</var> PST</small><br><strong>Scheduled</strong> - We want to notify you about an upcoming scheduled maintenance activity aimed at upgrading our database storage infrastructure. This upgrade is a crucial step toward ensuring improved performance, scalability, and reliability of our systems.</p>tag:status.punchh.com,2005:Incident/199132772024-02-05T05:30:00-08:002024-02-05T07:31:05-08:00High response time in API's<p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>05:30</var> PST</small><br><strong>Resolved</strong> - We had an incident of High response time in all APIs and experienced API Service performance issues caused by heavy database usage. We are investigating the issue and hope to resolve this problem promptly.</p>tag:status.punchh.com,2005:Incident/196883702024-01-30T03:30:24-08:002024-01-30T03:30:24-08:00Planned Database Maintenance - No Downtime Expected<p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>03:30</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>00:30</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>05:55</var> PST</small><br><strong>Scheduled</strong> - We want to notify you about an upcoming scheduled maintenance activity aimed at upgrading our database storage infrastructure. This upgrade is a crucial step toward ensuring improved performance, scalability, and reliability of our systems.</p>tag:status.punchh.com,2005:Incident/198364092024-01-26T03:31:40-08:002024-01-26T03:31:40-08:00Reports are not available on Dashboard<p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>03:31</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>03:31</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>03:26</var> PST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>01:39</var> PST</small><br><strong>Investigating</strong> - We're currently experiencing an issue impacting the availability of reports on the Dashboard. Our team is actively investigating the issue to get it resolved as quickly as possible.</p>tag:status.punchh.com,2005:Incident/198314612024-01-25T12:01:01-08:002024-01-25T12:01:01-08:00Hotfix Deployment - No Downtime<p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>12:01</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>11:00</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>09:59</var> PST</small><br><strong>Scheduled</strong> - Fix the replication lag in our systems, that caused by some background worker processes.</p>tag:status.punchh.com,2005:Incident/196760202024-01-11T02:40:35-08:002024-01-11T02:40:35-08:00Punchh Platform Hotfix Release Deployment - No Downtime<p><small>Jan <var data-var='date'>11</var>, <var data-var='time'>02:40</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>11</var>, <var data-var='time'>01:40</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>11</var>, <var data-var='time'>01:34</var> PST</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>