Resolved – Fiber transport issue

As of approximately 5 minutes ago, repairs appear to have been completed on the fiber break affecting subscribers today.   I have re-routed all traffic over primary paths and the bits are flowing freely again. 
I’ll send another update once all of the details are available on the outage, and continue to monitor this circuit for issues.  For now, enjoy full speeds again!
Thanks,
– Jacob

6:30 PM Update – Fiber transport issue

As of 6PM tonight, crews are on-site replacing and re-splicing the damaged sections of fiber.  I am told that this may take a few hours, at least, but has begun.  In the interim, the capacity upgrades to the aforementioned backup link have been completed.  Subscribers will still experience slower than normal service, but it should be functional at least for typical web browsing.  Streaming video will not work satisfactorily until the primary link is restored.
Thanks for your understanding.
– Jacob

4:30PM Update – Fiber transport issue

Word from ATT is that the current service interruption is being caused by a fiber cut in Napa, caused during road construction.  We have not yet received an estimated time of repair though.
A capacity upgrade on the limited capacity backup link that traffic is currently flowing over was ordered around noon today.  That upgrade has been partially completed, and I’m hoping it will be finalized shortly.
Thanks for your understanding.
– Jacob

Update – Fiber transport issue

The problem transport circuit is still down as of noon and we are awaiting updates from the provider.  At 11AM I re-routed traffic over a different path to Fremont.  This backup path is a very low capacity circuit, but I have placed an upgrade order for it that will hopefully be completed shortly.  Subscribers should be up and running, albeit very slowly, as of about 11:15 this morning.  
I’ll send an additional update once we have an ETR from the provider of the primary circuit.
Thanks,
– Jacob

Update – Troubleshooting transport link failure

8:56PM – I’m still on hold with the transport fiber provider’s support department while they investigate.  While on-hold I have manually moved the majority of subscriber traffic to a backup link for a partial service restoration.  Unfortunately, this backup link has about 5% the required capacity to carry the evening traffic load.  I’m working on ordering a temporary capacity increase on this circuit for the duration of this outage.
Most subscribers should have service restored, though speeds will be quite low for now.
More updates to follow.
Thanks, 
– Jacob

Resolved – Connectivity issue between Ukiah and Fremont

Service was fully restored on the affected link between Ukiah and Fremont at 8:37 this evening.  I’m still awaiting followup from the fiber provider on the final resolution, but the bits are flowing freely again!
I’ll post a full report on the outage later this weekend on the website.
Thank you for your patience during this unexpected (and really long) service interruption.  The positive messages received from many subscribers during the outage are very much appreciated.
Have a great rest of the weekend.
-Jacob