[Resolved] - IST Data Center Outage

Update 9/19/2015 12:00pm

All affected departmental websites (ME, MSE, NUC), research/student group websites, and Calnet authentication are now back in service.

We will continue tracking the IST system status, and will provide updates if any anomalies appear again.

--------------------------------

Due to a fire/overheating issue at the IST Data Center, most major Campus IT applications and services are currently unavailable. 

The affected IT services include:
- All departmental web sites hosted at the Data Center, including ME, NUC, MSE.
- All research and student groups web sites serviced by DECF (except Wordpress sites that have been migrated to WP Engine).
- Most Campus IT services, including BearFacts, bCourses, and Calnet authentication.

We have already filed a ticket with IST.  The IST team is working on assessing the impact of this event.  Additional status updates will be posted on the IST System Status site:

http://systemstatus.berkeley.edu

*NOTE: Due to the Calnet authentication outage, bMail can only be accessed by users whose browsers have cached their Calnet credentials, and by users who are accessing via an email client (e.g. Thunderbird, or any mobile devices).

[Resolved] - File Server Maintenance

Update Monday 11/10 9:50am:

The file server array rebuild is now complete.  All affected DECF services are now operations.  Daily backup will be suspended for 2 days as we monitor the operational status of the file server.  Please email consult at newton.berkeley.edu if you notice any anomalies.

--------------------------------------

Update Sunday 11/9 8:47am:

The rebuild of the first drive is complete.  The rebuild process for the second drive has begun.

---------------------------------------
Update Saturday 11/8 10:48pm:

One of the two failed drives was replaced and the array rebuilding was started on schedule at 12:00am this morning.  However, the rebuild rate is currently slower than expected.  We will replace and start the rebuild of the second failed drive as soon as the first one is completed.

---------------------------------------

We will be performing an emergency maintenance on the DECF file server this weekend starting at 12am Saturday 11/8.  We will be replacing 2 failing hard drives on the file server.  We expect the maintenance and rebuild to be finished within 48 hours.

The following services will be affected:

1. Access to personal files and home directories, including personal ~ webpages.
2. Access to 1111 lab machines and Archipelagos clusters.
3. Access to DECF software.

Updates to the progress will be posted on our DECF status blog:
http://decf-status.blogspot.com/

If you have any questions, please email consult (at) newton.berkeley.edu.

[Resolved] - Network Outage - Sutardja Dai Zone

Outage resolved.

--------------------

A Campus network router crashed this morning around 11:00am, which is affecting network and phone connectivity in the Sutardja Dai zone (including Etcheverry Hall).  IST is currently working on identifying the source of the outage and restoring services.  More information can be found on IST System Status page:

http://systemstatus.berkeley.edu

[Resolved] - File Server Maintenance

Maintenance completed.

-----------------------------

We will be taking the file server down to replace a failing hard drive.  The projected completion time for the maintenance should be within 24 hours.  Access to personal files and to DECF machines (1111, 1171, Archipelagos, Reindeer) will be affected.

Web services and web directories access should not be affected.  If you notice anything out of the ordinary regarding web access. Please contact consult.

For other questions or concerns, please email consult (at) newton.berkeley.edu.

[Resolved] - Emergency File Server Maintenance

Update, Monday 2/3 6:20am:

The RAID rebuild is done, and the file server is now back online.  All DECF operations have returned to normal.  Please email consult@newton if you notice any irregularities.

------------------------------------

Update, Sunday 2/2 11:40am:

The RAID array is still being rebuilt.  The rebuild progress is much slower than anticipated.  The progress as of this writing is at 15%.  We don't currently have an estimated time for the rebuild.

------------------------------------

Update, Sunday 2/2 4:00am:

The bad disk has been replaced.  We are currently rebuilding the RAID array.  This process usually takes a few hours.  We will provide another update when the rebuild is done and when the file server is up again.

------------------------------------

There is a failed disk on our file server.  We will be shutting down the file server at 3am on Sunday 2/2 in order to replace the bad disk.  Access to personal files and computing labs and cluster are affected.  Further updates will be posted here.

Web sites are not affected.  However, access to kepler will be limited until the file server maintenance is done.

[Resolved] Research/Student Groups Web Server Down

Update Oct 8:

All web directories have been mounted back on kepler.  This issue is now resolved.

-FC

-----------------------------

Update 3:19pm, Oct 4:

We have finished rebuilding the web server that hosts the research/student groups, and have restored all web sites from backup (dated 9/30).  The cause of the disk corruption was associated with the power outage we had on Monday.  The IST Datacenter team was able to quickly re-allocate a new virtual machine provision for us.

Currently, the web directories are not mounted on kepler.  That means the web sites will only be readable.  We wil continue to monitor the web server health over the weekend to make sure everything is running without problems.  We will re-mount the web directories on kepler and re-initiate the backup scheduler on Tuesday if nothing is amiss.

Special arrangement can be made if you have an urgent need to update your site over the weekend.

Please email consult@newton if you have any questions or concern.

-FC

----------------------------

Update 7:40pm, Oct 3:

We have opened a ticket with the IST Datacenter team, and are waiting for a new virtual machine provision.  We will be rebuilding the server once the virtual machine is in place.  We will be restoring all web sites from backup once the server is rebuilt.

-FC

-----------------------------

The virtual machine that hosts the research and student groups web sites is currently down due to a corrupted disk.  We have contacted the IST Datacenter team to verify and rectify this issue.  Any updates will be posted here.

-FC

[Resolved] - 2013-09-30 Campus-wide Power Outage

Update 12:15pm, Oct 1:

The network problem in 1111 Etch has been resolved.  All DECF services are operational.

FC

----------------------------

Update 8:45am, Oct 1:

All DECF services except the 1111 lab are back to normal operation.  Please email consult@newton if you notice anything out of the ordinary.

FC

---------------------

Update 11:53pm:

The departmental web sites for ME, MSE and BioE, and research group web sites hosted at DECF are back in partial service.  We are still waiting for power to be restored in Etcheverry Hall.

FC

---------------------

Update 10:30pm:

Power is still out on most part of Campus.  IST reported that power has restored at the Datacenter, and that temperature was beginning to drop to normal operation range.  We are waiting for IST to completely restore its service at the Datacenter before we can get the ME, MSE, and BioE departmental websites back online.  Other DECF services remain offline pending the power restoration at Etcheverry Hall.

FC
---------------------

Original 7:20pm:

All DECF services are currently unavailable due to the Campus-wide power outage.  All departmental websites are currently not accessible.  We will be working to bring our services back online as soon as the power outage issue is resolved.

Please also see IST's status page for additional information:
systemstatus.berkeley.edu

FC

[Resolved] - DECF Network Outage

Update: 9/17/2013:

All DECF services are now back to normal.  We are still experiencing partial network connectivity to Campus' hosted services (including Calmail, bMail, and Calnet authentication).  We have working with Campus' Shared Services to resolve this issue.

---------------------------


Due to the widespread power outage this morning, the DECF network is still down pending a resolution from CNS. All DECF services are affected at this time including Consult's email. If you have any questions or concern. Please contact Finsen directly at finsen "at"  berkeley.edu.

End of Semester Systems Maintenance 5/20 - 5/24

Update 2013/05/20:

System maintenance is complete for all client machines (1111, 1171, Archipelagos, Reindeer).

------------------------------------  
 
All DECF client machines (1111,1171, Archipelagos, and Reindeer) will be unavailable from 5/20 - 5/24 for systems maintenance.

If you are running jobs remotely, please plan accordingly to have your
jobs finish before 5/20 to avoid jobs being terminated.

Machines that ready to be used will be listed here.

If you have any questions, please contact consult@newton.berkeley.edu

File Server Outage (Fixed)

Updated 1/4/2013:

The failed disk has been replaced.  We have finished rebuilding the file server.  All DECF services are back to normal.  Please email consult@newton if you encounter any issues.

Thanks,

Finsen

--------------------------------------------------------------------------------------------------------

The DECF file server has a failed drive. We have to take the server down in order to replace the failed drive and rebuild the RAID array. DECF clusters usages (Reindeer, Archipelagos, 1111, and 1171) are affected.

However, web services are not affected.

Departmental Web Server Down

Update: 1:30pm:

All department websites are now back online.

----------------------------------

Our Departmental web server has been down since 9:30am.  The affected departments are ME, MSE, and BioE.  It is suspected to be due to a fried motherboard.  We are working on bringing the server back online.

Most research and students groups web sites are not affected.

DECF Windows Clients Upgrade

 Update:

All client machines and servers are upgraded.

-FC

----------------
We will be upgrading our Windows machines in 1111 Etch during Spring Break (3/26 - 3/29). The Windows machines will be formatted and upgraded to Windows 7. All files on the local computers will be wiped. After the upgrade, you will be able to log onto the Windows computers with your Calnet credentials.

If you are an active DECF Windows users, please do the following prior to our upgrade:

1. Backup your files stored on the local computers to your Windows home directory. You should be able to find your home directory within "My Computer".
2. If you are still actively using the 1111 Etch Windows computers and would like to continue using the DECF Windows machines, respond to consult@newton.berkeley.edu with your DECF user name and your Calnet ID for continuation of your Windows account.

*If you are no longer an active user of the DECF Windows machine, simply disregard this email; we will remove the account for you.

Please email consult@newton.berkeley.edu if you have questions.

Emergnecy File Server Shutdown 4/30

We will be performing an emergency shutdown of the DECF file server tonight 4/30 due to a failing disk. We are currently performing back-end maintenance to ensure files are not lost. We will be transitioning to a back-up file server tonight. During this transition, you will not be able to access your home directories, and that files written will not be saved. The transition is projected to take about 30 minutes. We will post status updates on: http://decf-status.blogspot.com/ Please email consult@newton.berkeley.edu if you have questions.

Emergency Shutdown: Archipelagos & Reindeer Clusters

** Emergency Shutdown: Archipelagos & Reindeer Clusters**

Update 1/18 1:15pm:

PPCS has fixed the chiller issue at Soda Hall. Server temperature back to normal.
Clusters operation resumes.

-FC

---

Update 1/17 4:11pm:

PPCS reported that the chiller serving the DECF server room had
triggered an alarm. PPCS is in the progress of fixing the issue right now.

If you have urgent jobs that need to be run, please use either the 1111 or 1171 Etch machines. Please keep in mind that classes in those labs start on Jan 23.

-FC

------

Update 1/17 1:35pm:

PPCS fixed AC as priority tag A. AC in server room is working
but seems to be performing intermittently. We will slowly bring the clusters
back online and monitor the server room temperature.

-FC

------

Update 1/16 11:57pm:

All machines are shutdown. Fans are on in server room to
bring temperature down. Contacted PPCS station engineer.
They are calling the on-call person to come out to check.

-FC

-----------

1/16 11:45pm

Server room temperature exceeded 90 degrees on 1/16 at
11:40pm. The Archipelagos and Reindeer clusters have
to be shut down as an emergency to protect the machines
and prevent temperature from rising further.

-FC

Networking Maintenance for 3115B Etch on Monday 1/16/2012

We will be performing networking maintenance in 3115B Etch on 1/16
Monday (MLK Day). We will be adding additional tubings, cables,
and ethernet drops to that room to improve networking capability.

If you are one of the students in that room, please be advised that
ethernet connections to/from that room will be unavailable that day.

If you have any questions or concerns, please email us at
consult@nuc.berkeley.edu.

End-of-Semester Maintenance

All DECF client machines (1111,1171, Archipelagos, and Reindeer) will be
unavailable from 12/19/2011 to 12/23/2011 for systems upgrade and
maintenance.

If you are running jobs remotely, please plan accordingly to have your
jobs finish before 12/19 to avoid jobs being terminated.

Machines that are upgraded and ready to use will be listed on our web
site at the DECF Systems Status page: http://decf-status.blogspot.com/

If you have any questions, please contact consult@newton.berkeley.edu

------
Updates 2011/12/21:

All client machines are updated and ready to be used. If you encounter any problems with the machines, please notify consult@newton.berkeley.edu.

Networking Maintenance for 3115B Etch on Friday 11/11/2011

We will be performing networking maintenance in 3115B Etch on 11/11 Friday
(Veterans Day).

We will be adding additional tubings, cables, and ethernet
drops to that room to improve networking capability.

If you are one of the students in that room, please be advised that
ethernet connections to/from that room will be unavailable that day.

If you have any questions or concerns, please email us at
consult@nuc.berkeley.edu.

File Server Migration 8/27 Saturday 9PM

We will be migrating the file server to a new machine on Saturday 8/27 at 9PM. This migration will allow us to perform necessary maintenance and upgrade on the old file server.

During the migration, the file server will be unavailable. You will NOT be able to:

• Access your accounts,
  
• Access your files stored in your accounts,
• Start any computational jobs on our clusters.

If you have any questions, please email consult@newton.berkeley.edu

www.nuc migration back 8/19/2011

The NE web/file server OS has been upgraded. I will be moving the NE website back to the original server tonight Friday (8/19/2011).

On Monday, I will go to all staff machines to change the Samba mapping.

www.nuc migration 8/10/2011

/home1 was restored on mail.nuc yesterday and everything has been moved over to the temporary server as of this afternoon.

Tonight, I will be moving the NUC website database over. mail.nuc will be reinstalled tomorrow.

Please ssh to www.nuc.berkeley.edu from now on.

If you encounter any problems w/ the website or have access issues, please email consult