Last year devastating floods hit the entire Coromandel Coast region of South India. The union territory of Puducherry and Tamil Nadu were particularly hard-hit. More than 500 people were killed and over 18 lakh (1.8 million) people were displaced. It is estimated that these floods have caused economic loss ranging from nearly Rs. 150 billion (US$2 billion) to over Rs. 1 trillion (US$15 billion), among the costliest natural disasters of the year.
Chennai city saw the worst downpour of the year from 30th November to 2nd December’15 when the city received 50 cm of rainfall in less than 24 hours - 160% more than the average rainfall in the city during monsoon period.
On 30th Nov’15, Mr. Mohan Kumar Subramanian (Head Infrastructure - Data Centre, Chennai) received a call from the Data Center that State Electricity Board (‘EB’) power had failed and he knew that something major has happened.By the next afternoon, the entire city was flooded and 60% of the city was without power.
On December 2, 2016, Chennai was officially declared as a disaster area by the government.
Meanwhile our team at Siruseri SIPCOT Data Center(‘DC’) was struggling to keep the building free from water stagnation. It was faced with two major problems – the water pouring in continuously and the depleting stock of diesel, which was essential to run the DGs.
It is important to understand that the DC, while running normal operations, requires upwards of 35 thousand liters of diesel per day to function without EB power. Unknown to Mohan at that time, EB power would not return for over 7 days. The fact that most roads were submerged and traffic and vehicle movement had come to a near standstill were grave compounding factors. By the end of the second day, all petrol pumps in Chennai would have run out of fuel, there would be very limited supplies of fuel coming into Chennai, and the roads leading to Chennai would have been flooded leaving the city, and the DC, isolated. The few diesel tankers that managed to make their way into the city were routed to hospitals and emergency services, and thereafter, even if Mohan could procure some left over fuel, the road to the DC was completely submerged which meant that a diesel tanker could not have made its way to the DC.Lack of fuel had also started to cause the general public to grow angry and restless, leading to protests at several petrol pumps and public anger towards diverting of fuel meant for petrol pumps to private establishments.
In these circumstances, with other DC’s in the city shutting down, no one could have faulted Mohan for giving up and acceding to the circumstances. But he did not. By the time EB power would resume, Mohan and his team would have procured, in these extremely trying circumstances, more than 2.8 lakh liters Diesel for the DC, and would have spent over 2000 man hours manually pumping out water from roads, basements and surrounding areas to help manage everything. In retrospect, it seems almost impossible to have done what Mohan did. Beyond a point, no amount of Business Continuity Planning and Disaster Recover Planning could help in such dire circumstances and one of the most important take always for Nxtra from the entire incident is that, having BCMS and DR plans is good, but having committed and resourceful people in the right places is invaluable. We realized that no BCMS or DR plan ever talks about the most important aspect of any disaster- the people who manage it and how long and how hard those people are willing to battle impossible circumstances, against all odds, in a situation where no one would have blamed them for giving up.
In addition to, and more importantly than, the operation of the DC itself, Mohan was also responsible for the safety and security of the 35 individuals who were helping run the DC during those hard times and also the safety and security of the families of the 35 individuals who had chosen to stay at the DC to help run operations rather than return home to their families. This meant that providing food, shelter and water for the 35 employees, and also ensuring that rescue workers / friends /colleagues around the city reached the families of these 35 individuals and took care of them.
Compounding problems included the following factors;
- Families had to be provided with adequate food,drinking water and medicines. Many families had babies and small children who required special care and attention.
- Communications to almost every part of the city started to fail leaving the families with no method of contacting Mohan in case of any emergency.
- Even in case of emergency in any family, travel to many parts of the city was virtually impossible so Mohan had to each out to the army and other emergency services to provide assistance.
- In the DC itself, there was no provision for having 35 individuals sleep over. There were no beds, mattresses, etc. Food and drinking water was scarce. Even toilet facilities were insufficient to meet the daily needs of so many individuals.
Food & portable water was sourced for more than 35 individuals, including employees, vendors and customers.Diesel was procured in parts from other districts as the city hospitals and other essential services were already short of diesel for generators. An essential activity of running DGs in turn was scheduled, which enabled us to provide backup for three days over and above the maximum capacity.
Even as the rains stopped on December 2nd, the electrical supply to the SIPCOT area was not restored as the electric substation was still under water and the panels were still wet. The team provided assistance by way of halogen lamps and water pumps to bring up the substation.
By the evening of December 4th, rural power supply was up and by December 5th afternoon the power supply to the Data Center and the neighboring buildings was restored.
We owe the success of our disaster mitigation plans and processes, at such difficult times, to a relentless and selfless team. With their families back home, and no assurance of complete safety, they undertook this challenging task and dedicated themselves to achieving it even as the world around them was crumbling.
At the end of the day, they were able to attain zero water stagnation inside DC, 100% uptime of business during crisis, and provision of food, water and other essential amenities to the team members,who strived to bring up the electric substation back online.