Connectivity in CICESE: From the First Workstations to 10 Gbps
Ensenada, Mexico, Nov. 26, 2012 -- As it gets ready to celebrate its 40th anniversary in 2013, Mexico’s Center for Scientific Research and Higher Education (CICESE) is once again at the forefront of high-performance computing on a global scale thanks to its 10 gigabits per second (Gbps) connectivity. “Using cutting-edge computational resources to support scientific research has been a constant in the history of CICESE,” reports the center's Roberto Ulises Cruz, in a new history of the center. What follows is an English-language excerpt from that report, tracing CICESE’s long collaboration with UC San Diego and Calit2:
In 1989, CICESE staff designed and installed the first fiber-optic backbone in a Mexican academic institution.
|
It’s worth noting that these network technologies developed at CICESE in the late 1980s and early 1990s, were on par with those that were being developed in the U.S. and Europe at that time.
Thanks to this, the next natural step was the acquisition of its own supercomputers. In 1997 there was only one academic supercomputer in Mexico; it was at UNAM. CICESE bought the second – a Silicon Graphics Origin 2000 with 10 R10000 195 MHz processors, each with 4 MB of cache and a total of 1GB of RAM. In 2002 a second was purchased: a Sun Fire 4800 Ultra Sparc III with eight 900MHz processors and five Sun Blade 1000 workstations that, when connected through a PC router with two interfaces and a switch, gave rise to CICESE’s first cluster, called Tribes.
Meanwhile, when the University Corporation for Development of the Internet (CUDI) was created in April 1999 as the entity responsible for operating [Mexico’s] telecommunications infrastructure and high-speed transmission lines to support research and education, and to enable the development of applications that would drive the next-generation Internet, Internet2 began to establish itself in Mexico starting in the year 2000.
The era of high-performance computing had arrived.
|
What resulted was the need to share resources in order to optimize them, in other words, to work collaboratively. How? In grids: networks linked together in networks.
In 2004, CICESE and UNAM, the Autonomous Metropolitan University, the University of Guadalajara, and CINVESTAV launched an initiative to integrate the Mexican Academic Grid (GRAMA) in an inter-institutional demonstration network to share supercomputing resources that would also serve to integrate them with international infrastructures.
The plan was to have at the end of the project a hardware infrastructure consisting of at least one high performance computing capability at each institution connected to Internet2; the software infrastructure to manage access and shared use of the hardware; and some applications running on the grid.
With this experience, in June 2005, CICESE was invited to participate in the Pacific Rim Applications and Grid Middleware Assembly (PRAGMA), an association of universities and research centers from different countries around the Pacific Rim.
As this was happening, in 2002 the National Science Foundation (NSF) of the United States gave the California Institute for Telecommunications and Information Technology (Calit2), headquartered in San Diego, its largest computer grant in recent years, for the project known as the OptIPuter.
The goal: to allow the researchers who generate terabytes and petabytes of data… to be able to visualize, analyze and correlate data interactively at multiple sites connected over optical networks.
The word OptIPuter refers to the use of optical networks, Internet Protocol (IP), as well as computer storage, processing and display technologies, and it was a visionary infrastructure to permit the close coupling of computational resources and parallel optical networks, using the Internet Protocol communication mechanisms.
Almost from its birth, Calit2 pushed the idea of involving Mexican institutions in the OptIPuter project. During 2003 and 2004, several meetings took place with executives of CUDI, CONACYT and CICESE. In July 2005, as part of the binational Governors Conference held in Torreon, there was already talk of the benefits of a link which would permit greater knowledge and an ability, for example, to respond to wildfires, floods and earthquakes along the border between the two countries. At the CUDI meeting held in fall 2006 in San Luis Potosi, Dr. Larry Smarr, director of Calit2 and head of the OptIPuter project, once again repeated that he was strongly interested in having Mexican institutions participate in the global project.
Thus, while CICESE was building a second computer cluster called Cataviña, mainly for computer modeling of ocean and atmospheric circulation, the institute signed a cooperation agreement with Calit2 that allowed the center in 2007 – during the commemoration of the 35th anniversary of the creation of CICESE – to begin building the first OptIPortal in Latin America, with a visualization wall made up of 12 monitors or mosaic panels, four displays across and three vertically, with limited connectivity – barely 34 megabits per second. The system went into operation in 2008.
Also in 2007, after the approval in a special session of CONACYT of a proposal to create national laboratories for scientific and technological infrastructure in specific areas, CICESE took the initiative to create the National Supercomputing Grids Laboratory, together with 10 Mexican institutions , mainly in the center and north of the country: the Universities of Guadalajara, Colima, Sonora, the Autonomous University of Nuevo Leon and of the Americas (Puebla), the National Polytechnic Institute, the Institute of Technology and Higher Studies of Monterrey, the Potosi Institute of Scientific and Technological Research (IPICYT) and CUDI.
This proposal would complement two other similar efforts: the Metropolitan Delta for High Performance Computing, powered by UNAM, UAM and CINVESTAV in Mexico City, and the National Supercomputer Center (CNS), which was consolidated in the same year with IPICYT, with funds coming from the same grant.
It is easy to understand that CONACYT’s policy at the time, driven largely by CUDI, was to support these initiatives and to create networks and research groups which could tackle problems from different perspectives, proposing more viable and multidisciplinary solutions by region, while bringing together scattered computing capabilities and supporting collaborative work.
But in CICESE, connectivity was a limiting factor, especially after the NSF had fulfilled its commitment to connect San Diego to Tijuana with a one-gigabyte-per-second link -- a link that could not be extended from Tijuana to Ensenada because of a lack of resources. The gateway to the most powerful computing infrastructure in the world remained 100 kilometers away from CICESE.
However, a year later, in 2008, thanks to a large project contracted with [the state oil company] PEMEX, CICESE got another cluster specifically to model currents in the Gulf of Mexico.
In mid-2008, CICESE’s high-performance computer infrastructure, including two supercomputers, two clusters, proposals to consolidate the National Grid Laboratory and the first OptIPortal in Mexico for visualization projects, all with Internet2 access, pushed the research center into the ranks of UNAM, UAM and IPICYT as the institutions with the greatest computing capacity in the country.
Perhaps it was this positioning, or the desire and need for researchers to be in contact with their peers, or the advantage of being close to California, where institutions such as Calit2 and UCSD had demonstrated time and again a desire to work with CICESE, or maybe some other unknown element, that pushed the computing center to the forefront of global development. Taken together, these reasons allowed CICESE, between 2009 and 2011, to continue collaborating with CONACYT to allocate the bi-national resources that were needed to connect this superhighway [between Calit2 and CICESE] at 10 gigabits per second.
And therein lies a reality: given the proximity and scope of research being done on networks at both UCSD and CICESE, collaborations based on networks at both institutions looked very promising. Disciplines that are data intensive and depend on high bandwidth, ranging from oceanography to disaster response, distributed learning and resource sharing of rich-media resources among students on both sides of the border, and the development of software tools that support network-based partnerships – these are just some of the focus areas.
Specifically, to enable this link required two things: 1) to improve the network equipment of the Corporation for Education Network Initiatives in California (CENIC), through an upgrade to 10 gigabits per second in the Point of Presence (POP) Bestel in Tijuana, pushing it from 1 to 10 Gb/s connectivity on the American side, which was achieved in February 2012; and 2) to enable the 10 Gb/s link between Tijuana and Ensenada, and in particular, to enable improvements in the "last mile" to benefit not only CICESE and the OptIPuter network, but also the local campuses of the UNAM and UABC which, at the time, were expecting to be connected at this speed. The first "light" transmitted through the optical link from Calit2 to Tijuana and CICESE occurred in early March 2012.
This bi-national U.S.-Mexico advanced networking project, in which Calit2, UCSD, NSF, CICESE, CONACYT and CUDI were participating, won the Innovations in Networking Award in the category of High Performance research Applications for 2012, handed out at the CENIC Annual Meeting in Palo Alto, California. The partners were cited for the role they played in enabling scientific and educational collaborations across the border between the two countries.
The official opening of the link took place against the backdrop of the Spring Meeting of CUDI 2012, held May 23-25 in Ensenada, and it represents, at both the medium and long term, an important step in the development of scientific research in Mexico.
Or as CICESE’s Director General, Federico Graef Ziehl, said in his inaugural talk to CUDI, it "represents a new way of doing science, because it creates new networks of collaboration between researchers and students that will let them devise solutions to problems that affect our lives as a nation and for humanity at large."
Yes, but what does it mean for CICESE? New and better applications, pointed out Jose Lozano, head of the Computing Department at CICESE. And Salvador Castañeda, a leader in the Telematics section, noted that "we will be the point of presence in CUDI in the northwest, and this is where the main research centers and universities will come who want to connect to Internet2. Not everyone will be able to be on the network, obviously. How fast? As fast as the ability of the user to hook up their computers to the network."
And before thinking about connecting at 100 Gb/s, which is the next step for CICESE, it’s first important to take advantage of the opportunity to access any laboratory or the equipment of any academic institution located in the Pacific Rim region or anywhere else in the world, that has a 10 Gb/s connection, interacting through other OptIPortals.
Interacting how? By using videoconferencing, sending with just one click high-resolution images and sharing media resources in a natural, transparent and immediate way.
This opens up a myriad of opportunities that were previously constrained by bandwidth, says Salvador Castañeda. If you already have telepresence, he said, you can extend it to any part of the world. The collaboration between CICESE and these other centers on the map (as he points to a world map displayed on the OptIPortal): “It’s completely instantaneous.”
Jose Lozano adds: "The same goes for projects that require large computer storage resources. If, for some reason, it is sometimes difficult to acquire equipment, through this link you can contact any storage service outside of CICESE, such as up at the San Diego Supercomputer Center at UCSD; you can have everything stored there, and transfer it to Ensenada much faster than if it were located on an external [disk] drive."
To get an idea of how fast the link is, a reader could take two DVDs, such as two movies, and drop them on his or her PC or laptop in one second, and it’s there. It’s that fast.
How good is the connectivity map of Mexico for education and research applications?
“It is very poor,” said Salvador Castaneda. Basically there are two points of presence at the moment. One crosses the border in Ciudad Juarez with 1 Gb/s from the University of Texas at El Paso (UTEP), which goes as far as UNAM. The other is this Tijuana node, which is already at 10 Gb/s.
And how much has the connection cost?
The director of Telematics at CICESE, Raúl Rivera, explains: "I was told about $80,000 in video displays alone. Besides the equipment and all the connectivity: for the access routers, we’re talking about a cost of about $130,000; security cost another $170 000. Plus, the quality-of-service equipment came in around $250,000. The cost is high because nearly all of [the equipment] is optical, and everything in optical communications is still expensive; it requires high-precision links over long distances, and that's what makes them more expensive."
Salvador Castaneda sees the reaction, and immediately offers up a response that he has made the guiding philosophy at CICESE: "We can say that money does not matter, but the fact is, you're connected to scientific centers of excellence around the world. That is what really matters."
[Written by Roberto Ulises Cruz, CICESE, and translated by Doug Ramsey, Calit2]
Related Links
CICESE
CENIC Innovations in Networking Award 2012
CONACYT
CUDI