DK Panda from Ohio State University gave this talk at the Stanford HPC Conference. “This talk will provide an overview of challenges in accelerating Hadoop, Spark and Memcached on modern HPC clusters. An overview of RDMA-based designs for Hadoop (HDFS, MapReduce, RPC and HBase), Spark, Memcached, Swift, and Kafka using native RDMA support for InfiniBand and RoCE will be presented.”
Big Data Meets HPC – Exploiting HPC Technologies for Accelerating Big Data Processing
March 12, 2018 by
Best Practices – Big Data Acceleration
March 8, 2016 by
“This talk will provide an overview of challenges in accelerating Hadoop, Spark and Memcached on modern HPC clusters. An overview of RDMA-based designs for multiple components of Hadoop (HDFS, MapReduce, RPC and HBase), Spark, and Memcached will be presented. Enhanced designs for these components to exploit in-memory technology and parallel file systems (such as Lustre) will be presented. Benefits of these designs on various cluster configurations using the publicly available RDMA-enabled packages from the OSU HiBD project (http://hibd.cse.ohio-state.edu) will be shown.”
Performance Optimization of Hadoop Using InfiniBand RDMA
October 11, 2014 by
“The Hadoop framework has become the most popular open-source solution for Big Data processing. Traditionally, Hadoop communication calls are implemented over sockets and do not deliver best performance on modern clusters with high-performance interconnects. This talk will examine opportunities and challenges in optimizing performance of Hadoop with Remote DMA (RDMA) support, as available with InfiniBand, RoCE (RDMA over Converged Enhanced Ethernet) and other modern interconnects.”
Mellanox InfiniBand Powers Yahoo! Japan
September 29, 2014 by
Today Mellanox Technologies announced that Yahoo! Japan has deployed Mellanox’s end-to-end FDR 56Gb/s InfiniBand solutions for database acceleration at several data center sites in Japan.
Interview: Mellanox Covers Exascale Computing, the Cloud and Big Data
June 6, 2014 by
“With a novel data analytics approach, Mellanox is providing double performance to data Extract, Transform, Load (ETL) solutions based on Hadoop Map Reduce. The code to enable such performance gains is part of the Hadoop community code. With acceleration of the networking stack, Mellanox is quadrupling the number of clients served on a single Memcached server, a dominating key-value caching service for large scale web applications.”
Mellanox Collaborates with IBM to Speed In-Memory Data Grids
April 30, 2014 by
Today Mellanox announced a collaboration with IBM to deliver a high-performance infrastructure for NoSQL Databases and In-Memory Data Grids.
IBM Goes Deep for Big Data Market with POWER8 Open Server Innovation
April 23, 2014 by
“This is the first truly disruptive advancement in high-end server technology in decades, with radical technology changes and the full support of an open server ecosystem that will seamlessly lead our clients into this world of massive data volumes and complexity,” said Tom Rosamilia, Senior Vice President, IBM Systems and Technology Group. “There no longer is a one-size-fits-all approach to scale out a data center. With our membership in the OpenPOWER Foundation, IBM’s POWER8 processor will become a catalyst for emerging applications and an open innovation platform.”
Video: RDMA Accelerated Big Data Applications
June 21, 2013 by
In this video from the HPC Advisory Council European Conference 2013, Eyal Gutkind from Mellanox presents: RDMA Accelerated Big Data Applications. Download the slides (PDF) and check out more videos at the HPC Advisory Council European Conference Video Gallery.
Video: Accelerating Big Data over RDMA
April 24, 2013 by
In this video from the 2013 Open Fabrics Developer Workshop, Sreev Doddabalapur from Mellanox presents: Accelerating Big Data over RDMA. You can check out more OFA videos at our Open Fabrics Workshop Video Gallery.
Video: Hadoop Acceleration with RDMA
February 8, 2013 by
In this video, Eyal Gutkind from Mellanox presents: Hadoop Acceleration with RDMA. The presentation was recorded at the HPC Advisory Council Stanford Conference 2013. Download the slides (PDF).
