Spark资源

Blog source from:  https://github.com/litaotao/litaotao.github.io/blob/master/_posts/new-spark/2016-03-10-spark-resouces-blogs-paper.md

————————————————————————————————————————————–

3. Databricks Blog

• Apache Spark 1.5 DataFrame API Highlights: Date/Time/String Handling, Time Intervals, and UDAFs
• Achieving End-to-end Security for Apache Spark with Databricks

上面两篇是 databricks 出的关于 databricks 专业版的描述，虽然没有从根本上解决问题，但是读起来还是挺有说服力的，哈哈，因为采用了很多很细节的方案。不错不错，各位有在做云产品的，在宣传自己的安全方案时可用参考参考哦。

• Deep Dive into Spark SQL’s Catalyst Optimizer
• A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets
• Understanding your Apache Spark application through visualization
• New Visualizations for Understanding Apache Spark Streaming Applications
• An Introduction to Writing Apache Spark Applications on Databricks
• A Gentle Introduction to Apache Spark on Databricks
• Apache Spark on Databricks for Data Scientists
• Import Notebook Apache Spark on Databricks for Data Engineers
• Structured Streaming In Apache Spark: A new high-level API for streaming
• Spark SQL Supported Syntax
• Combining Machine Learning Frameworks with Apache Spark
• Deep Dive: Memory Management in Apache Spark
• https://databricks.com/blog/2017/07/11/introducing-apache-spark-2-2.html
• https://spark-summit.org/east-2017/events/improving-python-and-spark-performance-and-interoperability/
• http://www.slideshare.net/wesm/high-performance-python-on-apache-spark
• https://www.slideshare.net/SparkSummit/spark-and-online-analytics-spark-summit-east-talky-by-shubham-chopra
• https://www.slideshare.net/SparkSummit/spark-parquet-in-depth-spark-summit-east-talk-by-emily-curtin-and-robbie-strickland
• https://www.slideshare.net/databricks/sparksql-a-compiler-from-queries-to-rdds

4. 文章，博客

• RDD论文英文版
• RDD论文中文版
• An Architecture for Fast and General Data Processing on Large Clusters
• How-to: Tune Your Apache Spark Jobs (Part 1)
• How-to: Tune Your Apache Spark Jobs (Part 2)
• By 0x0fff Spark Misconceptions
• By 0x0fff Spark Architecture
• By 0x0fff Spark DataFrames are faster, aren’t they?
• By 0x0fff Spark Architecture: Shuffle
• By 0x0fff Modern Data Architecture
• By 0x0fff Spark Architecture Talk
• By 0x0fff Apache Spark Future
• By 0x0fff Data Industry Trends
• By 0x0fff Spark Memory Management
• By 0x0fff Spark Architecture Video
• 借助 Redis ，让 Spark 提速 45 倍！
• 量化派基于Hadoop、Spark、Storm的大数据风控架构
• 基于Spark的异构分布式深度学习平台
• 你对Hadoop和Spark生态圈了解有几许？
• Hadoop vs Spark
• 雅虎开源CaffeOnSpark：基于Hadoop/Spark的分布式深度学习
• [2016 上海第二次 spark meetup: 1. spark_meetup.pdf](http://litaotao.github.io/files/1. spark_meetup.pdf)
• [2016 上海第二次 spark meetup: 2. Flink_ An unified stream engine.pdf](http://litaotao.github.io/files/2. Flink_ An unified stream engine.pdf)
• [2016 上海第二次 spark meetup: 3. Spark在计算广告领域的应用实践.pdf](http://litaotao.github.io/files/3. Spark在计算广告领域的应用实践.pdf)
• [2016 上海第二次 spark meetup: 4. splunk_spark.pdf](http://litaotao.github.io/files/4. splunk_spark.pdf)
• 基于Spark的医疗和金融大数据
• Monitoring Spark with Graphite and Grafana
• Spark使用CombineTextInputFormat缓解小文件过多导致Task数目过多的问题
• Databricks Empowers Enterprises to Secure Their Apache Spark Workloads
• Spark配置参数
• Running Spark Python Applications
• How-to: Prepare Your Apache Hadoop Cluster for PySpark Jobs
• Apache Spark’s Hidden REST API
• [SQL, sqlContext, hiveContext]
  • What is the difference between Apache Spark SQLContext vs HiveContext?
  • SQL Window Functions
  • Performance Tuning SQL Queries
  • HIVE-vs-RDBMS
  • Windowing and Analytics Functions
• [Spark Memory Issues]
  • http://www.cnblogs.com/wrencai/p/4231934.html
  • http://stackoverflow.com/questions/32349611/what-should-be-the-optimal-value-for-spark-sql-shuffle-partitions-or-how-do-we-i
  • http://stackoverflow.com/questions/21138751/spark-java-lang-outofmemoryerror-java-heap-space
  • A Beginner’s Guide on Troubleshooting Spark Applications
  • http://www.mincoder.com/article/2381.shtml
• http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-1/
• http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-2/
• http://blog.cloudera.com/blog/2014/05/apache-spark-resource-management-and-yarn-app-models/
• https://spark.apache.org/docs/latest/job-scheduling.html
• https://databricks.com/blog/2015/06/22/understanding-your-spark-application-through-visualization.html
• http://blog.csdn.net/u012684933/article/details/50378725

5. 视频

• YouTube: what is apache spark
• Introduction to Spark Architecture
• Top 5 Mistakes When Writing Spark Applications
• slide Top 5 mistakes when writing Spark applications
• Tuning and Debugging Apache Spark
• slide Tuning and Debugging Apache Spark
• A Deeper Understanding of Spark Internals – Aaron Davidson (Databricks)
• slide A Deeper Understanding of Spark Internals – Aaron Davidson (Databricks)
• Building, Debugging, and Tuning Spark Machine Learning Pipelines – Joseph Bradley (Databricks)
• slide Building, Debugging, and Tuning Spark Machine Learning Pipelines
• Spark DataFrames Simple and Fast Analysis of Structured Data – Michael Armbrust (Databricks)
• slide Spark DataFrames Simple and Fast Analysis of Structured Data – Michael Armbrust (Databricks)
• Spark Tuning for Enterprise System Administrators
• slide Spark Tuning for Enterprise System Administrators
• Structuring Spark: DataFrames, Datasets, and Streaming
• slide Structuring Spark: DataFrames, Datasets, and Streaming
• Spark in Production: Lessons from 100+ Production Users
• slide Spark in Production: Lessons from 100+ Production Users
• Production Spark and Tachyon use Cases
• slide Production Spark and Tachyon use Cases
• SparkUI Visualization
• slide SparkUI Visualization
• Everyday I’m Shuffling – Tips for Writing Better Spark Programs, Strata San Jose 2015
• slide Everyday I’m Shuffling – Tips for Writing Better Spark Programs, Strata San Jose 2015
• Large Scale Distributed Machine Learning on Apache Spark
• Securing your Spark Applications
• slide Securing your Spark Applications
• Building a REST Job Server for Interactive Spark as a Service
• slide Building a REST Job Server for Interactive Spark as a Service
• Exploiting GPUs for Columnar DataFrame Operations
• slide Exploiting GPUs for Columnar DataFrame Operations
• Easy JSON Data Manipulation in Spark – Yin Huai (Databricks)
• slide Easy JSON Data Manipulation in Spark – Yin Huai (Databricks)
• Sparkling: Speculative Partition of Data for Spark Applications – Peilong Li
• slide Sparkling: Speculative Partition of Data for Spark Applications – Peilong Li
• Advanced Spark Internals and Tuning – Reynold Xin
• slide Advanced Spark Internals and Tuning – Reynold Xin
• The Future of Real Time in Spark
• slide The Future of Real Time in Spark
• Spark 2 0
• slide Spark 2 0
• Democratizing Access to Data
• slide Democratizing Access to Data
• Not Your Father’s Database: How to Use Apache Spark Properly in Your Big Data Architecture
• slide Not Your Father’s Database: How to Use Apache Spark Properly in Your Big Data Architecture
• Disrupting Big Data with Apache Spark in the Cloud
• slide Disrupting Big Data with Apache Spark in the Cloud

我一直很欣赏 databricks 出的 video 和 slide，结构非常清晰，这个是其中一个非常好的演讲，里面有很多值得借鉴的地方，特别是当你像别人介绍你的工作，产品的时候。[我有一个感受，很少有人能清晰，有条理的介绍自己正在做的产品，对于一些小众的产品，甚至一些职业的销售也难以做到清晰，简明的叙述。这个 video 和 slide 有很大的参考价值。我自己感觉仔细研究这些 video 和 slide 有时候比看上一两本专业讲销售的书还要管用。]

• Getting The Best Performance With PySpark
• slide Getting The Best Performance With PySpark
• Disrupting Big Data with Apache Spark in the Cloud
• slide Disrupting Big Data with Apache Spark in the Cloud
• Large Scale Deep Learning with TensorFlow
• slide Large Scale Deep Learning with TensorFlow
• 700 Queries Per Second with Updates: Spark As A Real-Time Web Service
• slide 700 Queries Per Second with Updates: Spark As A Real-Time Web Service
• blog 700 SQL Queries per Second in Apache Spark with FiloDB
• Operational Tips for Deploying Spark
• slide Operational Tips for Deploying Spark
• Connecting Python To The Spark Ecosystem
• slide Connecting Python To The Spark Ecosystem
• Livy: A REST Web Service For Apache Spark
• slide Livy: A REST Web Service For Apache Spark
• Understanding Memory Management In Spark For Fun And Profit
• slide Understanding Memory Management In Spark For Fun And Profit
• Deep Dive: Memory Management in Apache Spark
• slide Deep Dive: Memory Management in Apache Spark
• High Performance Python on Apache Spark
• slide High Performance Python on Apache Spark
• A Deep Dive into Structured Streaming
• slide A Deep Dive into Structured Streaming
• slide Spark Job Server and Spark as a Query Engine (Spark Meetup 5/14)
• Apache spark job server example with installation
• Productionizing Spark and the Spark REST Job Server
• slide Productionizing Spark and the Spark REST Job Server
• Data Profiling and Pipeline Processing with Spark
• slide Data Profiling and Pipeline Processing with Spark
• Spark at Bloomberg: Dynamically Composable Analytics
• slide Spark at Bloomberg: Dynamically Composable Analytics
• Data Storage Tips for Optimal Spark Performance – Vida Ha (Databricks)
• slide Data Storage Tips for Optimal Spark Performance-(Vida Ha, Databricks)
• Sparkling Pandas – using Apache Spark to scale Pandas – Holden Karau and Juliet Hougland