Blog source from: https://github.com/litaotao/litaotao.github.io/blob/master/_posts/new-spark/2016-03-10-spark-resouces-blogs-paper.md
————————————————————————————————————————————–
3. Databricks Blog
- Apache Spark 1.5 DataFrame API Highlights: Date/Time/String Handling, Time Intervals, and UDAFs
- Achieving End-to-end Security for Apache Spark with Databricks
上面两篇是 databricks 出的关于 databricks 专业版的描述，虽然没有从根本上解决问题，但是读起来还是挺有说服力的，哈哈，因为采用了很多很细节的方案。不错不错，各位有在做云产品的，在宣传自己的安全方案时可用参考参考哦。
- Deep Dive into Spark SQL’s Catalyst Optimizer
- A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets
- Understanding your Apache Spark application through visualization
- New Visualizations for Understanding Apache Spark Streaming Applications
- An Introduction to Writing Apache Spark Applications on Databricks
- A Gentle Introduction to Apache Spark on Databricks
- Apache Spark on Databricks for Data Scientists
- Import Notebook Apache Spark on Databricks for Data Engineers
- Structured Streaming In Apache Spark: A new high-level API for streaming
- Spark SQL Supported Syntax
- Combining Machine Learning Frameworks with Apache Spark
- Deep Dive: Memory Management in Apache Spark
- https://databricks.com/blog/2017/07/11/introducing-apache-spark-2-2.html
- https://spark-summit.org/east-2017/events/improving-python-and-spark-performance-and-interoperability/
- http://www.slideshare.net/wesm/high-performance-python-on-apache-spark
- https://www.slideshare.net/SparkSummit/spark-and-online-analytics-spark-summit-east-talky-by-shubham-chopra
- https://www.slideshare.net/SparkSummit/spark-parquet-in-depth-spark-summit-east-talk-by-emily-curtin-and-robbie-strickland
- https://www.slideshare.net/databricks/sparksql-a-compiler-from-queries-to-rdds
4. 文章，博客
- RDD论文英文版
- RDD论文中文版
- An Architecture for Fast and General Data Processing on Large Clusters
- How-to: Tune Your Apache Spark Jobs (Part 1)
- How-to: Tune Your Apache Spark Jobs (Part 2)
By 0x0fffSpark MisconceptionsBy 0x0fffSpark ArchitectureBy 0x0fffSpark DataFrames are faster, aren’t they?By 0x0fffSpark Architecture: ShuffleBy 0x0fffModern Data ArchitectureBy 0x0fffSpark Architecture TalkBy 0x0fffApache Spark FutureBy 0x0fffData Industry TrendsBy 0x0fffSpark Memory ManagementBy 0x0fffSpark Architecture Video- 借助 Redis ，让 Spark 提速 45 倍！
- 量化派基于Hadoop、Spark、Storm的大数据风控架构
- 基于Spark的异构分布式深度学习平台
- 你对Hadoop和Spark生态圈了解有几许？
- Hadoop vs Spark
- 雅虎开源CaffeOnSpark：基于Hadoop/Spark的分布式深度学习
- [2016 上海第二次 spark meetup: 1. spark_meetup.pdf](http://litaotao.github.io/files/1. spark_meetup.pdf)
- [2016 上海第二次 spark meetup: 2. Flink_ An unified stream engine.pdf](http://litaotao.github.io/files/2. Flink_ An unified stream engine.pdf)
- [2016 上海第二次 spark meetup: 3. Spark在计算广告领域的应用实践.pdf](http://litaotao.github.io/files/3. Spark在计算广告领域的应用实践.pdf)
- [2016 上海第二次 spark meetup: 4. splunk_spark.pdf](http://litaotao.github.io/files/4. splunk_spark.pdf)
- 基于Spark的医疗和金融大数据
- Monitoring Spark with Graphite and Grafana
- Spark使用CombineTextInputFormat缓解小文件过多导致Task数目过多的问题
- Databricks Empowers Enterprises to Secure Their Apache Spark Workloads
- Spark配置参数
- Running Spark Python Applications
- How-to: Prepare Your Apache Hadoop Cluster for PySpark Jobs
- Apache Spark’s Hidden REST API
- [SQL, sqlContext, hiveContext]
- [Spark Memory Issues]
- http://www.cnblogs.com/wrencai/p/4231934.html
- http://stackoverflow.com/questions/32349611/what-should-be-the-optimal-value-for-spark-sql-shuffle-partitions-or-how-do-we-i
- http://stackoverflow.com/questions/21138751/spark-java-lang-outofmemoryerror-java-heap-space
- A Beginner’s Guide on Troubleshooting Spark Applications
- http://www.mincoder.com/article/2381.shtml
- http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-1/
- http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-2/
- http://blog.cloudera.com/blog/2014/05/apache-spark-resource-management-and-yarn-app-models/
- https://spark.apache.org/docs/latest/job-scheduling.html
- https://databricks.com/blog/2015/06/22/understanding-your-spark-application-through-visualization.html
- http://blog.csdn.net/u012684933/article/details/50378725
5. 视频
- YouTube: what is apache spark
- Introduction to Spark Architecture
- Top 5 Mistakes When Writing Spark Applications
slideTop 5 mistakes when writing Spark applications- Tuning and Debugging Apache Spark
slideTuning and Debugging Apache Spark- A Deeper Understanding of Spark Internals – Aaron Davidson (Databricks)
slideA Deeper Understanding of Spark Internals – Aaron Davidson (Databricks)- Building, Debugging, and Tuning Spark Machine Learning Pipelines – Joseph Bradley (Databricks)
slideBuilding, Debugging, and Tuning Spark Machine Learning Pipelines- Spark DataFrames Simple and Fast Analysis of Structured Data – Michael Armbrust (Databricks)
slideSpark DataFrames Simple and Fast Analysis of Structured Data – Michael Armbrust (Databricks)- Spark Tuning for Enterprise System Administrators
slideSpark Tuning for Enterprise System Administrators- Structuring Spark: DataFrames, Datasets, and Streaming
slideStructuring Spark: DataFrames, Datasets, and Streaming- Spark in Production: Lessons from 100+ Production Users
slideSpark in Production: Lessons from 100+ Production Users- Production Spark and Tachyon use Cases
slideProduction Spark and Tachyon use Cases- SparkUI Visualization
slideSparkUI Visualization- Everyday I’m Shuffling – Tips for Writing Better Spark Programs, Strata San Jose 2015
slideEveryday I’m Shuffling – Tips for Writing Better Spark Programs, Strata San Jose 2015- Large Scale Distributed Machine Learning on Apache Spark
- Securing your Spark Applications
slideSecuring your Spark Applications- Building a REST Job Server for Interactive Spark as a Service
slideBuilding a REST Job Server for Interactive Spark as a Service- Exploiting GPUs for Columnar DataFrame Operations
slideExploiting GPUs for Columnar DataFrame Operations- Easy JSON Data Manipulation in Spark – Yin Huai (Databricks)
slideEasy JSON Data Manipulation in Spark – Yin Huai (Databricks)- Sparkling: Speculative Partition of Data for Spark Applications – Peilong Li
slideSparkling: Speculative Partition of Data for Spark Applications – Peilong Li- Advanced Spark Internals and Tuning – Reynold Xin
slideAdvanced Spark Internals and Tuning – Reynold Xin- The Future of Real Time in Spark
slideThe Future of Real Time in Spark- Spark 2 0
slideSpark 2 0- Democratizing Access to Data
slideDemocratizing Access to Data- Not Your Father’s Database: How to Use Apache Spark Properly in Your Big Data Architecture
slideNot Your Father’s Database: How to Use Apache Spark Properly in Your Big Data Architecture- Disrupting Big Data with Apache Spark in the Cloud
slideDisrupting Big Data with Apache Spark in the Cloud
我一直很欣赏 databricks 出的 video 和 slide，结构非常清晰，这个是其中一个非常好的演讲，里面有很多值得借鉴的地方，特别是当你像别人介绍你的工作，产品的时候。[我有一个感受，很少有人能清晰，有条理的介绍自己正在做的产品，对于一些小众的产品，甚至一些职业的销售也难以做到清晰，简明的叙述。这个 video 和 slide 有很大的参考价值。我自己感觉仔细研究这些 video 和 slide 有时候比看上一两本专业讲销售的书还要管用。]
- Getting The Best Performance With PySpark
slideGetting The Best Performance With PySpark- Disrupting Big Data with Apache Spark in the Cloud
slideDisrupting Big Data with Apache Spark in the Cloud- Large Scale Deep Learning with TensorFlow
slideLarge Scale Deep Learning with TensorFlow- 700 Queries Per Second with Updates: Spark As A Real-Time Web Service
slide700 Queries Per Second with Updates: Spark As A Real-Time Web Serviceblog700 SQL Queries per Second in Apache Spark with FiloDB- Operational Tips for Deploying Spark
slideOperational Tips for Deploying Spark- Connecting Python To The Spark Ecosystem
slideConnecting Python To The Spark Ecosystem- Livy: A REST Web Service For Apache Spark
slideLivy: A REST Web Service For Apache Spark- Understanding Memory Management In Spark For Fun And Profit
slideUnderstanding Memory Management In Spark For Fun And Profit- Deep Dive: Memory Management in Apache Spark
slideDeep Dive: Memory Management in Apache Spark- High Performance Python on Apache Spark
slideHigh Performance Python on Apache Spark- A Deep Dive into Structured Streaming
slideA Deep Dive into Structured StreamingslideSpark Job Server and Spark as a Query Engine (Spark Meetup 5/14)- Apache spark job server example with installation
- Productionizing Spark and the Spark REST Job Server
slideProductionizing Spark and the Spark REST Job Server- Data Profiling and Pipeline Processing with Spark
slideData Profiling and Pipeline Processing with Spark- Spark at Bloomberg: Dynamically Composable Analytics
slideSpark at Bloomberg: Dynamically Composable Analytics- Data Storage Tips for Optimal Spark Performance – Vida Ha (Databricks)
slideData Storage Tips for Optimal Spark Performance-(Vida Ha, Databricks)- Sparkling Pandas – using Apache Spark to scale Pandas – Holden Karau and Juliet Hougland
Advertisements
