Architecting Modern Data Platforms
Architecting Modern Data Platforms

Author: Jan Kunigk

Publisher: "O'Reilly Media, Inc."

Published: 2018-12-05

Total Pages: 688

ISBN-13: 1491969229

DOWNLOAD EBOOK

There’s a lot of information about big data technologies, but splicing these technologies into an end-to-end enterprise data platform is a daunting task not widely covered. With this practical book, you’ll learn how to build big data infrastructure both on-premises and in the cloud and successfully architect a modern data platform. Ideal for enterprise architects, IT managers, application architects, and data engineers, this book shows you how to overcome the many challenges that emerge during Hadoop projects. You’ll explore the vast landscape of tools available in the Hadoop and big data realm in a thorough technical primer before diving into: Infrastructure: Look at all component layers in a modern data platform, from the server to the data center, to establish a solid foundation for data in your enterprise Platform: Understand aspects of deployment, operation, security, high availability, and disaster recovery, along with everything you need to know to integrate your platform with the rest of your enterprise IT Taking Hadoop to the cloud: Learn the important architectural aspects of running a big data platform in the cloud while maintaining enterprise security and high availability

Data Mesh
Data Mesh

Author: Zhamak Dehghani

Publisher: "O'Reilly Media, Inc."

Published: 2022-03-08

Total Pages: 387

ISBN-13: 1492092363

DOWNLOAD EBOOK

Many enterprises are investing in a next-generation data lake, hoping to democratize data at scale to provide business insights and ultimately make automated intelligent decisions. In this practical book, author Zhamak Dehghani reveals that, despite the time, money, and effort poured into them, data warehouses and data lakes fail when applied at the scale and speed of today's organizations. A distributed data mesh is a better choice. Dehghani guides architects, technical leaders, and decision makers on their journey from monolithic big data architecture to a sociotechnical paradigm that draws from modern distributed architecture. A data mesh considers domains as a first-class concern, applies platform thinking to create self-serve data infrastructure, treats data as a product, and introduces a federated and computational model of data governance. This book shows you why and how. Examine the current data landscape from the perspective of business and organizational needs, environmental challenges, and existing architectures Analyze the landscape's underlying characteristics and failure modes Get a complete introduction to data mesh principles and its constituents Learn how to design a data mesh architecture Move beyond a monolithic data lake to a distributed data mesh.

Big Data Platforms and Applications
Big Data Platforms and Applications

Author: Florin Pop

Publisher: Springer Nature

Published: 2021-09-28

Total Pages: 300

ISBN-13: 3030388360

DOWNLOAD EBOOK

This book provides a review of advanced topics relating to the theory, research, analysis and implementation in the context of big data platforms and their applications, with a focus on methods, techniques, and performance evaluation. The explosive growth in the volume, speed, and variety of data being produced every day requires a continuous increase in the processing speeds of servers and of entire network infrastructures, as well as new resource management models. This poses significant challenges (and provides striking development opportunities) for data intensive and high-performance computing, i.e., how to efficiently turn extremely large datasets into valuable information and meaningful knowledge. The task of context data management is further complicated by the variety of sources such data derives from, resulting in different data formats, with varying storage, transformation, delivery, and archiving requirements. At the same time rapid responses are needed for real-time applications. With the emergence of cloud infrastructures, achieving highly scalable data management in such contexts is a critical problem, as the overall application performance is highly dependent on the properties of the data management service.

Cloud Data Design, Orchestration, and Management Using Microsoft Azure
Cloud Data Design, Orchestration, and Management Using Microsoft Azure

Author: Francesco Diaz

Publisher: Apress

Published: 2018-06-28

Total Pages: 451

ISBN-13: 1484236157

DOWNLOAD EBOOK

Use Microsoft Azure to optimally design your data solutions and save time and money. Scenarios are presented covering analysis, design, integration, monitoring, and derivatives. This book is about data and provides you with a wide range of possibilities to implement a data solution on Azure, from hybrid cloud to PaaS services. Migration from existing solutions is presented in detail. Alternatives and their scope are discussed. Five of six chapters explore PaaS, while one focuses on SQL Server features for cloud and relates to hybrid cloud and IaaS functionalities. What You'll Learn Know the Azure services useful to implement a data solution Match the products/services used to your specific needs Fit relational databases efficiently into data design Understand how to work with any type of data using Azure hybrid and public cloud features Use non-relational alternatives to solve even complex requirements Orchestrate data movement using Azure services Approach analysis and manipulation according to the data life cycle Who This Book Is For Software developers and professionals with a good data design background and basic development skills who want to learn how to implement a solution using Azure data services

Designing Data-Intensive Applications
Designing Data-Intensive Applications

Author: Martin Kleppmann

Publisher: "O'Reilly Media, Inc."

Published: 2017-03-16

Total Pages: 658

ISBN-13: 1491903104

DOWNLOAD EBOOK

Data is at the center of many challenges in system design today. Difficult issues need to be figured out, such as scalability, consistency, reliability, efficiency, and maintainability. In addition, we have an overwhelming variety of tools, including relational databases, NoSQL datastores, stream or batch processors, and message brokers. What are the right choices for your application? How do you make sense of all these buzzwords? In this practical and comprehensive guide, author Martin Kleppmann helps you navigate this diverse landscape by examining the pros and cons of various technologies for processing and storing data. Software keeps changing, but the fundamental principles remain the same. With this book, software engineers and architects will learn how to apply those ideas in practice, and how to make full use of data in modern applications. Peer under the hood of the systems you already use, and learn how to use and operate them more effectively Make informed decisions by identifying the strengths and weaknesses of different tools Navigate the trade-offs around consistency, scalability, fault tolerance, and complexity Understand the distributed systems research upon which modern databases are built Peek behind the scenes of major online services, and learn from their architectures

Data Engineering with Google Cloud Platform
Data Engineering with Google Cloud Platform

Author: Adi Wijaya

Publisher: Packt Publishing Ltd

Published: 2022-03-31

Total Pages: 440

ISBN-13: 1800565062

DOWNLOAD EBOOK

Build and deploy your own data pipelines on GCP, make key architectural decisions, and gain the confidence to boost your career as a data engineer Key Features Understand data engineering concepts, the role of a data engineer, and the benefits of using GCP for building your solution Learn how to use the various GCP products to ingest, consume, and transform data and orchestrate pipelines Discover tips to prepare for and pass the Professional Data Engineer exam Book DescriptionWith this book, you'll understand how the highly scalable Google Cloud Platform (GCP) enables data engineers to create end-to-end data pipelines right from storing and processing data and workflow orchestration to presenting data through visualization dashboards. Starting with a quick overview of the fundamental concepts of data engineering, you'll learn the various responsibilities of a data engineer and how GCP plays a vital role in fulfilling those responsibilities. As you progress through the chapters, you'll be able to leverage GCP products to build a sample data warehouse using Cloud Storage and BigQuery and a data lake using Dataproc. The book gradually takes you through operations such as data ingestion, data cleansing, transformation, and integrating data with other sources. You'll learn how to design IAM for data governance, deploy ML pipelines with the Vertex AI, leverage pre-built GCP models as a service, and visualize data with Google Data Studio to build compelling reports. Finally, you'll find tips on how to boost your career as a data engineer, take the Professional Data Engineer certification exam, and get ready to become an expert in data engineering with GCP. By the end of this data engineering book, you'll have developed the skills to perform core data engineering tasks and build efficient ETL data pipelines with GCP.What you will learn Load data into BigQuery and materialize its output for downstream consumption Build data pipeline orchestration using Cloud Composer Develop Airflow jobs to orchestrate and automate a data warehouse Build a Hadoop data lake, create ephemeral clusters, and run jobs on the Dataproc cluster Leverage Pub/Sub for messaging and ingestion for event-driven systems Use Dataflow to perform ETL on streaming data Unlock the power of your data with Data Studio Calculate the GCP cost estimation for your end-to-end data solutions Who this book is for This book is for data engineers, data analysts, and anyone looking to design and manage data processing pipelines using GCP. You'll find this book useful if you are preparing to take Google's Professional Data Engineer exam. Beginner-level understanding of data science, the Python programming language, and Linux commands is necessary. A basic understanding of data processing and cloud computing, in general, will help you make the most out of this book.

Customer Data Platforms
Customer Data Platforms

Author: Martin Kihn

Publisher: John Wiley & Sons

Published: 2020-11-06

Total Pages: 242

ISBN-13: 1119790131

DOWNLOAD EBOOK

Master the hottest technology around to drive marketing success Marketers are faced with a stark and challenging dilemma: customers demand deep personalization, but they are increasingly leery of offering the type of personal data required to make it happen. As a solution to this problem, Customer Data Platforms have come to the fore, offering companies a way to capture, unify, activate, and analyze customer data. CDPs are the hottest marketing technology around today, but are they worthy of the hype? Customer Data Platforms takes a deep dive into everything CDP so you can learn how to steer your firm toward the future of personalization. Over the years, many of us have built byzantine “stacks” of various marketing and advertising technology in an attempt to deliver the fabled “right person, right message, right time” experience. This can lead to siloed systems, disconnected processes, and legacy technical debt. CDPs offer a way to simplify the stack and deliver a balanced and engaging customer experience. Customer Data Platforms breaks down the fundamentals, including how to: Understand the problems of managing customer data Understand what CDPs are and what they do (and don't do) Organize and harmonize customer data for use in marketing Build a safe, compliant first-party data asset that your brand can use as fuel Create a data-driven culture that puts customers at the center of everything you do Understand how to use AI and machine learning to drive the future of personalization Orchestrate modern customer journeys that react to customers in real-time Power analytics with customer data to get closer to true attribution In this book, you’ll discover how to build 1:1 engagement that scales at the speed of today’s customers.

Google Cloud Platform for Architects
Google Cloud Platform for Architects

Author: Vitthal Srinivasan

Publisher: Packt Publishing Ltd

Published: 2018-06-26

Total Pages: 355

ISBN-13: 1788833074

DOWNLOAD EBOOK

Get acquainted with GCP and manage robust, highly available, and dynamic solutions to drive business objective Key Features Identify the strengths, weaknesses and ideal use-cases for individual services offered on the Google Cloud Platform Make intelligent choices about which cloud technology works best for your use-case Leverage Google Cloud Platform to analyze and optimize technical and business processes Book Description Using a public cloud platform was considered risky a decade ago, and unconventional even just a few years ago. Today, however, use of the public cloud is completely mainstream - the norm, rather than the exception. Several leading technology firms, including Google, have built sophisticated cloud platforms, and are locked in a fierce competition for market share. The main goal of this book is to enable you to get the best out of the GCP, and to use it with confidence and competence. You will learn why cloud architectures take the forms that they do, and this will help you become a skilled high-level cloud architect. You will also learn how individual cloud services are configured and used, so that you are never intimidated at having to build it yourself. You will also learn the right way and the right situation in which to use the important GCP services. By the end of this book, you will be able to make the most out of Google Cloud Platform design. What you will learn Set up GCP account and utilize GCP services using the cloud shell, web console, and client APIs Harness the power of App Engine, Compute Engine, Containers on the Kubernetes Engine, and Cloud Functions Pick the right managed service for your data needs, choosing intelligently between Datastore, BigTable, and BigQuery Migrate existing Hadoop, Spark, and Pig workloads with minimal disruption to your existing data infrastructure, by using Dataproc intelligently Derive insights about the health, performance, and availability of cloud-powered applications with the help of monitoring, logging, and diagnostic tools in Stackdriver Who this book is for If you are a Cloud architect who is responsible to design and manage robust cloud solutions with Google Cloud Platform, then this book is for you. System engineers and Enterprise architects will also find this book useful. A basic understanding of distributed applications would be helpful, although not strictly necessary. Some working experience on other public cloud platforms would help too.

Data Science on the Google Cloud Platform
Data Science on the Google Cloud Platform

Author: Valliappa Lakshmanan

Publisher: "O'Reilly Media, Inc."

Published: 2017-12-12

Total Pages: 403

ISBN-13: 1491974532

DOWNLOAD EBOOK

Learn how easy it is to apply sophisticated statistical and machine learning methods to real-world problems when you build on top of the Google Cloud Platform (GCP). This hands-on guide shows developers entering the data science field how to implement an end-to-end data pipeline, using statistical and machine learning methods and tools on GCP. Through the course of the book, you’ll work through a sample business decision by employing a variety of data science approaches. Follow along by implementing these statistical and machine learning solutions in your own project on GCP, and discover how this platform provides a transformative and more collaborative way of doing data science. You’ll learn how to: Automate and schedule data ingest, using an App Engine application Create and populate a dashboard in Google Data Studio Build a real-time analysis pipeline to carry out streaming analytics Conduct interactive data exploration with Google BigQuery Create a Bayesian model on a Cloud Dataproc cluster Build a logistic regression machine-learning model with Spark Compute time-aggregate features with a Cloud Dataflow pipeline Create a high-performing prediction model with TensorFlow Use your deployed model as a microservice you can access from both batch and real-time pipelines

Cloud Native Patterns
Cloud Native Patterns

Author: Cornelia Davis

Publisher: Simon and Schuster

Published: 2019-05-12

Total Pages: 573

ISBN-13: 1638356858

DOWNLOAD EBOOK

Summary Cloud Native Patternsis your guide to developing strong applications that thrive in the dynamic, distributed, virtual world of the cloud. This book presents a mental model for cloud-native applications, along with the patterns, practices, and tooling that set them apart. Purchase of the print book includes a free eBook in PDF, Kindle, and ePub formats from Manning Publications. About the Technology Cloud platforms promise the holy grail: near-zero downtime, infinite scalability, short feedback cycles, fault-tolerance, and cost control. But how do you get there? By applying cloudnative designs, developers can build resilient, easily adaptable, web-scale distributed applications that handle massive user traffic and data loads. Learn these fundamental patterns and practices, and you'll be ready to thrive in the dynamic, distributed, virtual world of the cloud. About the Book With 25 years of experience under her belt, Cornelia Davis teaches you the practices and patterns that set cloud-native applications apart. With realistic examples and expert advice for working with apps, data, services, routing, and more, she shows you how to design and build software that functions beautifully on modern cloud platforms. As you read, you will start to appreciate that cloud-native computing is more about the how and why rather than the where. What's inside The lifecycle of cloud-native apps Cloud-scale configuration management Zero downtime upgrades, versioned services, and parallel deploys Service discovery and dynamic routing Managing interactions between services, including retries and circuit breakers About the Reader Requires basic software design skills and an ability to read Java or a similar language. About the Author Cornelia Davis is Vice President of Technology at Pivotal Software. A teacher at heart, she's spent the last 25 years making good software and great software developers. Table of Contents PART 1 - THE CLOUD-NATIVE CONTEXT You keep using that word: Defining "cloud-native" Running cloud-native applications in production The platform for cloud-native software PART 2 - CLOUD-NATIVE PATTERNS Event-driven microservices: It's not just request/response App redundancy: Scale-out and statelessness Application configuration: Not just environment variables The application lifecycle: Accounting for constant change Accessing apps: Services, routing, and service discovery Interaction redundancy: Retries and other control loops Fronting services: Circuit breakers and API gateways Troubleshooting: Finding the needle in the haystack Cloud-native data: Breaking the data monolith