KDnuggets
JULY 31, 2019
I had a feeling that R has developed as a language to such a degree that many of us are using it now in completely different ways. This means that there are likely to be numerous tricks, packages, functions, etc that each of us use, but that others are completely unaware of, and would find useful if they knew about them.
Confluent
JULY 24, 2019
Using Jaeger tracing, I’ve been able to answer an important question that nearly every Apache Kafka ® project that I’ve worked on posed: how is data flowing through my distributed system? Quick disclaimer: if you’re simply looking for an answer to that question, this post won’t provide that answer directly. Instead, in this post I will point you to an earlier blog post where I already answered that question and then I will focus on what should be your next question: now that I’m relying on Jaege
Cloudera
JULY 10, 2019
Open source has been core to the missions of both Hortonworks and Cloudera and central to our values and culture. With more than 700 engineers in the new Cloudera, our company writes a prodigious amount of open source code each year that’s contributed to more than 30 different open source projects. We’re also a very innovative open source company, having collectively launched more than a dozen new open source projects since the founding of the two companies. .
Teradata
JULY 9, 2019
Integrated data and analytics has a proven track record of helping organize operations, enhance customer experience and improve revenue and market growth.
Advertisement
Whether you’re creating complex dashboards or fine-tuning large language models, your data must be extracted, transformed, and loaded. ETL and ELT pipelines form the foundation of any data product, and Airflow is the open-source data orchestrator specifically designed for moving and transforming data in ETL and ELT pipelines. This eBook covers: An overview of ETL vs.
Data Engineering Podcast
JULY 28, 2019
Summary The ETL pattern that has become commonplace for integrating data from multiple sources has proven useful, but complex to maintain. For a small number of sources it is a tractable problem, but as the overall complexity of the data ecosystem continues to expand it may be time to identify new ways to tame the deluge of information. In this episode Tim Ward, CEO of CluedIn, explains the idea of eventual connectivity as a new paradigm for data integration.
Netflix Tech
JULY 30, 2019
v2.0 and beyond By Anoop Panicker and Kishore Banala Conductor is a workflow orchestration engine developed and open-sourced by Netflix. If you’re new to Conductor, this earlier blogpost and the documentation should help you get started and acclimatized to Conductor. Netflix Conductor: A microservices orchestrator In the last two years since inception, Conductor has seen wide adoption and is instrumental in running numerous core workflows at Netflix.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
Confluent
JULY 17, 2019
Together, MongoDB and Apache Kafka ® make up the heart of many modern data architectures today. Integrating Kafka with external systems like MongoDB is best done though the use of Kafka Connect. This API enables users to leverage ready-to-use components that can stream data from external systems into Kafka topics, as well as stream data from Kafka topics into external systems.
Cloudera
JULY 9, 2019
Every business aims to deliver products and services quickly and efficiently based upon customer wants and needs. Today, much of that speed and efficiency relies on insights driven by big data. Yet big data management often serves as a stumbling block, because many businesses continue to struggle with how to best capture and analyze their data. Unorganized data presents another roadblock.
Teradata
JULY 14, 2019
Analytics can help enterprises answer the toughest business questions by leveraging all of the data across an organization.
Data Engineering Podcast
JULY 22, 2019
Summary The current trend in data management is to centralize the responsibilities of storing and curating the organization’s information to a data engineering team. This organizational pattern is reinforced by the architectural pattern of data lakes as a solution for managing storage and access. In this episode Zhamak Dehghani shares an alternative approach in the form of a data mesh.
Speaker: Tamara Fingerlin, Developer Advocate
In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!
Netflix Tech
JULY 12, 2019
By Drew Koszewnik This is the story about how the Content Setup Engineering team used Hollow, a Netflix OSS technology, to re-architect and simplify an essential component in our content pipeline?—?delivering a large amount of business value in the process. The Context Each movie and show on the Netflix service is carefully curated to ensure an optimal viewing experience.
KDnuggets
JULY 26, 2019
This article takes a closer look at the four fantastic things we should keep in mind when approaching every new data science project.
Confluent
JULY 3, 2019
One of the football (as per European terminology) highlights of the summer is the FIFA Women’s World Cup. France, Brazil, and the USA are the favourites, and this year Italy is present at the event for the first time in 20 years. From a data perspective, the World Cup represents an interesting source of information. There’s a lot of dedicated press coverage, as well as the standard social media excitement following any kind of big event.
Cloudera
JULY 15, 2019
At a time when machine learning, deep learning, and artificial intelligence capture an outsize share of media attention, jobs requiring SQL skills continue to vastly outnumber jobs requiring those more advanced skills. Influential data scientists often point to SQL as the most important yet underrated skill for anyone who works with data. SQL is today—and will remain for the foreseeable future—a vital foundational skill for a wide range of data professionals working in different roles across dif
Advertisement
Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.
Teradata
JULY 16, 2019
Teradata Vantage's NewSQL Engine's performance-enhancing options include column-row hybrid partitioning. Find out how to take advantage of this great feature.
Data Engineering Podcast
JULY 14, 2019
Summary Successful machine learning and artificial intelligence projects require large volumes of data that is properly labelled. The challenge is that most data is not clean and well annotated, requiring a scalable data labeling process. Ideally this process can be done using the tools and systems that already power your analytics, rather than sending data into a black box.
Netflix Tech
JULY 1, 2019
Bringing Rich Experiences to Memory-Constrained TV Devices By Jason Munning, Archana Kumar, Kris Range Netflix has over 148M paid members streaming on more than half a billion devices spanning over 1,900 different types. In the TV space alone, there are hundreds of device types that run the Netflix app. We need to support the same rich Netflix experience on not only high-end devices like the PS4 but also memory and processor-constrained consumer electronic devices that run a similar chipset as w
KDnuggets
JULY 26, 2019
Different neural network architectures excel in different tasks. This particular article focuses on crafting convolutional neural networks in Python using TensorFlow and Keras.
Advertisement
Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?
Confluent
JULY 16, 2019
Have you ever realized that, according to the latest FBI report , more than 80% of all crimes are property crimes, such as burglaries? And that the FBI clearance figures indicate that only 13% of all burglaries in 2017 were cleared due to lack of witnesses and/or physical evidence? How cool would it be to build your own burglar alarm system that can alert you before the actual event takes place simply by using a few network-connected cameras and analyzing the camera images with Apache Kafka ® ,
Pandora Engineering
JULY 29, 2019
Credit: Kanok Sulaiman Disclaimer: These are my experiences from being a Pandora software developer intern in the summer of 2019. All opinions expressed are my own, and represent no one except myself. I recently spent the last summer of my undergraduate program as an intern for Pandora Media in Oakland, CA. I gained a lot from my experience, and I’m writing this post to detail the application process, the lessons that I learned, and the company culture.
Teradata
JULY 28, 2019
In a cost-cutting culture, directly linking data projects to top business initiatives is a good way to keep them from getting clipped. Learn more.
Data Engineering Podcast
JULY 8, 2019
Summary The market for data warehouse platforms is large and varied, with options for every use case. ClickHouse is an open source, column-oriented database engine built for interactive analytics with linear scalability. In this episode Robert Hodges and Alexander Zaitsev explain how it is architected to provide these features, the various unique capabilities that it provides, and how to run it in production.
Speaker: Jay Allardyce, Deepak Vittal, Terrence Sheflin, and Mahyar Ghasemali
As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.
Cloud Academy
JULY 16, 2019
Designing a streaming data pipeline presents many challenges, particularly around specific technology requirements. When designing a cloud-based solution, an architect is no longer faced with the question, “How do I get this job done with the technology we have?” but rather, “What is the right technology to support my use case?” In this blog post, we will walk through some initial scoping steps and walk through an example.
KDnuggets
JULY 26, 2019
Education, coding, SQL, big data platforms, storytelling and more. These are the 13 skills you need to master to become a rockstar data scientist.
Confluent
JULY 1, 2019
This question comes up on Stack Overflow and such places a lot , so here’s something to try and help. tl;dr: You need to set advertised.listeners (or KAFKA_ADVERTISED_LISTENERS if you’re using Docker images) to the external address (host/IP) so that clients can correctly connect to it. Otherwise, they’ll try to connect to the internal host address—and if that’s not reachable, then problems ensue.
Zalando Engineering
JULY 14, 2019
Project Highlights Kopf - Kubernetes Operator Pythonic Framework now supports built-in resources and can be used to write controllers of any kind (pods, namespaces, mixed), not only of custom resources. Check out the latest release for more details [link] Skipper publishes new releases weekly. Some of the important features were implemented such as support to proxy Kubernetes API server and support Kubernetes externalName services from ingress.
Speaker: Nikhil Joshi, Founder & President of Snic Solutions
Is your manufacturing operation reaching its efficiency potential? A Manufacturing Execution System (MES) could be the game-changer, helping you reduce waste, cut costs, and lower your carbon footprint. Join Nikhil Joshi, Founder & President of Snic Solutions, in this value-packed webinar as he breaks down how MES can drive operational excellence and sustainability.
Teradata
JULY 7, 2019
Learn the five key steps to minimize downtime and ensure a seamless analytics software upgrade.
Data Engineering Podcast
JULY 1, 2019
Summary Anomaly detection is a capability that is useful in a variety of problem domains, including finance, internet of things, and systems monitoring. Scaling the volume of events that can be processed in real-time can be challenging, so Paul Brebner from Instaclustr set out to see how far he could push Kafka and Cassandra for this use case. In this interview he explains the system design that he tested, his findings for how these tools were able to work together, and how they behaved at diffe
InData Labs
JULY 11, 2019
In the highly competitive business world of today, data reign supreme. Customer personal data, comprehensive operating statistics, sales figures, or inter-company information may play a core role in strategic decision making. It’s vital to keep an eye on the quantity and quality of data that can be captured and extracted from different web sources.
KDnuggets
JULY 30, 2019
The Tensor Processing Unit (TPU) is Google's custom tool to accelerate machine learning workloads using the TensorFlow framework. Learn more about what TPUs do and how they can work for you.
Speaker: Anindo Banerjea, CTO at Civio & Tony Karrer, CTO at Aggregage
When developing a Gen AI application, one of the most significant challenges is improving accuracy. This can be especially difficult when working with a large data corpus, and as the complexity of the task increases. The number of use cases/corner cases that the system is expected to handle essentially explodes. 💥 Anindo Banerjea is here to showcase his significant experience building AI/ML SaaS applications as he walks us through the current problems his company, Civio, is solving.
Expert insights. Personalized for you.
123 
Let's personalize your content