Start Data Engineering
JUNE 26, 2020
1. AWS account Sign up for an AWS account at AWS Sign Up. You will be eligible for some free services for the first time sign up, ref: AWS Free Tier get your access key by clicking on your name -> My Security Credentials on the top pane and then clicking Create New Access Key.
Simon Späti
JUNE 14, 2020
Today we have more requirements with ever-growing tools and framework, complex cloud architectures, and with data stack that is changing rapidly. I hear claims: “Business Intelligence (BI) takes too long to integrate new data”, or “understanding how the numbers match up is very hard and needs lots of analysis”. The goal of this article is to make business intelligence easier, faster and more accessible with techniques from the sphere of data engineering.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Team Data Science
JUNE 6, 2020
Welcome back to this Toronto Specific data engineering project. We left off last time concluding finance has the largest demand for data engineers who have skills with AWS, and sketched out what our data ingestion pipeline will look like. I began building out the data ingestion pipeline by launching an EC2 instance. I should note that if you have created an AWS account, but have not yet created an Identity Access Management (IAM) admin role, and are therefore still using root credentials, I am s
Confluent
JUNE 4, 2020
The rise of IoT devices means that we have to collect, process, and analyze orders of magnitude more data than ever before. As sensors and devices become ever more ubiquitous, […].
Speaker: Jay Allardyce
As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.
Data Engineering Podcast
JUNE 29, 2020
Summary We have machines that can listen to and process human speech in a variety of languages, but dealing with unstructured sounds in our environment is a much greater challenge. The team at Audio Analytic are working to impart a sense of hearing to our myriad devices with their sound recognition technology. In this episode Dr. Chris Mitchell and Dr.
Teradata
JUNE 17, 2020
Juneteenth has been declared a U.S. holiday at Teradata, as we stand with the black community and reflect on what we can do to fight racism and injustice, and embrace diversity.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
Netflix Tech
JUNE 30, 2020
By Steve Urban , Sridhar Seetharaman , Shilpa Motukuri , Tom Mack , Erik Strauss , Hema Kannan , CJ Barker Netflix is revolutionizing the way a modern studio operates. Our mission in Studio Engineering is to build a unified, global, and digital studio that powers the effective production of amazing content. [link] Netflix produces some of the world’s most beloved and award-winning films and series, including The Irishman, The Crown, La Casa de Papel, Ozark, and Tiger King.
Advancing Analytics: Data Engineering
JUNE 16, 2020
You might have seen that I’ve been pretty busy recently, digging into the new Azure Synapse Analytics preview, announced back at Microsoft Build 2020. I’ve explored the spark engine, SQL serverless/On-Demand and various other bits… but I’m still getting the same question of “Cool!…. but what actually is it?”. One of the problems here is that Azure SQL Data Warehouse was rebranded as “Azure Synapse Analytics”… but it’s not the same as the full workspace.
Confluent
JUNE 9, 2020
tl;dr When a client wants to send or receive a message from Apache Kafka®, there are two types of connection that must succeed: The initial connection to a broker (the […].
Data Engineering Podcast
JUNE 22, 2020
Summary The majority of analytics platforms are focused on use internal to an organization by business stakeholders. As the availability of data increases and overall literacy in how to interpret it and take action improves there is a growing need to bring business intelligence use cases to a broader audience. GoodData is a platform focused on simplifying the work of bringing data to employees and end users.
Speaker: Nikhil Joshi, Founder & President of Snic Solutions
In an era where efficiency is king, are you leveraging the right tools to transform your manufacturing processes? A Manufacturing Execution System (MES) is critical for enhancing operational efficiency, reducing waste, and optimizing energy usage—key factors for improving your bottom line and lowering your carbon footprint. Join Nikhil Joshi, a manufacturing technology expert with 18+ years of hands-on experience, in this new webinar as he uncovers the secrets of MES and how to best utilize thes
Teradata
JUNE 22, 2020
When it comes to being a modern data warehouse, your age really is just a number. It’s the underlying capabilities that actually count. Read more.
Start Data Engineering
JUNE 2, 2020
I have seen and been asked the following questions by students, backend engineers and analysts who want to get into the data engineering industry. What approach should i take to land a Data Engineering job? I really want to get into DE. What can I do to learn more about it? In this article, I will try to provide a general approach that you as a beginner, student, backend engineer or analyst can use to land your first data engineering job.
U-Next
JUNE 4, 2020
Today, the way Human Resources function has changed from how it used to operate in the 90s and early 2000. Introduction of Human Resource Management (HRM) software in the late 90s had already revolutionized the way HR departments across various industries functioned. The HR department is not contained in the back-office anymore, where dealing with paperwork and recruiting were the only processes they were involved in.
dbt Developer Hub
JUNE 30, 2020
Before I dive into how to create this, I have to say this. You probably don’t need this. I, along with my other Fishtown colleagues, have spent countless hours working with clients that ask for near-real-time streaming data. However, when we start digging into the project, it is often realized that the use case is not there. There are a variety of reasons why near real-time streaming is not a good fit.
Speaker: Anindo Banerjea, CTO at Civio & Tony Karrer, CTO at Aggregage
When developing a Gen AI application, one of the most significant challenges is improving accuracy. This can be especially difficult when working with a large data corpus, and as the complexity of the task increases. The number of use cases/corner cases that the system is expected to handle essentially explodes. 💥 Anindo Banerjea is here to showcase his significant experience building AI/ML SaaS applications as he walks us through the current problems his company, Civio, is solving.
Confluent
JUNE 30, 2020
You know the fundamentals of Apache Kafka®. You are a Spring Boot developer working with Apache Kafka. You have chosen Spring Kafka for your integration. You have implemented your first […].
Data Engineering Podcast
JUNE 15, 2020
Summary Machine learning is a process driven by iteration and experimentation which requires fast and easy access to relevant features of the data being processed. In order to reduce friction in the process of developing and delivering models there has been a recent trend toward building a dedicated feature. In this episode Simba Khadder discusses his work at StreamSQL building a feature store to make creation, discovery, and monitoring of features fast and easy to manage.
Teradata
JUNE 29, 2020
Vantage Trial provides free, 30-day access to Teradata Vantage in the cloud along with easy-to-use, web-based tools and applications for performing advanced analytics. Learn more.
Start Data Engineering
JUNE 19, 2020
Intro A lot of tutorials show how to write spark code with just the API and code samples, but they do not explain how to write “efficient Apache Spark” code.
Speaker: Donna Laquidara-Carr, PhD, LEED AP, Industry Insights Research Director at Dodge Construction Network
In today’s construction market, owners, construction managers, and contractors must navigate increasing challenges, from cost management to project delays. Fortunately, digital tools now offer valuable insights to help mitigate these risks. However, the sheer volume of tools and the complexity of leveraging their data effectively can be daunting. That’s where data-driven construction comes in.
Zalando Engineering
JUNE 21, 2020
Overview XGBoost is a popular Python library for gradient boosted decision trees. The implementation allows practitioners to distribute training across multiple compute instances (or workers), which is especially useful for large training sets. One tool used at Zalando for deploying production machine learning models is the managed service from Amazon called SageMaker.
Rockset
JUNE 2, 2020
MongoDB World is going all virtual with MongoDB.live. Registration is free and there’s tons of content to get excited about. It’s so easy to get overwhelmed on what to pick (heck, you could just watch all of them)! If you’re short on time, fear not- here are our top 10 MongoDB sessions to watch out for: 10 Join the Data Movement: MongoDB and Apache Kafka One of the go-to picks for companies that need a streaming platform is Apache Kafka.
Confluent
JUNE 23, 2020
Company mergers are becoming more common as businesses strive to improve performance and grow market share by saving costs and eliminating competition through acquisitions. But how do business mergers relate […].
Data Engineering Podcast
JUNE 8, 2020
Summary The landscape of data management and processing is rapidly changing and evolving. There are certain foundational elements that have remained steady, but as the industry matures new trends emerge and gain prominence. In this episode Astasia Myers of Redpoint Ventures shares her perspective as an investor on which categories she is paying particular attention to for the near to medium term.
Speaker: Aindra Misra, Senior Manager, Product Management (Data, ML, and Cloud Infrastructure) at BILL
Join us for an insightful webinar that explores the critical intersection of data privacy and AI governance. In today’s rapidly evolving tech landscape, building robust governance frameworks is essential to fostering innovation while staying compliant with regulations. Our expert speaker, Aindra Misra, will guide you through best practices for ensuring data protection while leveraging AI capabilities.
Teradata
JUNE 28, 2020
Fighting the pandemic means fighting the misinformation that comes along with it. That's why one of our Global COVID-19 Hackathon teams created a tool to help identify fake news.
Start Data Engineering
JUNE 26, 2020
EMR AWS EMR is a managed service provided by AWS to run Spark, HDFS, HIVE and other select software.
Zalando Engineering
JUNE 30, 2020
Our Engineering Blog was launched in June 2020 after a long break of the previous tech blog. This post describes the technical setup behind engineering.zalando.com. You will learn: Which static site generator we selected and why. What customizations we applied to design the blog and the publishing process. How we serve static HTML using Skipper and S3.
Preset
JUNE 25, 2020
In this blog we will understand better what are Time Series and provide some examples of time series visualizations in Superset
Advertisement
Entity Resolution Sometimes referred to as data matching or fuzzy matching, entity resolution, is critical for data quality, analytics, graph visualization and AI. Learn what entity resolution is, why it matters, how it works and its benefits. Advanced entity resolution using AI is crucial because it efficiently and easily solves many of today’s data quality and analytics problems.
Confluent
JUNE 19, 2020
When I have a small software project that I want to share with the world, I don’t write my own version control system with a web UI. I don’t even […].
Monte Carlo
JUNE 11, 2020
One of our customers recently posed this question related to data quality metrics: I would like to set up an OKR for ourselves [the data team] around data availability. I’d like to establish a single data quality KPI that would summarize availability, freshness, quality. What’s the best way to do this? I can’t tell you how much joy this request brought me.
Teradata
JUNE 24, 2020
Learn how Teradata Vantage's advanced analytics capabilities can analyze and predict useful diagnoses and insights in biomedicine and healthcare.
Rockset
JUNE 16, 2020
MongoDB.live took place last week, and Rockset had the opportunity to participate alongside members of the MongoDB community and share about our work to make MongoDB data accessible via real-time external indexing. In our session, we discussed the need for modern data-driven applications to perform real-time aggregations and joins, and how Rockset uses MongoDB change streams and Converged Indexing to deliver fast queries on data from MongoDB.
Speaker: Evelyn Chou
Choosing the right business intelligence (BI) platform can feel like navigating a maze of features, promises, and technical jargon. With so many options available, how can you ensure you’re making the right decision for your organization’s unique needs? 🤔 This webinar brings together expert insights to break down the complexities of BI solution vetting.
Expert insights. Personalized for you.
130 
Let's personalize your content