As firms transfer to Spark and a Lakehouse structure, they’re realizing that the info instruments are lagging manner behind. You might want to be a programmer to successfully use Spark and Airflow.
There are some low-code ETL instruments, however is that sufficient? Corporations wish to deal with their information pipelines like mission-critical apps. They need DevOps for information, with GIT, take a look at protection, and CI/CD. As well as, firms want end-to-end visibility of their information pipelines, together with monitoring, metadata, and column-level lineage.
The place are the info instruments for these capabilities and the way do non-programmers use these?
In right now’s podcast, you’ll see how Prophecy’s low-code information engineering platform, constructed on Spark and Airflow, gives a whole answer for creating, deploying, and monitoring information pipelines. We’ll converse with Raj Bains, who’s the founder & CEO of Prophecy.
Raj has spent 20 years constructing highly effective instruments – Microsoft Visible Studio, founding engineer for CUDA at NVIDIA, product supervisor for Hive at Hortonworks. He’s obsessed with making all organizations productive with information and making the lives of knowledge engineers higher by lowering toil and growing the enjoyment.
Additionally, in case you like what you hear, Prophecy has a particular supply for our listeners. Go to Prophecy.IO/request-a-demo and put in “SED” for the referral code. In case you determine to buy inside 3 months of the demo, you get 10% off. If you are going to buy after 3 months, you continue to get 5% off.
Sponsorship inquiries: sponsor@softwareengineeringdaily.com
Transcript
Transcript offered by We Edit Podcasts. Software program Engineering Each day listeners can go to weeditpodcasts.com to get 15% off the primary three months of audio enhancing and transcription providers with code: SED. Because of We Edit Podcasts for partnering with SE Each day. Please click on right here to view this present’s transcript.
Sponsors
WorkOS is a developer platform to make your app enterprise-ready. With a couple of easy APIs, you possibly can instantly add frequent enterprise options like Single Signal-On, SAML, SCIM consumer provisioning, and extra. Builders will discover lovely docs and SDKs that make integration a breeze. WorkOS is form of like “Stripe for enterprise options.” WorkOS powers apps like Webflow, Hopin, Vercel, and greater than 100 others. The platform is rock stable, totally SOC-2 compliant, and prepared for even the biggest enterprise environments. So what are you ready for? Combine WorkOS right now and make your app enterprise-ready. To study extra and get began, go to softwareengineeringdaily.com/workos
Cox Automotive transforms the best way that the world buys, sells, owns, and makes use of vehicles – they usually have the info to know how the world will get from level A to level B. With manufacturers like Kelley Blue E-book, Autotrader, Vendor.com and others, Cox Automotive is hiring software program engineers, information scientists, scrum masters, and different technologists to assist create significant change within the business. If you wish to innovate in a collaborative office, one which values your time and work-life steadiness, then go to COXAUTOTECH.COM to search out profession alternatives at Cox.
Puppet, the business chief for infrastructure automation, invitations you to a hearth chat to learn to enhance collaboration and construct a tradition of joint accountability for compliance and safety throughout your group.
Be part of Alex Hin, Senior Product Supervisor at Puppet, for a dialogue of how IT, safety, compliance, and threat administration groups can work collectively to guard your group from threat.
You’ll be able to be a part of Puppet for the dialog on Thursday, April seventh at 11 AM jap and begin constructing your personal tradition of accountability that enhances compliance by registering at occasions.puppet.com/hearth.
strongDM helps you to handle and audit entry to servers, databases, and Kubernetes clusters, regardless of the place your staff are. With strongDM, you possibly can simply prolong your id supplier to handle infrastructure entry. You’ll be able to automate onboarding, offboarding, and transferring individuals inside roles. strongDM. Handle and audit distant entry to infrastructure. Begin your free 14 day trial right now at: strongdm.com/SEDaily
Knowledge engineers scuffling with unreliable information depend on Monte Carlo, the world’s first end-to-end, totally automated Knowledge Observability Platform! Monte Carlo allows information groups with visibility into the standard and reliability of their analytical information to maximise enterprise influence. Begin trusting your information with Monte Carlo right now! Go to softwareengineeringdaily.com/montecarlodata to study extra.