A surprising amount of B2B data still moves like this: a client's system exports a file on a schedule, drops it on an SFTP server, and something on your side is supposed to notice. The gap between "a file landed in a folder" and "my application received clean, structured data" is where integration teams lose their weeks.
This guide walks through what it actually takes to go from SFTP to webhook: detecting the file, parsing it, mapping it to your schema, validating it, and delivering it as a JSON event your application can consume, reliably, every time the client sends a new file.
Why SFTP to webhook is harder than it sounds
Most SFTP tooling stops at the transfer. Managed file transfer products and hosted SFTP services can tell you a file arrived, and some can fire a notification webhook with the file's name and path. That is genuinely useful, but it hands your application a pointer to raw bytes, not data. Someone still has to download the file, work out the layout, map the columns, validate the rows, and push the result into your system.
The hard part is that every client's file is different. One sends UTF-16 with a byte order mark, another wraps everything in quotes, a third renames a column in October without telling anyone. A pipeline that worked on the sample file in the kickoff call breaks on week two's real file. If you have read our guide on SFTP file automation, this is the layer above it: not moving the file, but understanding it.
Step 1: Detect the file without polling pain
There are two patterns for noticing a new file. If you host the SFTP endpoint, the server can emit an event on upload; this is the clean path, with no polling and no missed windows. If the client hosts the server, you poll on an interval, list the directory, and diff against what you have already processed. Polling needs idempotency: track processed files by name plus size or checksum, because clients re-upload corrected files under the same name more often than you would expect.
Watch for partial uploads too. A large file appears in the directory before the client's upload finishes. Naive pipelines grab it mid-write and parse half a file. The usual fixes are upload-complete markers, size stability checks across two polls, or atomic rename conventions.
Step 2: Parse defensively
Real client files are messier than any spec. Delimiters vary, encodings vary, header rows appear on line three under a title block, and Excel files carry merged cells and hidden sheets. A production parser needs encoding detection, delimiter sniffing, header row detection, and a policy for blank lines and trailing garbage. Our post on common CSV import errors covers the failure catalogue in detail.
Step 3: Map to your schema, once per client
Mapping is where file-based integrations diverge from API integrations. With an API, the contract is fixed. With files, the contract is whatever each client's system exports, so the mapping has to be defined per client: their "EMP_NO" is your "employee_id", their "DOB" needs a date format conversion, their status codes need normalization.
The workable pattern is to learn the mapping from the first file, have a human review and approve it, then apply the exact same mapping to every subsequent file automatically. Automatic suggestions get you most of the way; the approval step is what keeps a silently changed column from flowing into production unnoticed. We wrote more about that approach in how to map customer CSV files automatically.
Step 4: Validate before you deliver
The webhook you deliver is only as trustworthy as the validation in front of it. Row-level checks (required fields, types, date formats, enums) catch bad records; file-level checks (row counts within expected range, mandatory columns present) catch bad files. The important design decision is what happens on failure: reject the whole file, deliver the good rows and quarantine the bad ones, or hold for review. For recurring feeds, delivering good rows and reporting exceptions usually beats all-or-nothing, because a single malformed row should not stop payroll data for four thousand people.
Step 5: Deliver as a webhook your app can trust
The delivery itself has its own checklist: sign the payload so the receiver can verify it came from your pipeline, include a stable event id so consumers can deduplicate, retry with backoff on non-2xx responses, and keep a delivery log you can replay from. For large files, send a summary event with a fetch URL rather than a fifty-megabyte POST body.
Finally, keep an audit trail of the whole run: which file, which mapping version, how many rows in, how many rows out, what failed and why. When a client asks why a record is missing, the answer should be one search away, not an archaeology project.
Build vs buy
Everything above is buildable in-house, and for one stable feed from one client it can be a reasonable week of work. The cost curve bends when feeds multiply: ten clients means ten mappings to maintain, ten flavors of malformed input, and a steady stream of "the file looks different this month" tickets landing on engineers. That maintenance load, not the initial build, is what pushes teams to a platform.
FileFeed does this exact path as a product: your clients drop files on SFTP (hosted by us or on their own server) or email them in, mapping is learned once per client and approved by you, every row is validated against your schema, and clean JSON arrives at your endpoint over a signed webhook with retries and a full run log. If you want to see it on your own data instead of a demo, send one real client file and it comes back mapped the same day.
Related resources