SFTP to Webhook: Turn Client File Drops into Clean JSON Events

SFTP to webhook means a file a client drops on an SFTP server ends up as an HTTP request to your API, with the rows already parsed, mapped to your schema and validated, as JSON. It takes six steps: receive the file, parse it, map it once per client, validate every row, deliver a signed JSON event, and verify that signature on your side. This guide walks through each step, shows the payload and the signature check, and ends with the build or buy call.

To see steps two to four on your own data first, drop one real client file here and watch it get mapped, checked and turned into JSON in your browser.

Why SFTP to webhook is harder than it sounds

Most SFTP tooling stops at the transfer. Managed file transfer products can tell you a file landed and some fire a notification with the file name and path. That is useful, but it hands your application a pointer to raw bytes, not data. Someone still has to download the file, work out the layout, map the columns, validate the rows and push the result into your system.

The hard part is that every client's file is different. One sends UTF-16 with a byte order mark, another wraps everything in quotes, a third renames a column in October without telling anyone. A pipeline that worked on the sample file in the kickoff call breaks on week two's real file. This post is one layer of the bigger picture in Recurring client file feeds: the complete guide: not moving the file, but understanding it.

How to go from SFTP to webhook in six steps

The steps are the same whether you build them or buy them. Each one has a failure mode that only shows up with real client files, so each section names it.

1. Receive the file

There are two ways to notice a new file. If you host the SFTP server, give each client their own credentials and folder, and let the server emit an event on upload. No polling, no missed windows. If the client hosts the server, you connect on a schedule, list the directory, and diff against what you have already processed. Polling needs idempotency: track processed files by name plus size or checksum, because clients re-upload corrected files under the same name more often than you would expect.

Watch for partial uploads. A large file appears in the directory before the client's upload finishes. A naive pipeline grabs it mid write and parses half a file. The fixes are an upload complete marker, a size stability check across two polls, or an atomic rename convention agreed with the client. Clients without an IT team often cannot run an SFTP client at all; for them, an inbound email address with a sender allow list is the same step over a different channel.

2. Parse defensively

Real client files are messier than any spec. Delimiters vary, encodings vary, the header row appears on line three under a title block, and Excel files carry merged cells and hidden sheets. A production parser needs encoding detection, delimiter sniffing, header row detection, and a policy for blank lines and trailing garbage. Numbers arrive as text with thousands separators; dates arrive as Excel serial numbers. Our post on common CSV import errors is the catalogue of what breaks here.

3. Map to your schema, once per client

Mapping is where file based integrations differ from API integrations. With an API the contract is fixed. With files the contract is whatever each client's system exports, so the mapping is defined per client: their EMP_NO is your employee_id, their DOB needs a date format conversion, their status codes need normalising to your list.

The pattern that works: suggest the mapping from the header row and a sample of values on the first file, have a person review and confirm it, then apply the exact same mapping to every file that follows. The confirmation step is what stops a silently renamed column from flowing into production. A required field with no mapped source should block the feed from going live, not fail quietly at 2 a.m. More on the suggestion side in how auto suggested field mapping works.

4. Validate every row before you deliver

The webhook is only as trustworthy as the validation in front of it. Row level checks (required fields, types, date formats, allowed values, uniqueness of IDs) catch bad records. File level checks (row count within an expected range, mandatory columns present) catch bad files. The output should be a readable error report: row 412, field email, value j.smith@, reason "not a valid email". That report is what you send the client so they fix the export instead of guessing.

Decide what happens on failure before the first real file. Reject the whole file, deliver the good rows and report the bad ones, or hold for review. For recurring feeds, delivering good rows and reporting exceptions usually beats all or nothing, because one malformed row should not stop payroll data for four thousand people. Transforms belong here too: trim whitespace, lower case emails, turn text into numbers, normalise dates, so that a value that is correct but in the wrong shape does not count as an error.

5. Deliver a signed JSON event

Delivery has its own checklist. Sign the payload so the receiver can verify it came from your pipeline. Include a stable delivery id so consumers can deduplicate. Retry with backoff on anything other than a 2xx. Keep a delivery log you can replay from. For large files, send an event that identifies the run and lets the receiver fetch the records, rather than a fifty megabyte POST body that times out halfway.

Here is the shape FileFeed sends when a file has been processed. The headers carry the signature, the event type and a delivery id; the body is a JSON envelope.

POST /webhooks/filefeed HTTP/1.1
Content-Type: application/json
X-FileFeed-Event: FILE_PROCESSED
X-FileFeed-Delivery-ID: 7f3c9a2e-1b4d-4e8a-9c6f-2d5e8b1a0c47
X-FileFeed-Signature: 9b1f3c...e2a7

{
  "event": "FILE_PROCESSED",
  "timestamp": "2026-10-01T02:14:09.000Z",
  "workspaceId": "ws_01HZX4",
  "data": {
    "clientName": "Acme Benefits",
    "filename": "employees_2026-10-01.csv",
    "pipelineRunId": "run_01J9K2",
    "pipelineName": "Acme employee roster",
    "recordCount": 4120,
    "status": "completed"
  }
}

Your endpoint reads the event, verifies the signature (step six), and then fetches the clean records for that pipelineRunId from the FileFeed API, in your schema, page by page, and upserts them. Failed rows are not in that set; they sit in the run's error report. A FILE_PROCESSING_FAILED event uses the same envelope and tells you to look at the run instead.

6. Verify the signature on your side

The signature is an HMAC-SHA256 of the raw request body, using the webhook secret you were shown once when you registered the endpoint, encoded as hex. Verify it over the raw bytes, not over a re-serialised object, because any change in key order or whitespace produces a different digest. Compare with a constant time function. In Node:

const crypto = require('crypto');
const express = require('express');

const app = express();

// keep the raw body; the signature is computed over these exact bytes
app.use(express.json({
  verify: function (req, res, buf) { req.rawBody = buf; }
}));

function verifySignature(rawBody, header, secret) {
  const expected = crypto
    .createHmac('sha256', secret)
    .update(rawBody)
    .digest('hex');
  const a = Buffer.from(expected, 'utf8');
  const b = Buffer.from(header || '', 'utf8');
  return a.length === b.length && crypto.timingSafeEqual(a, b);
}

app.post('/webhooks/filefeed', function (req, res) {
  const ok = verifySignature(
    req.rawBody,
    req.get('X-FileFeed-Signature'),
    process.env.FILEFEED_WEBHOOK_SECRET
  );
  if (!ok) return res.status(401).end();

  if (req.body.event === 'FILE_PROCESSED') {
    // fetch records for req.body.data.pipelineRunId, then upsert
    queueImport(req.body.data.pipelineRunId);
  }
  res.status(200).end();
});

Three habits keep this safe over time. Return 200 quickly and do the import in a background job, so retries do not pile up behind a slow database. Store the delivery id and ignore a repeat. Rotate the secret by registering a new webhook, switching your config, then deleting the old one.

Keep an audit trail of every run

For each run you want: which file, which mapping version, how many rows in, how many passed, how many failed and why, and what was delivered. When a client asks why a record is missing, the answer should be one search away. When a mapping was wrong, you should be able to fix it and run the same file again without asking the client to resend. In FileFeed this is the Runs screen: the original file, the error report, the processed output and a reprocess button, per client.

Build or buy

Everything above is buildable in house, and for one stable feed from one client it is a reasonable week of work. The curve bends when feeds multiply: ten clients means ten mappings to maintain, ten flavours of malformed input, and a steady stream of "the file looks different this month" tickets landing on engineers. That maintenance load, not the initial build, is what pushes teams to a platform.

FileFeed does this exact path as a product. Clients drop CSV or Excel files on SFTP we host or on their own server, or email them to an address with a sender allow list. Mapping is suggested from the file and confirmed once per client. Every row is validated against your schema with built in transforms for trim, email format, numbers and dates. Clean JSON reaches your endpoint through the HMAC signed webhook above, with retries and a full run history. It costs $700 per month, flat, for up to 20 client connections, billed monthly, with no per row pricing. Hellora, an employee referral platform (formerly EQORefer), runs feeds from more than 30 HR systems on it.

Frequently asked questions about SFTP to webhook

What does SFTP to webhook mean?

It means a file dropped on an SFTP server is turned into an HTTP request to your API, with the contents already parsed, mapped and validated into JSON. Your application never has to know about SFTP, CSV parsing or the client's column names. The webhook is signed so your backend can check it came from the pipeline.

Can I do SFTP to webhook with just an SFTP server and a cron job?

Yes, for one or two clients. The cron job lists the folder, picks up new files, runs a parser, applies a mapping table and POSTs the result. What it will not have on day one is partial upload handling, a per client mapping you can change without a deploy, a readable error report, signed delivery with retries, and a run history. Those are the parts that cost time once you have ten clients.

Why does the webhook carry a run id instead of all the rows?

Because files are big and HTTP timeouts are short. A 4,000 row roster is fine in a body; a 400,000 row inventory file is not. Sending an event that identifies the run, then fetching the records page by page, works for both and lets you retry a page instead of the whole delivery. It also keeps the signature check cheap.

How do I verify a webhook signature?

Compute HMAC-SHA256 over the raw request body with your webhook secret, hex encode it, and compare it to the signature header using a constant time compare. Capture the raw body before any JSON parsing, because re-serialising the object changes the bytes. Reject with 401 if it does not match, and store the delivery id so a retried event is processed once.

What happens when a client renames a column?

If the mapping is confirmed per client and a required field now has no source column, the run fails validation with a clear message and nothing is delivered. You update the mapping for that client, reprocess the same file, and the webhook fires with the clean result. If the renamed column was optional, the rows go through with that field empty and the error report shows it, so you can still decide to fix the mapping.

What if the client cannot use SFTP?

Give them an inbound email address with a sender allow list. The attachment goes through the same parse, map, validate and deliver steps, and your webhook does not change. Clients without an IT team often prefer it, and the allow list keeps anyone else from submitting a file for that client.

Try it on one real file

The fastest way to see steps two to four is to drop one real client file into the browser demo: the mapping, the failing rows and the clean JSON appear in about a minute, with no account. When you are ready for the full path from SFTP drop to signed webhook, Automated FileFeeds is the product version of this post, or book a demo and bring the file.