Recurring client file feeds: the complete guide

A recurring client file feed is a file that one of your clients sends you on a schedule, in their own format, that your product has to turn into clean records every time it arrives. This guide is for implementation, onboarding and integration teams at B2B software companies that receive these files over SFTP or email and still handle them by hand. It covers why clients send files instead of calling your API, the six parts every feed needs, build versus buy, setup step by step, and which tools do what.

If you would rather see it on your own data first, drop one real client file here and watch it get mapped, checked and turned into JSON in your browser.

What a recurring client file feed is

Picture an HR platform with two hundred employer clients. Every night, each employer's HR system exports a roster of employees and drops it on an SFTP server, or someone in HR emails it as an attachment. The platform needs those rows in its database by morning, in one consistent shape, no matter which HR system they came from. That is a recurring client file feed: one client, one file type, one schedule, repeated for as long as the client stays.

The word recurring is what separates this from most things people call "CSV import". Three things look similar and are not the same problem.

Not a one time CSV import

A one time import is a person uploading a spreadsheet into your app, fixing the columns in a UI and clicking submit. Embedded importers are built for this: the person is present and can answer questions about the data. A recurring feed has no person in the loop at 2 a.m. when the file lands. The mapping decisions are made once, up front, and applied to every file that follows. If a tool asks a human to match columns on every upload, it is an importer, not a feed.

Not an API integration

An API integration means the client's system calls your endpoint, or you call theirs, with JSON both sides agreed on. That is the cleanest option when it exists. In enterprise onboarding it rarely does. The client's HR, payroll or inventory system has a scheduled export, not an outbound integration to your product. Building one needs their IT team, a security review and a budget line. A file feed uses the export they already have.

Not plain file transfer

SFTP hosting and routing tools such as Couchdrop, AWS Transfer Family or Files.com solve the transfer part: a login and a folder per client, and a notification when a file arrives. What they hand you is the raw file. The columns are still named whatever the client's system named them, dates are still in three formats, and nobody has checked whether row 412 has an email address. A feed includes the transfer and everything after it. For the longer version of that gap, read what file ingestion actually involves.

Why clients still send files

Files are how established systems talk to the outside world. Workday, ADP, UKG, BambooHR, Paylocity and most custom HR systems ship with scheduled report exports. Payroll providers export pay period files. Benefits carriers and TPAs exchange eligibility and census files. Retailers and distributors send inventory and price lists. Logistics partners send carrier files. The export is already built, scheduled and approved by someone's security team. The file is the path of least resistance for the client, so it is the path you get.

"Just use our API" sounds reasonable from inside a software company. From the client's side it means: find an engineer, read your docs, build and test an integration, and get it through change control. In practice the request sits in a backlog, the go live date slips, and the implementation team accepts a file anyway, just later and under more pressure.

This is also why the problem scales badly. With five clients, five custom parsers are fine. With fifty, each new client is a small project, each format change is an incident, and the implementation team becomes the bottleneck for revenue. Hellora, an employee referral platform (formerly EQORefer), onboards clients from more than 30 HR systems this way. Their story shows what the per client work looks like before and after a feed.

The six parts of a feed

Whatever tool you use, a working feed has the same six parts. If one is missing, that part is being done by a person, usually by email, usually late.

1. Intake: SFTP or email

Intake is how the file reaches you. The two channels enterprise clients actually use are SFTP and email. SFTP comes in two flavours: you host a server and give each client their own credentials and folder, or the client hosts the server and you connect on a schedule and pull. Email means the client, or their system, sends the file as an attachment to an address you own, and you accept it only from senders you have approved. Good intake isolates each client, restricts who can send, records when every file arrived and triggers processing automatically. It also has to cope with a file still being written when you first see it, and the same file sent twice.

2. Per client mapping

Mapping says which column in the client's file goes to which field in your schema. The same field arrives under a different name from every system: your employeeId is Employee ID in one export, EmpID in another, Associate Number in a third and Worker_ID in a fourth. The mapping belongs to a client, not to a file, so it is built once at onboarding and reused on every file after that. A good tool suggests the mapping from the header row and a sample of values, you confirm or correct it, and it is saved against that client. A required field that is not mapped should block the feed from going live, not fail quietly at run time.

3. Validation

Validation checks every row against your schema before anything is delivered. Required fields are present. Emails look like emails. Dates parse. IDs are unique within the file. Values that must come from a list actually do. The output is a row level error report: row 412, field email, value j.smith@, reason "not a valid email". That report is the most useful artifact in the whole feed, because it is what you send to the client so they can fix their export instead of guessing. For a catalogue of what goes wrong, see common CSV import errors.

4. Transformation

Transformation fixes values that are correct but not in your shape. Dates are the classic case: 03/04/2026 is 4 March in one country and 3 April in another, 2026-03-04 is unambiguous, and an Excel export may give you the serial number 46085 instead of a date. Enums are the other one: employment status arrives as Full-time, FT, Full Time or 1 and your system wants full_time. Then trailing spaces in IDs, emails in mixed case, numbers stored as text. In a well built feed these are small named functions (trim, format email, to number, normalise date) attached to a field in the mapping, not custom code per client.

5. Delivery: JSON to your API

Delivery is where the clean records reach your product. The pattern that works is a webhook: when a file has been processed, your endpoint receives a signed request with the result, in your schema, as JSON. Signed means the request carries an HMAC signature made with a secret only you and the feed platform know, so your backend can verify it came from the feed and was not altered. Your side is small: one endpoint, one signature check, one upsert. The SFTP to webhook guide walks through the payload and the signature check.

6. Monitoring and error handling

This is the part most home built feeds skip. For every client you need to know when the last file arrived, how many rows passed and failed, and what was delivered. You need to notice when an expected file does not arrive. And when something fails, you need to fix the mapping and run the same file again without asking the client to resend it. A run history with the original file, the error report and the delivered output, kept per client, turns "the import broke" into a five minute fix.

Build or buy

Most teams start by building, because the first feed looks small. Here is what the build contains once it has to run unattended for real clients.

• An SFTP server with per client users, folders and key management, or a mailbox you poll for attachments, plus a provisioning script so you are not creating users by hand.
• A watcher: a cron job or function that notices new files, waits for uploads to finish, de-duplicates and queues them.
• Parsers for CSV with every delimiter and encoding you will meet, and for XLSX with multiple sheets and header rows that do not start on line one.
• A mapping table per client, with a way to edit it that does not require a deploy.
• Validation rules that produce a readable error report, not a stack trace.
• Transformation functions for dates, enums, numbers and strings, applied per field per client.
• Delivery with retries and idempotency, so the same file never creates the same records twice.
• Alerting and a run log, with a way for a non engineer to see what happened and re-run a file.

Rather than quote an industry number, do the arithmetic for your own team. Write down three figures: the engineering hours to onboard one new client format today, the number of new clients you expect next year, and the hours you spend each month on broken or changed files from existing clients. Multiply the first two, add twelve times the third, and you have the annual cost of your feed layer in engineering hours. The first version of a script is a week. Keeping fifty of them alive is a job.

Building makes sense in three situations: you have one or two clients and no plan to add more; the files are so unusual that no general tool will parse them; or file ingestion is your product. If none of those apply, you are buying the six parts above as a service and keeping your engineers on the product your clients pay for. Our data onboarding tools comparison goes through the market by use case.

How to set up a recurring client file feed, step by step

The order below works with any tool, including a home built one. It is also the order of FileFeed's setup wizard, described after the generic version.

1. Connection. Decide how this client's files will reach you. Create their isolated SFTP space, or connect to their server, or set up the inbound email address and approve their sender. Then ask for one real file.
2. Schema. Define the shape every file should end up in: field names, types, which fields are required, which must be unique, which must come from a fixed list. This is your contract. Keep it as small as the product needs.
3. Pipeline and mappings. Open the client's sample file against the schema. Map each source column to a schema field. Attach transforms where values need cleaning. Make sure every required field is mapped before you continue.
4. Webhook. Register the endpoint that should receive the results, store the signing secret, and verify the signature on your side. Test with the sample file before the client sends anything on a schedule.
5. Runs. Let the first real file through. Read the run: rows in, rows passed, rows failed and why. Send the error report to the client if their export needs a fix.

How this looks in FileFeed

In FileFeed, those five steps are the five screens of the wizard. A Connection is a hosted SFTP folder with its own credentials and IP allow list, a connection to the client's own SFTP server with password or key auth, or a dedicated email address with a sender allow list. The Schema screen defines fields, types, required and unique constraints. In Pipeline and Mappings, FileFeed reads the client's sample file, auto suggests the column mapping, and you confirm it once; the pipeline will not go live until every required field is mapped. Built in transforms such as trim, formatEmail, toNumber and date normalisation are attached per field. The Webhook screen registers your endpoint and shows the secret once; every delivery is HMAC signed and carries the clean JSON in your schema. Runs is the history: every file, every row level error, the original and processed output, and a reprocess button for when you fix a mapping.

FileFeed accepts CSV and Excel (XLSX) files over all three intake channels, and the whole run history is kept as an audit trail. To see steps two and three on your own data before creating an account, try it in the browser with a real file.

Tools for recurring client file feeds

The market splits into tools that move files, tools built for one time imports, and tools built for recurring feeds. A short, honest read on the ones that come up most, as of October 2026.

For side by side detail, see the alternatives hub, or the comparisons with OneSchema, Flatfile and Couchdrop. If your feeds are HR rosters or benefits eligibility files, the HR tech and benefits pages show what the schema and the typical client exports look like in those two cases.

Checklist for your first feed

Ten things to have in hand before you call a feed live. Most failed first feeds are missing one of the first four.

1. One real file from the client, not a template or a sample they made for you.
2. A named contact on the client side who can change the export if a column is missing.
3. Your target schema written down, with required fields marked and kept to what the product uses.
4. The schedule: how often the file arrives, at what time, in which time zone, and what "late" means.
5. The intake channel tested end to end: credentials delivered, a test file received, sender approved if it is email.
6. The mapping confirmed against the real file, including every required field.
7. Transforms attached for dates, enums, IDs and emails, checked against actual values from the file.
8. The webhook endpoint deployed, signature verification in place, and an upsert key chosen so a reprocessed file does not duplicate records.
9. A decision on failing rows: who gets the error report, and whether the client fixes the export or you adjust the mapping.
10. Someone watching the run history for the first week, and an alert for a file that does not arrive on schedule.

Frequently asked questions about recurring client file feeds

How do I automate SFTP file imports from clients?

Give each client their own SFTP credentials and folder, detect new files automatically, map the client's columns to your schema once, validate every row, transform values into your format, and deliver the clean records to your application over a webhook. You can build those pieces with a managed SFTP server, a cron job and parsing code, or use a feed platform that provides all six and lets a non engineer add a client from a dashboard.

What does SFTP to webhook mean?

It means a file dropped on an SFTP server ends up as an HTTP request to your API, with the contents already parsed, mapped and validated into JSON. The webhook is signed so your backend can verify the sender, and your application never has to know about SFTP, CSV parsing or the client's column names. The full walkthrough, including the payload and signature check, is in our SFTP to webhook guide.

File based integration or API: which should I ask clients for?

Ask for whichever the client can deliver this month. An API integration is cleaner when both sides already have one. For most enterprise clients the scheduled export already exists and the API integration does not, so insisting on the API delays go live. A recurring file feed with validation and JSON delivery gives your application the same clean records it would get from an API, while the client keeps using the export they already trust.

How much does a recurring file feed cost?

If you build it, the cost is engineering hours per new client plus the ongoing time spent on changed or broken files, which grows with the number of clients. FileFeed charges $90 per client connection per month, billed annually, after a $3,000 pilot with a 30 day money back guarantee, with no per row pricing, so the cost does not change when a client sends a bigger file. Details are on the pricing page.

Can clients send files by email instead of SFTP?

Yes. Clients without an IT team often find it easier to email an attachment than to set up an SFTP client. A feed platform gives you a dedicated inbound address and a sender allow list, so only approved addresses can submit files for that client. The attachment then goes through the same mapping, validation and delivery as a file dropped on SFTP. FileFeed supports both channels on the same pipeline model.

What happens when a row fails validation?

The row is recorded in an error report with its row number, the field, the value and the reason it failed. You see the report in the run history and can send it to the client so they fix the export, or adjust the mapping or a transform on your side and reprocess the same file without asking for it again. Nothing is silently dropped or silently loaded.

How long does setup take?

With a real sample file and your schema written down, configuring the first feed in FileFeed is a five step wizard: connection, schema, mapping, webhook, first run. The mapping and transforms take minutes because they are suggested from the file and confirmed, not coded. The part that takes longest is usually on your side: deploying the webhook endpoint and choosing your upsert key.

Do clients need to change their export format?

No. The mapping and transformation layer exists so the client can keep sending the export their system already produces. You absorb the column names, date formats and enum values on your side, once per client. The only time you go back to the client is when a required field is missing from their export entirely, and the validation error report tells you exactly which one.

Start with one file

The fastest way to understand your own feed problem is to take one real client file and run it through all six parts. Drop it into the browser demo to see the mapping, the failing rows and the clean JSON in a minute, with no account. When you are ready to run it on a schedule, Automated FileFeeds is the product version of everything in this guide, at $90 per client connection per month, after a $3,000 pilot with a 30 day money back guarantee. Or book a demo and bring the file with you.