FAQ

Questions, answered

How Cicada works for the teams who send us data and for the raters who judge it. Specific answers, taken from how the platform actually behaves.

For AI teams

What we do, what to send us and what comes back.

What does Cicada do?

We run human evaluation of model output. You send prompts, each with two candidate responses. Vetted raters compare the two against a rubric agreed with you, QA reviewers check their work, and you get back preference pairs and a full record of every judgement, with the quality figures behind them.

Link to this answer
How does a project run?
  • We agree the rubric and the written guidelines raters will follow.
  • We set up the project: pay per rating, a time limit per task, what share of each batch is rated by several people, a qualification exam if you want one, and a minimum rater level for harder batches.
  • You send batches; we upload them and qualified raters work through them one task at a time.
  • QA reviewers pass, return or fail completed ratings. Returned and failed work always comes with a written reason.
  • We export the results and send them to you with the batch’s quality report.

See How it works for the detail of each stage.

Link to this answer
What format should my data be in?

A CSV file (UTF-8, with a header row). The columns are:

  • prompt: required, and must not be empty;
  • response_a and response_b: the two responses to compare;
  • gold_answer: optional. Fill it in for rows you already know the answer to, so they can be used as hidden gold tasks. It must be one of the seven preference labels exactly, from “A is much better” to “B is much better”.

Any other columns are ignored. Excel files aren’t supported, so save spreadsheets as CSV first. A single file can be up to 25 MB and 200,000 rows. We can send you a sample CSV to start from.

Link to this answer
What do I get back?

Two exports per batch:

  • Full record (CSV): one row per task in the batch, finished or not, with task_uid, prompt, response_a, response_b, status, rater_username, time_taken_seconds, overall_rating, overall_reasoning, qa_verdict, qa_comment, answers_json (the rubric answers) and responses_json.
  • Preference pairs (JSONL): one line per completed task with a clear preference, with task_id, prompt, chosen_response, rejected_response, reward_score (1 to 3, by strength of preference) and human_justification. Ties are left out, and so is work that QA failed or sent back. There is no rater identity in this file.

Each row carries its QA verdict, so you can see what has been reviewed. If you want only reviewed work, we export once review is complete. Alongside the data we send the quality report: gold accuracy per rater, agreement on overlapping work, and gold coverage.

Link to this answer
Do I get a log-in to see progress?

Not yet. There’s no client dashboard; we send exports and quality reports to you directly, as often as you need them.

Link to this answer
How do you control quality?
  • Gold tasks: rows with a known answer are mixed invisibly into ordinary work. Raters are scored on them with partial credit for near misses, so accuracy is measured on work nobody knew was being marked.
  • Overlap: a share of each batch (5% unless you choose otherwise) is rated independently by several raters (3 unless you choose 2 to 10). We report the majority verdict, pairwise agreement and Fleiss’ kappa, and flag groups where raters disagree rather than averaging them.
  • QA review: reviewers pass, return or fail ratings, with a written reason for returns and fails. A failed rating is removed, its pay is reversed and the task goes back to other raters.
  • Qualification exams: when a project requires one, raters must reach the pass mark (70% unless you set another) before they get live work.
  • Levels: batches can be restricted to raters at or above a level. Levels are set by our team or, if you want, automatically from gold accuracy and QA pass rate.
Link to this answer
How fast is it, and what does it cost?

Both depend on the task: how long and hard each judgement is, what expertise or language it needs, how much overlap and review you want, and how many raters we put on it. We quote each project, with turnaround, after seeing a sample. See Pricing.

Link to this answer
Who sees my data?

Only the people working on your project. Raters and QA reviewers see only the project they are assigned to, and they agree to keep task content confidential: no copying, sharing or pasting it into other tools. Your data is stored in India (Google Cloud, Mumbai), and uploaded files are read into the database and not kept as files. We use your content only to run your project. If you need an NDA or a data processing agreement, ask us when you get in touch.

See Security for how it is protected and who can access what.

Link to this answer
How do you know raters aren’t using AI?

Raters agree not to use AI tools to make or write their judgements, and breaking that is grounds for removal and reversed pay. Hidden gold tasks, agreement checks and QA review catch work that doesn’t hold up. Rating sessions also require a screen share, though the video stays in the rater’s browser and isn’t watched or recorded, so it is a commitment rather than surveillance.

Link to this answer

For raters

Joining, rating, getting reviewed and getting paid.

How do I join?

Create an account with a username, an email address and a password. Then you fill in a short application: your education, work, languages, the subjects you know well and your availability. It saves as you go, and you can edit it any time.

Link to this answer
What happens after I sign up?

First, you take the onboarding course: short films, reading and checks covering the work, how it’s checked, how you’re paid and the rules, ending with you signing the rules. Your progress is saved, and the exam and live work open when you finish.

An admin adds you to a project. Until then there’s nothing to rate on your dashboard. When you’re added, you get a notification in the app, and your dashboard shows the project’s pay per task and, if the project requires one, its exam.

Link to this answer
Can I skip the onboarding course?

No. Everyone who rates takes it once, and nothing in it can be skipped: films can’t be fast-forwarded, readings stay open for a minimum time, and every check has to be answered correctly. You can stop and come back whenever you like, and revisit any part of it later from your sidebar.

Link to this answer
Is there an exam?

Only if your project requires one. If it does:

  • it is scored against known answers, with partial credit for near misses;
  • you see your score at the end, not question by question, and pass at or above the project’s pass mark (70% unless the project sets another);
  • it is unpaid and untimed, and doesn’t need a screen share;
  • if you don’t pass, you can retake it after 24 hours, up to 3 attempts in all; each retake starts fresh;
  • once you pass, you are qualified for that project.
Link to this answer
How does rating work?

Start a session, and you are given one task at a time: a prompt and two responses. Score each response against the rubric, write your reasoning, choose which response is better on a seven-point scale, and submit. Every rubric field has to be answered. Then the next task comes up.

If you can’t finish a task you can release it, but your answers on it are dropped. Answers aren’t saved until you submit.

Link to this answer
Is there a time limit?

Yes. Each task has a time limit set by the project, 30 minutes unless the project sets another. There’s no countdown on screen, so if you need a break, submit or release the task first. If you run past the limit, the task goes back to the pool, your answers on it are discarded and you get a notification. Time is measured by our server.

Link to this answer
Why do I have to share my screen, and what is recorded?

To start a rating session, your browser asks you to share your entire screen. Nothing is recorded: the video stays in your browser and is never uploaded, watched or stored, and no screenshots are taken.

What we log for each session:

  • when it started and ended, and how it ended (for example, you stopped sharing);
  • what your browser reported you shared: whole screen, window or tab;
  • a signal from your browser every 30 seconds while it runs;
  • how many tasks you submitted.

Admins, and project managers for their projects, can see these logs. If you stop sharing, the session ends; unsubmitted answers on the open task are lost, and that task stays with you until its time limit runs out. Exams don’t use screen sharing.

Link to this answer
How much does it pay, and when?

Each project pays a flat amount per rating, set by us and shown on your dashboard and before you start a session. It’s the same for every task on the project, whatever your level and however long the task takes. The amount is credited to your balance as soon as you submit. Gold tasks look like any other task and pay the same. Exam questions are unpaid.

Link to this answer
What happens when QA reviews my work?
  • Passed: nothing changes. You keep the pay.
  • Needs rework: it comes back to you with the reviewer’s reason. You must fix it before you get new work. You keep the original pay, and the fix isn’t paid again.
  • Failed: the rating is removed, its pay is reversed on your ledger with the reviewer’s reason, and the task goes to other raters. You can appeal.

A reviewer can also correct a rating directly. That marks it passed and doesn’t change your pay.

Link to this answer
How do appeals work?

Open the failed rating and explain why you think the decision was wrong (10 to 5,000 characters). There’s no deadline, and you can appeal each failed rating once. An admin or the project’s manager decides, and you are notified either way. If your appeal is upheld, exactly the amount that was reversed is credited back. The rating itself isn’t reinstated.

Link to this answer
What are levels?

Everyone starts at level 1. Some batches are open only to raters at or above a certain level. Levels don’t change your pay.

Our team can set your level. On projects with automatic tiering switched on, your level is recalculated each time QA reviews your work, from your gold accuracy (60%) and QA pass rate (40%), once there are at least five results to go on. The thresholds are 50% for level 2, 70% for level 3, 85% for level 4 and 95% for level 5, and you move one level at a time, up or down.

Other raters on your project can see your username, level and task count on the leaderboard.

Link to this answer
How do I withdraw my earnings?
  • Go to Earnings and add the name on your bank account, the account number, IFSC code and your PAN. A UPI ID is optional. Changing these details needs your password.
  • Request any amount from ₹1 up to your available balance.
  • The request goes from requested to approved to paid, or is rejected. You are notified at each step.
  • Our team sends the money outside the platform, to the bank details on your request, then marks it paid. Your balance goes down only when it is marked paid.
Link to this answer
Are there fees? What about tax?

Cicada charges no fees. Your bank may charge its own.

We ask for your PAN so we can meet any tax reporting or deduction the law requires of us, but we don’t deduct tax today. You are an independent contractor, and you are responsible for declaring and paying tax on what you earn. See the Rater Agreement.

Link to this answer
Can my balance go below zero?

Yes, if a rating is failed after you have already withdrawn its pay. Your next earnings make up the difference before you can withdraw again. Every movement is listed on your ledger with its reason.

Link to this answer

Account & security

Passwords, sessions, cookies and your data.

I’ve forgotten my password.

Use Forgot password and enter the email address on your account, exactly as you saved it. We email a link that works once and expires after an hour. Resetting signs you out everywhere, and we email you to confirm.

If your account was created by an admin, it may not have an email address yet. Add one in account settings while you can still sign in. If you’re already locked out, contact us.

Link to this answer
How do I add or change my email address?

In account settings. You’ll need to enter your password.

Link to this answer
How long do I stay signed in?

Up to seven days on each device. Signing out, changing or resetting your password, or having your account deactivated ends all of your sessions on every device.

Link to this answer
What cookies do you use?

One: access_token, which keeps you signed in. The app also keeps a few small flags in your browser, with no personal data in them, to remember that you finished the tour, hid a checklist or dismissed the rating tips. No analytics or tracking. See the Cookie Policy.

Link to this answer
How do I see or delete my data?

Email [email protected] from the address on your account and tell us your username. We can send you a summary of what we hold, correct it, or close your account and delete it. Records of work done and money paid are kept as the law requires. See the Privacy Policy.

Link to this answer
How do I report a security problem?

Email [email protected] with “Security” in the subject. Our disclosure policy explains what to include.

Link to this answer

Didn’t find your answer?

Write to us at [email protected], or use the contact form.

Contact us