LocalCloud Dataproc Local Emulator
Get started
local.cloud — Dataproc Local Emulator .md Open tab
LocalCloud Google Cloud in-a-box

Dataproc

Dataproc provides managed Spark and Hadoop execution on Google Cloud. LocalCloud runs supported cluster jobs and serverless batches through local Docker runtimes.

When to use Dataproc

Typical uses

Use when a pipeline uses Spark or Hadoop to transform partitioned datasets, and you need to submit jobs or serverless batches through the Dataproc API.

Examples

  • Spark ETL and data transformations
  • Hadoop and batch-processing tests
Analytics

Explore another service

View all
Local workflow

Example usage

Install the CLI, start LocalCloud, and export its environment in the same shell. Generated endpoints account for remapped ports.

1. Install and start · shell
curl -fsSL https://local.cloud/install.sh | sh
localcloud start --local-only
# Export GOOGLE_CLOUD_PROJECT and DATAPROC_EMULATOR_HOST
eval "$(localcloud env)"

Run a small Spark transformation

  1. Prepare an input fixture and a Spark job.
  2. Submit a supported cluster job or serverless batch locally.
  3. Inspect job status and verify the transformed output.

Documented workflows

  • Local cluster lifecycle
  • Spark/Hadoop execution in cluster and serverless modes

Service boundaries

  • Requires Docker daemon access
  • Cloud scaling, automated node repair, and production SLAs are cloud-only
Try it in your project

Code examples and integrations

Start with the local guides. Official Google Cloud samples may need endpoint changes and can cover features outside the workflows listed here.

Integration

Run Dataproc through local Docker

Execution setup, job lifecycle, and Docker runtime boundaries.

Open guide
Integration

Build an integration test loop

Start the runtime, verify readiness, exercise your SDK, and clean up test state.

Open guide
Agent testing

Use Dataproc with an AI agent

Before the agent changes application logic, have it create local Dataproc state, read it back with the project SDK, and note which feature the check covered. That first local call catches accidental production routing and needs no GCP account, service-account key, or billing project.

Configure this repo

Set up this repository to use LocalCloud for local GCP development. First read https://local.cloud/ai/agents.md, then inspect this repo, identify the GCP services and SDK language, configure emulator environment variables, and run the narrowest integration test against localhost. Do not use real GCP credentials or production endpoints.

Production endpoints, auth, IAM, quotas, regional behavior, and final release validation still need the target Google Cloud project.

Agent resources has more prompts and the AGENTS.md template.

Related guides

Desktop view