## lamindb.Collection

| class lamindb.Collection(artifacts: Artifact | list[Artifact], key: str, description: str | None = None, schema: Schema | None = None, meta: Any | None = None, reference: str | None = None, reference_type: str | None = None, run: Run | None = None, revises: Collection | None = None, skip_hash_lookup: bool = False, branch: Branch | None = None, space: Space | None = None) |
class lamindb.Collection(*db_args)

 Bases: "SQLRecord", "IsVersioned", "TracksRun", "TracksUpdates"

 Versioned collections of artifacts, such as sharded datasets across
 many parquet files or zarr stores.

 Use a collection when several artifacts should behave as one
 dataset — one version history, and usually one schema and one query
 surface. Through the ".append()" method, you can add new artifacts
 to a collection in an ACID way. You can also time-travel to
 previous versions of the collection.

 You can build collections with tens of thousands of parquet files
 or zarr stores. This allows storing very high numbers of
 observations, for example 10k parquet files with 100 million rows
 each allow storing a trillion observations.

 Collections are particularly useful if they enforce a common schema
 for their artifacts: pass "schema" to achieve this. You can then
 confidently use "collection.open()" to open a collection of parquet
 files directly with Polars or PyArrow as you'll know that the
 columns of these parquet files will harmonize. Or, analogously, you
 use "collection.mapped()" to access the collection with the
 "MappedCollection" sampler for AnnData objects.

 For all other accessor patterns, you can use the raw file paths of
 the artifacts inside the collection, for example, using DuckDB:

 import duckdb

 con = duckdb.connect()
 s3_paths = [a.path.as_posix() for a in collection.artifacts.all()]
 con.execute(f"CREATE VIEW my_view AS SELECT * FROM read_parquet({s3_paths})")

 Or with "annbatch", you can sample from a collection of ".zarr"
 stores:

 import anndata as ad
 import zarr
 from annbatch import Loader

 paths = [artifact.cache() for artifact in collection.artifacts.all()]
 loader = Loader(shuffle=True, batch_size=4096, chunk_size=256, preload_nchunks=64)
 loader.add_datasets(
 datasets=[ad.io.sparse_dataset(zarr.open(p)["X"]) for p in paths],
 obs=[ad.io.read_elem(zarr.open(p)["obs"]) for p in paths],
 )
 for batch in loader:
 pass

 If you already know lakehouse tables (Iceberg, Delta Lake,
 DuckLake), LaminDB's "Collection" is the analogous concept: many
 files as one dataset, with a shared schema, ACID appends, and time
 travel. See Architecture.

 Parameters:
| * **artifacts** -- "Artifact | list[Artifact]" One or several |
 artifacts.

 * **key** -- "str" A file-path like key, analogous to the "key"
 parameter of "Artifact" and "Transform".

| * **description** -- "str | None = None" A description. |

| * **schema** -- "Schema | None = None" A schema that every |
 artifact in the collection must satisfy.

| * **meta** -- "Artifact | None = None" An artifact that defines |
 metadata for the collection.

| * **reference** -- "str | None = None" A simple reference, e.g. |
 an external ID or a URL.

| * **reference_type** -- "str | None = None" The type of the |
 simple reference, e.g. ""url"".

| * **run** -- "Run | None = None" The run that creates the |
 collection.

| * **revises** -- "Collection | None = None" An old version of |
 the collection.

 * **skip_hash_lookup** -- "bool = False" Skip the hash lookup so
 that a new collection is created even if a collection with the
 same hash already exists.

| * **branch** -- "Branch | None = None" A branch. If "None", uses |
 the current branch.

| * **space** -- "Space | None = None" A space. If "None", uses |
 the current space.

 See also:

 "Artifact"
 Datasets & models stored as files, folders, or arrays.

 Organize datasets
 A basic guide on how to organize data.

# Examples

 Create a collection from a list of "Artifact" objects:

 collection = ln.Collection([artifact1, artifact2], key="my_project/my_collection")

 A sharded sequence corpus, e.g. a trillion observations as 10k
 parquet files of 100 million rows:

 artifacts = ln.Artifact.from_dir("s3://bucket/sequences/").save()
 collection = ln.Collection(artifacts, key="genomes/sequences").save()

| property transform: Transform | None |

 Transform whose run created the collection.

 property name: str

 Name of the collection.

 Splits "key" on "/" and returns the last element.

 property ordered_artifacts: QuerySet

 Ordered "QuerySet" of ".artifacts".

 Accessing the many-to-many field "collection.artifacts" directly
 gives you non-deterministic order.

 Using the property ".ordered_artifacts" allows to iterate
 through a set that's ordered by the order of the list that
 created the collection.

| property data_artifact: Artifact | None |

 Access to a single data artifact.

 If the collection has a single data & metadata artifact, this
 allows access via:

 collection.data_artifact  # first & only element of collection.artifacts
 collection.meta_artifact  # metadata

 uid: str

 Universal id, valid across DB instances.

 key: str

 Name or path-like key.

| description: str | None |

 A description or title.

| hash: str | None |

 Hash of collection content.

| reference: str | None |

 A reference like URL or external ID.

| reference_type: str | None |

 Type of reference, e.g., cellxgene Census collection_id.

| meta_artifact: Artifact | None |

 An artifact that stores metadata that indexes a collection.

 It has a 1:1 correspondence with an artifact. If needed, you can
 access the collection from the artifact via a private field:
 "artifact._meta_of_collection".

| schema: Schema | None |

 The validating schema of this collection ←
 "validated_collections".

 ulabels: RelatedManager[ULabel]

 ULabels annotating the collection (see "Feature") ←
 "collections".

 input_of_runs: RelatedManager[Run]

 Runs that use this collection as an input ← "input_collections".

 recreating_runs: RelatedManager[Run]

 Runs that re-created the record after initial creation ←
 "recreated_collections".

 artifacts: RelatedManager[Artifact]

 Artifacts in collection ← "collections".

 linked_in_records: RelatedManager[Record]

 This collection is linked in these records as a value ←
 "linked_collections".

 records: RelatedManager[Record]

 Linked records ← "collections".

 references: RelatedManager[Reference]

 Linked references ← "collections".

 projects: RelatedManager[Project]

 Linked projects ← "collections".

 ablocks: RelatedManager[CollectionBlock]

 Attached blocks ← "collection".

 get(*, is_run_input=False, **expressions)

 Get a single record.

 Parameters:
| * **idlike** (int | str | None, default: "None") -- Either a |
 uid stub, uid or an integer id.

 * **expressions** -- Fields and values passed as Django query
 expressions.

 Raises:
 **lamindb.errors.ObjectDoesNotExist** -- In case no matching
 record is found.

 Return type:
 Artifact

 See also:

 * Guide: Query, search & load data

 * Django documentation: Queries

 -[ Examples ]-

 record = ln.Record.get("FvtpPJLJ")
 record = ln.Record.get(name="my-label")

 append(artifact, run=None)

 Append an artifact to the collection.

 This does not modify the original collection in-place, but
 returns a new version of the original collection with the
 appended artifact.

 Parameters:
 * **artifact** (Artifact) -- An artifact to add to the
 collection.

| * **run** (Run | None, default: "None") -- The run that |
 creates the new version of the collection.

 Return type:
 Collection

 -[ Examples ]-

 collection_v1 = ln.Collection(artifact, key="My collection").save()
 collection_v2 = collection.append(another_artifact)  # returns a new version of the collection
 collection_v2.save()  # save the new version

 open(engine='pyarrow', is_run_input=None, **kwargs)

 Open a dataset for streaming.

 Works for "pyarrow" and "polars" compatible formats (".parquet",
 ".csv", ".ipc" etc. files or directories with such files).

 Parameters:
 * **engine** ("Literal"["'pyarrow'", "'polars'"], default:
 "'pyarrow'") -- Which module to use for lazy loading of a
 dataframe from "pyarrow" or "polars" compatible formats.

| * **is_run_input** ("bool" | "None", default: "None") -- |
 Whether to track this artifact as run input.

 * ****kwargs** -- Keyword arguments for
 "pyarrow.dataset.dataset" or "polars.scan_*" functions.

 Return type:
| "Dataset" | "Iterator"["LazyFrame"] |

 -[ Notes ]-

 For more info, see guide: Query arrays in storage .

 mapped(layers_keys=None, obs_keys=None, obsm_keys=None, obs_filter=None, join='inner', encode_labels=True, unknown_label=None, cache_categories=True, parallel=False, dtype=None, stream=False, is_run_input=None)

 Return a map-style dataset.

 Returns a pytorch map-style dataset by virtually concatenating
 "AnnData" arrays.

 By default ("stream=False") "AnnData" arrays are moved into a
 local cache first.

 "__getitem__" of the "MappedCollection" object takes a single
 integer index and returns a dictionary with the observation data
 sample for this index from the "AnnData" objects in the
 collection. The dictionary has keys for "layers_keys" (".X" is
 in ""X""), "obs_keys", "obsm_keys" (under "f"obsm_{key}"") and
 also ""_store_idx"" for the index of the "AnnData" object
 containing this observation sample.

 Note:

 For a guide, see Use machine learning data loaders.This method
 currently only works for collections or query sets of
 "AnnData" artifacts.

 Parameters:
| * **layers_keys** ("str" | "list"["str"] | "None", default: |
 "None") -- Keys from the ".layers" slot. "layers_keys=None"
 or ""X"" in the list retrieves ".X".

| * **obs_keys** ("str" | "list"["str"] | "None", default: |
 "None") -- Keys from the ".obs" slots.

| * **obsm_keys** ("str" | "list"["str"] | "None", default: |
 "None") -- Keys from the ".obsm" slots.

| * **obs_filter** ("dict"["str", "str" | "list"["str"]] |
 "None", default: "None") -- Select only observations with
 these values for the given obs columns. Should be a
 dictionary with obs column names as keys and filtering
 values (a string or a list of strings) as values.

| * **join** ("Literal"["'inner'", "'outer'"] | "None", |
 default: "'inner'") -- ""inner"" or ""outer"" virtual
 joins. If "None" is passed, does not join.

| * **encode_labels** ("bool" | "list"["str"], default: "True") |
 -- Encode labels into integers. Can be a list with elements
 from "obs_keys".

| * **unknown_label** ("str" | "dict"["str", "str"] | "None", |
 default: "None") -- Encode this label to -1. Can be a
 dictionary with keys from "obs_keys" if
 "encode_labels=True" or from "encode_labels" if it is a
 list.

 * **cache_categories** ("bool", default: "True") -- Enable
 caching categories of "obs_keys" for faster access.

 * **parallel** ("bool", default: "False") -- Enable sampling
 with multiple processes.

| * **dtype** ("str" | "None", default: "None") -- Convert |
 numpy arrays from ".X", ".layers" and ".obsm"

 * **stream** ("bool", default: "False") -- Whether to stream
 data from the array backend.

| * **is_run_input** ("bool" | "None", default: "None") -- |
 Whether to track this collection as run input.

 Return type:
 "MappedCollection"

 -[ Example ]-

 Create a Pytorch data loader from a collection:

 import lamindb as ln
 from torch.utils.data import DataLoader

 ds = ln.Collection.get(description="my collection")
 mapped = collection.mapped(obs_keys=["cell_type", "batch"])
 dl = DataLoader(mapped, batch_size=128, shuffle=True)

 # also works for querysets of artifacts, '...' represents some filtering condition
 # additional filtering on artifacts of the collection
 mapped = collection.artifacts.filter(...).order_by("-created_at").mapped()
 # or directly from a queryset of artifacts
 mapped = ln.Artifact.filter(..., otype="AnnData").order_by("-created_at").mapped()

 cache(*, is_run_input=None, mute=False, **kwargs)

 Download cloud artifacts in collection to local cache.

 Follows syncing logic: only downloads outdated artifacts.

 Returns ordered paths to locally cached on-disk artifacts via
 ".ordered_artifacts.all()":

 Parameters:
 * **mute** ("bool", default: "False") -- Silence logging of
 caching progress.

| * **is_run_input** ("bool" | "None", default: "None") -- |
 Whether to track this collection as run input.

 Return type:
 "list"["UPath"]

 load(join='outer', is_run_input=None, **kwargs)

 Cache and load to memory.

 Returns an in-memory concatenated "DataFrame" or "AnnData"
 object.

 Return type:
| "DataFrame" | "AnnData" |

 verify_schema()

 Verify all collection artifacts match the collection schema.

 Return type:
 "None"

 save(using=None)

 Save the collection.

 Parameters:
| **using** ("str" | "None", default: "None") -- The database |
 to which you want to save.

 Return type:
 "Collection"

 restore()

 Restore a collection from trash.

 Return type:
 "None"

 view_lineage(with_children=True, return_graph=False)

 View data lineage graph.

 Return type:
| Digraph | None |