midZddlmZddlmZmZddlmZddlmZddl Z ddl m Z ddl mZd e jd efd Zd e jd e fd ZeGddZeGddZeGddZeGddZy)z' Basic configuration for Dora is here. ) Namespace) dataclassfield)fnmatch)PathN) DictConfig) OmegaConfdataargsc|jD]=}t|tsJt||s"t ||}|1t |||?y)z>Update the given dataclass from the argument parser args. N)__dict__ isinstancestrhasattrgetattrsetattr)r r keyvalues C/var/www/stems/demucs_env/lib/python3.12/site-packages/dora/conf.pyupdate_from_argsrsO}}*#s### 4 D#&E c5) *cfgctj|d}t|tsJ|j D]K\}}t|t sJt ||rt|||2td|jd|y)z6Update the given dataclass from the hydra config. T)resolvezObject of type z does not have an attribute N) r to_containerrdictitemsrrrAttributeError __class__)r rdctrrs rupdate_from_hydrar!s  d 3C c4  iikF U#s### 4  D#u % ?4>>2BC??Be"EF F FrceZdZUdZdZeed<dZeed<dZ eed<dZ eed <d Z e jeed <d Zeed <d Ze jeed<eeZe j(eed<dZeed<dZeed<dZeed<dZeed<d Ze jeed<d Ze jeed<d Ze jeed<y ) SlurmConfiga+ Configuration when scheduling a job. This differs slightly from Slurm/Submitit params because this will automatically scale some values based on the number of GPUs. Args: gpus (int): number of total GPUs to schedule. Number of nodes and tasks per nodes will be automatically inferred. mem_per_gpu (float): amount of memory in GB to schedule per gpus. time (int): maximum duration for the job in minutes. cpus_per_gpu (int): number of cpus per gpu, this will set the `cpus_per_task` automatically, based on the number of gpus and `one_task_per_node`, unless `cpus_per_task` is explicitely provided. cpus_per_task (int or None): number of cpus per task. partition (str): partition name comment (str): comment for the job. setup (List[str]): list of shell commands to execute before the actual command. Use it for `module load`. max_num_timeout (int): maximum number of requeue. one_task_per_node (bool): if True, schedules a single task per node, otherwise, will schedule one task per gpu (default is False). array_parallelism (int): when using job arrays, how many tasks can run in parallel. qos: (str or None): qos param for slurm. account: (str or None): account param for slurm. ..warning:: this assumes one task per GPU. Set `one_task_per_node` if you do not want that. Tasks without any gpus are not really supported at the moment. gpus( mem_per_gpuitime cpus_per_gpuN cpus_per_tasklearnlab partitioncommentdefault_factorysetupmax_num_timeout constraintFone_task_per_nodearray_parallelismexcludeqosaccount)__name__ __module__ __qualname____doc__r%int__annotations__r'floatr(r*r+tpOptionalr-rr.rlistr1Listr3r5r6boolr8r9r:r;rrr#r#.s@D#MKD#L#&*M2;;s#*Is $GR[[ $5E2773<5OSJ#t# s $GR[[ $ CS  $GR[[ $rr#cJeZdZUdZdZeed<dZeed<dZeed<dZ eed<y) SubmitRulesa: Submit rules describe in which case Shepherd will schedule new jobs. Args: retry (bool): if true, all failed or canceled jobs will be rescheduled. update_pending (bool): if true, all pending jobs whose Slurm parameters have changed will be replaced. update (bool): if true, all pending or running jobs whose Slurm parameters have changed will be replaced. replace (bool): if true, all running jobs will be replaced by new jobs. replace_done (bool): if true, all done jobs will be relaunched. Fretryupdatereplace replace_doneN) r<r=r>r?rKrGrArLrMrNrHrrrJrJas1 E4FDGTL$rrJcfeZdZUdZdZeed<dZeed<dZeed<dZ eed<dZ eed <d Z eed <y ) ShepConfigz Configuration for Shepherd. Mostly naming conventions for folders and files. There should be little reasons to change that. zjob.pkljob_fileby_idorphanssubmititsubmitit_folderlatestlatest_submititarraysN) r<r=r>r?rQrrArRrSrUrWrXrHrrrPrPvsEHcE3GS%OS%#OS#FCrrPcReZdZUdZedZeed<eeZ e je ed<dZ eed<dZe j eed <dZe j e ed <d Ze ed <d Ze ed <eeZeed<dZe ed<dZeed<dZe ed<dZe ed<de defdZfdZxZS) DoraConfiga< Main Dora configuration. The main parameters to change are the following. Args: dir (Path or str): path where Dora will save all useful informations, logs. This is also where you should store your checkpoints (see `dora.xp.XP`). exclude (List[str]): list of patterns of argument names to ignore when computing the XP signature and doing deduplication. For instance 'num_workers', etc. git_save (bool): when True, experiments can only be scheduled from a clean repo. A shallow clone of the repo will be made and execution will happen from there. This does not impact `dora run` unless you pass the `--git_save` flag. shared (Path or None): if provided, the path to a central repository of XPs. For the moment, this only supports sharing hyper-params, logs etc. will stay in the per user folder. grid_package (str or None): if provided, package to look for grids. Default to the package with the `train.py` module followed by `.grids`. z ./outputsdirr/r9Fgit_saveNshared grid_packagez history.jsonhistoryxpsshepzrendezvous.txtrendezvous_fileuse_rendezvousgrids_gridscodes_codesarg_namereturncB|jD]}t||syy)zUReturn True if the given argument name should be excluded from the signature.TF)r9r)selfrhpatterns r is_excludedzDoraConfig.is_excludeds)|| Gx) rc`|dvrddlm}|t||}t|||y)N)r[r]r$)to_absolute_path)r\rorsuper __setattr__)rknamerrors rrqzDoraConfig.__setattr__s4 $ $ 2 -e45 D%(r)r<r=r>r?rr[rArrEr9rCrFrr\rGr]rDr^r_r`rPrarbrcrergrmrq __classcell__)rs@rrZrZs$[!C!!$7GRWWS\7Hd $FBKK $%)L"++c")"GS!CZ8D*8+OS+ ND FCFCCD))rrZ)r?argparser dataclassesrrrpathlibrtypingrComegaconf.dictconfigr omegaconfr Anyrr!r#rJrPrZrHrrr{s(+*266** FBFF F F /%/% /%d  (      1)1) 1)r