Aller au contenu principal

Implement your own submitter

In this tutorial we'll be implementing our own submitter in Silex for the compositing software Natron.

Natron is an open source compositing software very similar to Nuke. The goal is to submit compositing scenes and output images with a write node.

Rez package and executable

The first thing to do is to make sure that the same version of Natron will be used everywhere on the farm. This is useful because we don't want inconsistencies between renders.

To do that we need to add a natron Rez package.

First download Natron's executable for Windows. Currently the latest version is 2.4.3.

We will put the executable on the network so it's not necessary to install it on every machine. It's fine since it's not that big and runs on the render farm.

Then create a package in \\rez-network-location\silex-rez\packages\dcc:

natron
├── platform-windows
│   ├── natron_env.py
│ └── Natron-2.4.3-Windows-x86_64
│ ├── bin
│ └── ...
└── package.py
package.py
name = "natron"
version = "2.4.3"

tools = [
"Natron",
"NatronRenderer"
]

variants = [
["platform-windows"]
]

def commands():
import sys
sys.path.append(root)
import natron_env
natron_env.commands(env, root)
platform-windows/natron_env.py
def commands(env, root):
env.PATH.prepend("{root}/Natron-2.4.3-Windows-x86_64/bin")
tip

We use env.PATH.prepend here because we want the executable to take priority over a local installed version.

info

platform-windows is an implicit package, that's why we use a variant in the package so it's resolved automatically when we say rez env natron on Windows. It will add to the path the executable path on the network.

Now you should be able to launch Natron's GUI with:

$ rez env natron -- natron

Command line usage 🖥️

Next step is to see how Natron can be used as a command line tool without an interface.

Check the documentation: https://natron.readthedocs.io/en/rb-2.5/devel/natronexecution.html

Natron has 3 different execution modes:

  • The execution of Natron projects (.ntp)
  • The execution of Python scripts that contain commands for Natron
  • An interpreter mode where commands can be given directly to the Python interpreter

We can see that there is a special NatronRenderer executable that does Natron -background automatically. It's perfect since we want to do batch rendering on the farm.

A basic command would be:

$ rez env natron -- natronrenderer -w WriteNode out.####.exr 1-10 project.ntp

We need to specify:

  • The name of the write node (WriteNode)
  • The destination path of the rendered images (out.####.exr). Notice the four # to indicate the frame numbering like out.0001.exr.
  • The frame range to render (1-10)
  • The scene to render (project.ntp)

With a real world example (download test file here):

$ rez env natron -- natronrenderer -w Write1 ./out.####.exr 1-10 .\project.ntp

Restoring the image cache...
Loading plugin cache...
Info: init.py script not loaded (this is not an error)
Loading PyPlugs...
Loading project: C:/Users/etudiant/Desktop/project.ntp
Write1 ==> Rendering started
Write1 ==> Frame: 1, Progress: 10.0%, 7.4 Fps, Time Remaining: 1 seconds
Write1 ==> Frame: 3, Progress: 20.0%, 7.0 Fps, Time Remaining: 1 seconds
Write1 ==> Frame: 2, Progress: 30.0%, 10.3 Fps, Time Remaining: 0 second
Write1 ==> Frame: 4, Progress: 40.0%, 8.9 Fps, Time Remaining: 0 second
Write1 ==> Frame: 5, Progress: 50.0%, 9.9 Fps, Time Remaining: 0 second
Write1 ==> Frame: 6, Progress: 60.0%, 11.3 Fps, Time Remaining: 0 second
Write1 ==> Frame: 7, Progress: 70.0%, 9.4 Fps, Time Remaining: 0 second
Write1 ==> Frame: 8, Progress: 90.0%, 10.5 Fps, Time Remaining: 0 second
Write1 ==> Frame: 9, Progress: 90.0%, 11.8 Fps, Time Remaining: 0 second
Write1 ==> Frame: 10, Progress: 100.0%, 13.1 Fps, Time Remaining: 0 second
Write1 ==> Rendering finished

Create the action ✔️

When the user clicks on the Submit action, it launches the action defined by silex_client/config/action/submit.yml which then inserts the proper submit action for the chosen software.

Simple action

First we are going to define a new action in the silex_client/config/submit directory.

This is going to be a simple action without user input. We will just create tasks and give them to the command SubmitToTractorCommand:

silex_client/config/submit/natron.yml
natron:
label: "Submit Natron scene"
steps:
natron_render:
label: "Setup render parameters"
index: 20
commands:
build_natron_tasks:
path: "silex_client.commands.farm.natron_render_tasks.NatronRenderTasksCommand"
label: "Natron Job parameters"

submit_to_tractor:
label: "Submit"
path: "silex_client.commands.farm.submit_to_tractor.SubmitToTractorCommand"
ask_user: true
parameters:
tasks:
value: !command-output "natron_render:build_natron_tasks:tasks"
job_title:
value: !command-output "natron_render:build_natron_tasks:file_name"
job_tags:
value:
- "natron"
silex_client/commands/farm/natron_render_tasks.py
from __future__ import annotations

import logging
import typing
from typing import Any, Dict

from silex_client.action.command_base import CommandBase
from silex_client.utils import farm

# Forward references
if typing.TYPE_CHECKING:
from silex_client.action.action_query import ActionQuery


class NatronRenderTasksCommand(CommandBase):
"""
Construct Natron render commands
See: https://natron.readthedocs.io/en/rb-2.5/devel/natronexecution.html
"""

@CommandBase.conform_command()
async def __call__(
self,
parameters: Dict[str, Any],
action_query: ActionQuery,
logger: logging.Logger,
):
# Use the command we used earlier with Rez
command = r"rez env natron -- natronrenderer -w Write1 P:\test_pipe\test\render\out.####.exr 1-10 P:\test_pipe\test\project.ntp"

# Create a farm Task by passing a list of arguments
tasks = [farm.Task("1-10", argv=command.split(" "))]

# Returning the results from the command
return {"tasks": tasks, "file_name": "project.ntp"}
info

Notice that the Natron scene was put on the P: drive because it needs to be synchronized in order for the computers on the farm to have access to the files.

For now we manually provide a command that will be executed in a single task on the farm. We use the silex_client.utils.farm.Task class that directly accepts a list of arguments and will create a command.

It's equivalent to:

task = farm.Task("1-10")
task.addCommand(farm.Command(argv=command.split(" ")))
tasks = [task]

Launching the job

After launching a Submit action (from the Maya shelf for example), press Continue until the action is over.

If everything goes well, go to the Tractor dashboard to see the result:

Double clicking on the Task (red rectangle), you get something like this:

====[2022/06/08 11:54:23 /J8451/T1/C1.2/jhenry@i7-mk8-2017-38 on md12-2021-002 ]====

Error while loading OpenGL: WGL: The driver does not appear to support OpenGL
OpenGL rendering is disabled.
WGL: The driver does not appear to support OpenGL
WGL: The driver does not appear to support OpenGL
ERROR: Natron: P:\test_pipe\test\project.ntp: No such file.
Restoring the image cache...
Loading plugin cache...
Info: init.py script not loaded (this is not an error)
Loading PyPlugs...
Why is there an error?

There's an error because the computer on the farm (md12-2021-002) can't access files on the P: drive.

Depending on the NAS where the project files are, we need to mount a network drive pointing to that location.

Wrapping with the mount command

To mount the network drive, we use the following Rez package:

$ rez env mount_render_drive -- mount_rd_drive ana

The problem is that creating two commands on Tractor is causing issues...

So we use a helper to wrap the command with the mount command. To do that we first need to use the CommandBuilder class that uses the builder pattern:

silex_client/commands/farm/natron_render_tasks.py
from silex_client.utils import command_builder, farm

class NatronRenderTasksCommand(CommandBase):
"""
Construct Natron render commands
See: https://natron.readthedocs.io/en/rb-2.5/devel/natronexecution.html
"""

@CommandBase.conform_command()
async def __call__(
self,
parameters: Dict[str, Any],
action_query: ActionQuery,
logger: logging.Logger,
):
command = (
command_builder.CommandBuilder(
"natronrenderer", rez_packages=["natron"], delimiter=None
)
.param("w", ["Write1", r"P:\test_pipe\test\render\out.####.exr", "1-10"])
.value(r"P:\test_pipe\test\project.ntp")
)

command = farm.wrap_with_mount(command, "ana")

task = farm.Task("1-10")
task.addCommand(command)

return {"tasks": [task], "file_name": "project.ntp"}

Which results in the following command:

$ rez env cmd_wrapper -- cmd-wrapper "--pre=\"rez env mount_render_drive -- mount_rd_drive ana\"" "--cmd=\"rez env natron -- natronrenderer -w Write1 P:\\test_pipe\\test\\render\\out.####.exr 1-10 P:\\test_pipe\\test\\project.ntp\""

and the logs:

====[2022/06/08 14:10:23 /J8459/T1/C1.1/jhenry@i7-mk8-2017-38 on md7-2016-048 ]====

----------------------------- PRE COMMAND 0 BEGIN -----------------------------
DELETE *
Vous poss�dez les connexions � distance suivantes�:
P: \\tars\PIPELINE
La poursuite de cette op�ration va rompre les connexions.
La commande s'est termin�e correctement.
---------------------------------
NET USE P
La commande s'est termin�e correctement.
---------------------------------
net use successful
AFTER MOUNT P
Les nouvelles connexions ne seront pas m�moris�es.
�tat Local Distant R�seau
-------------------------------------------------------------------------------

OK P: \\ana\PIPELINE Microsoft Windows Network

La commande s'est termin�e correctement.

----------------- PRE COMMAND 0 END in 0h 0m 1s (exit code: 0) -----------------

------------------------------ MAIN COMMAND BEGIN ------------------------------
Error while loading OpenGL: WGL: The driver does not appear to support OpenGL
OpenGL rendering is disabled.
WGL: The driver does not appear to support OpenGL
WGL: The driver does not appear to support OpenGL
Clearing the image cache...
Loading plugin cache...
Info: init.py script not loaded (this is not an error)
Loading PyPlugs...
Loading project: P:/test_pipe/test/project.ntp

Write1 ==> Rendering started
Write1 ==> Frame: 1, Progress: 10.0%, 8.0 Fps, Time Remaining: 1 seconds
Write1 ==> Frame: 2, Progress: 20.0%, 8.0 Fps, Time Remaining: 0 second
Write1 ==> Frame: 3, Progress: 30.0%, 11.3 Fps, Time Remaining: 0 second
Write1 ==> Frame: 7, Progress: 40.0%, 8.8 Fps, Time Remaining: 0 second
Write1 ==> Frame: 4, Progress: 50.0%, 10.7 Fps, Time Remaining: 0 second
Write1 ==> Frame: 6, Progress: 60.0%, 12.4 Fps, Time Remaining: 0 second
Write1 ==> Frame: 5, Progress: 70.0%, 14.5 Fps, Time Remaining: 0 second
Write1 ==> Frame: 10, Progress: 80.0%, 12.5 Fps, Time Remaining: 0 second
Write1 ==> Frame: 8, Progress: 90.0%, 13.7 Fps, Time Remaining: 0 second
Write1 ==> Frame: 9, Progress: 100.0%, 15.2 Fps, Time Remaining: 0 second
Write1 ==> Rendering finished
----------------- MAIN COMMAND END in 0h 0m 8s (exit code: 0) -----------------

====[2022/06/08 14:10:33 process complete, exit code: 0]====

It works!!!! 🎉🎉🎉

Adding parameters

Single parameter

Now the thing is that our submitter only renders the same scene and same frames all the time... 🤔

First we need the user to select his project file in the submitter. For that we specify a parameters dictionnary and use the TaskFileParameterMeta parameter type:

silex_client/commands/farm/natron_render_tasks.py
from silex_client.utils.parameter_types import TaskFileParameterMeta

# Forward references
if typing.TYPE_CHECKING:
from silex_client.action.action_query import ActionQuery


class NatronRenderTasksCommand(CommandBase):
"""
Construct Natron render commands
See: https://natron.readthedocs.io/en/rb-2.5/devel/natronexecution.html
"""

parameters = {
"scene_file": {
"label": "Project file",
"type": TaskFileParameterMeta(extensions=[".ntp"]),
},
}

In the interface it will show a file explorer component so the user can select his scene to submit. It only shows .ntp files as specified in the parameter type.

Then we use that parameter in the command code:

silex_client/commands/farm/natron_render_tasks.py
scene_file: pathlib.Path = parameters["scene_file"]

command = (
command_builder.CommandBuilder(
"natronrenderer", rez_packages=["natron"], delimiter=None
)
.param("w", ["Write1", r"P:\test_pipe\test\render\out.####.exr", "1-10"])
.value(scene_file.as_posix())
)
info

Notice that we specified the type scene_file: pathlib.Path. Parameters are automatically casted in their own types when given to a command.

caution

Use PurePath.as_posix() method to convert any Windows path into standard POSIX path with forward slashes. This can cause issues on the farm when parsing the command otherwise.

Image output path

Now that we can provide the scene, the rendered image destination will depend on the task the same way. For that we use the BuildOutputPath command. We need to add it to the YAML action definition:

silex_client/config/submit/natron.yml
natron:
label: "Submit Natron scene"
steps:
build_output_path:
label: "Build output path"
index: 10
commands:
select_extension:
label: "Output extension"
path: "silex_client.commands.select_list.SelectList"
parameters:
param_name: "Output extension"
parameters_list:
- "exr"
- "png"
- "jpg"
- "tiff"

build_output_path:
label: "Build output path"
path: "silex_client.commands.build_output_path.BuildOutputPath"
ask_user: true
parameters:
create_temp_dir: false
create_output_dir: false
output_type:
value: !command-output "build_output_path:select_extension"
hide: true
use_current_context:
value: true
hide: true
name:
value: "render"

# ...

It will first ask for an extension and then build the path. Use it by connecting outputs and inputs with the YAML !command-output directive:

silex_client/config/submit/natron.yml
natron:
steps:
natron_render:
commands:
build_natron_tasks:
ask_user: true
parameters:
output_directory:
value: !command-output "build_output_path:build_output_path:directory"
output_filename:
value: !command-output "build_output_path:build_output_path:file_name"
output_extension:
value: !command-output "build_output_path:select_extension"
silex_client/commands/farm/natron_render_tasks.py
class NatronRenderTasksCommand(CommandBase):
"""
Construct Natron render commands
See: https://natron.readthedocs.io/en/rb-2.5/devel/natronexecution.html
"""

parameters = {
"output_directory": {"type": pathlib.Path, "hide": True},
"output_filename": {"type": str, "hide": True},
"output_extension": {"type": str, "hide": True},
"scene_file": {
"label": "Project file",
"type": TaskFileParameterMeta(extensions=[".ntp"]),
},
"write_node": {"type": str},
}

@CommandBase.conform_command()
async def __call__(
self,
parameters: Dict[str, Any],
action_query: ActionQuery,
logger: logging.Logger,
):
output_directory: pathlib.Path = parameters["output_directory"]
output_filename: str = parameters["output_filename"]
output_extension: str = parameters["output_extension"]

scene_file: pathlib.Path = parameters["scene_file"]
write_node: str = parameters["write_node"]

output_path = (
output_directory
/ write_node
/ f"{output_filename}_{write_node}.####.{output_extension}"
)

command = (
command_builder.CommandBuilder(
"natronrenderer", rez_packages=["natron"], delimiter=None
)
.param("w", [write_node, output_path, "1-10"])
.value(scene_file.as_posix())
)

command = farm.wrap_with_mount(command, "ana")

task = farm.Task("1-10")
task.addCommand(command)

return {"tasks": [task], "file_name": scene_file.stem}

Which will result in this output path (by selecting png for example):

P:\test_pipe\shots\s03\p060\layout_main\publish\v000\png\render\Write1\test_pipe_s03_p060_layout_main_publish_v000_render_Write1.####.png

Frame range splitting

The last thing to do is to use the full power of a render farm: parallelization.

In short: spread the computation over multiple computers and make them run simultaneously.

To do that we need to split the frame range the artist wants to render, let's say 1-500 into chunks of frames. Like 10 frames on each computer.

To represent frame range expressions, we use the FrameSet class from the Fileseq Python library. It allows representing complex range expressions like 1-10, 50-80x2 (with paddings...).

We provide a FrameSet parameter and a task_size which will define the number of frames per task/computer:

silex_client/commands/farm/natron_render_tasks.py
from __future__ import annotations

import logging
import pathlib
import typing
from typing import Any, Dict, List

from fileseq import FrameSet
from silex_client.action.command_base import CommandBase
from silex_client.utils import command_builder, farm, frames
from silex_client.utils.parameter_types import TaskFileParameterMeta

# Forward references
if typing.TYPE_CHECKING:
from silex_client.action.action_query import ActionQuery


class NatronRenderTasksCommand(CommandBase):
"""
Construct Natron render commands
See: https://natron.readthedocs.io/en/rb-2.5/devel/natronexecution.html
"""

parameters = {
"scene_file": {
"label": "Project file",
"type": TaskFileParameterMeta(extensions=[".ntp"]),
},
"output_directory": {"type": pathlib.Path, "hide": True},
"output_filename": {"type": str, "hide": True},
"output_extension": {"type": str, "hide": True},
"frame_range": {
"label": "Frame range",
"type": FrameSet,
"value": "1-50x1",
},
"task_size": {
"label": "Task size",
"tooltip": "Number of frames per computer",
"type": int,
"value": 10,
},
"write_node": {"type": str},
}

@CommandBase.conform_command()
async def __call__(
self,
parameters: Dict[str, Any],
action_query: ActionQuery,
logger: logging.Logger,
):
output_directory: pathlib.Path = parameters["output_directory"]
output_filename: str = parameters["output_filename"]
output_extension: str = parameters["output_extension"]

scene_file: pathlib.Path = parameters["scene_file"]
write_node: str = parameters["write_node"]
frame_range: FrameSet = parameters["frame_range"]
task_size: int = parameters["task_size"]

output_path = (
output_directory
/ write_node
/ f"{output_filename}_{write_node}.####.{output_extension}"
)

natron_cmd = command_builder.CommandBuilder(
"natronrenderer", rez_packages=["natron"], delimiter=None
)

tasks: List[farm.Task] = []
frame_chunks = frames.split_frameset(frame_range, task_size)

for chunk in frame_chunks:
# Copy the initial command
chunk_cmd = natron_cmd.deepcopy()

chunk_cmd.param("w", [write_node, output_path, str(chunk)])
chunk_cmd.value(scene_file.as_posix())

command = farm.wrap_with_mount(
chunk_cmd, action_query.context_metadata["project_nas"]
)

task = farm.Task(str(chunk))
task.addCommand(command)
tasks.append(task)

return {"tasks": tasks, "file_name": scene_file.stem}

tip

We use frames.split_frameset to split a FrameSet expression into equal chunks.

info

Notice that we used action_query.context_metadata["project_nas"] in the wrap_with_mount function call. The project NAS is stored in the action context so we use that instead of an explicit value.

Handling frame padding

When using a frame range with a frame padding: 1-10x2 and a task size of 3 will give ["1-5x2", "7,9"] but the issue is that Natron doesn't understand -w Write1 out.###.exr 1-5x2 ....

Using rez env natron -- natronrenderer -h, we can see the syntax required:

One or more frame ranges, separated by commas. Each frame range must be one of the following:

  • <frame> (a single frame number, e.g. 57)
  • <firstFrame>-<lastFrame> (e.g. 10-40)
  • <firstFrame>-<lastFrame>:<frameStep> (e.g. 1-10:2 would render 1,3,5,7,9) Examples: 1-10:1,20-30:2,40-50 1329,2450,123,1-10:2

So we can easily replace any x character by : in 1-50x2 -> 1-50:2.

silex_client/commands/farm/natron_render_tasks.py
chunk_range = str(chunk).replace("x", ":")
chunk_cmd.param("w", [write_node, output_path, chunk_range])

Now it works perfectly!! 👌👌

Final natron.yml action definition file

natron:
label: "Submit Natron scene"
steps:
build_output_path:
label: "Build output path"
index: 10
commands:
select_extension:
label: "Output extension"
path: "silex_client.commands.select_list.SelectList"
parameters:
param_name: "Output extension"
parameters_list:
- "exr"
- "png"
- "jpg"
- "tiff"

build_output_path:
label: "Build output path"
path: "silex_client.commands.build_output_path.BuildOutputPath"
ask_user: true
parameters:
create_temp_dir: false
create_output_dir: false
output_type:
value: !command-output "build_output_path:select_extension"
hide: true
use_current_context:
value: true
hide: true
name:
value: "render"

natron_render:
label: "Setup render parameters"
index: 20
commands:
build_natron_tasks:
path: "silex_client.commands.farm.natron_render_tasks.NatronRenderTasksCommand"
label: "Natron Job parameters"
ask_user: true
parameters:
output_directory:
value: !command-output "build_output_path:build_output_path:directory"
output_filename:
value: !command-output "build_output_path:build_output_path:file_name"
output_extension:
value: !command-output "build_output_path:select_extension"

submit_to_tractor:
label: "Submit"
path: "silex_client.commands.farm.submit_to_tractor.SubmitToTractorCommand"
ask_user: true
parameters:
tasks:
value: !command-output "natron_render:build_natron_tasks:tasks"
job_title:
value: !command-output "natron_render:build_natron_tasks:file_name"
job_tags:
value:
- "natron"

Final natron_render_tasks.py command file

from __future__ import annotations

import logging
import pathlib
import typing
from typing import Any, Dict, List

from fileseq import FrameSet
from silex_client.action.command_base import CommandBase
from silex_client.utils import command_builder, farm, frames
from silex_client.utils.parameter_types import TaskFileParameterMeta

# Forward references
if typing.TYPE_CHECKING:
from silex_client.action.action_query import ActionQuery


class NatronRenderTasksCommand(CommandBase):
"""
Construct Natron render commands
See: https://natron.readthedocs.io/en/rb-2.5/devel/natronexecution.html
"""

parameters = {
"scene_file": {
"label": "Project file",
"type": TaskFileParameterMeta(extensions=[".ntp"]),
},
"output_directory": {"type": pathlib.Path, "hide": True},
"output_filename": {"type": str, "hide": True},
"output_extension": {"type": str, "hide": True},
"frame_range": {
"label": "Frame range",
"type": FrameSet,
"value": "1-50x1",
},
"task_size": {
"label": "Task size",
"tooltip": "Number of frames per computer",
"type": int,
"value": 10,
},
"write_node": {"type": str, "value": "Write1"},
}

@CommandBase.conform_command()
async def __call__(
self,
parameters: Dict[str, Any],
action_query: ActionQuery,
logger: logging.Logger,
):
output_directory: pathlib.Path = parameters["output_directory"]
output_filename: str = parameters["output_filename"]
output_extension: str = parameters["output_extension"]

scene_file: pathlib.Path = parameters["scene_file"]
write_node: str = parameters["write_node"]
frame_range: FrameSet = parameters["frame_range"]
task_size: int = parameters["task_size"]

output_path = (
output_directory
/ write_node
/ f"{output_filename}_{write_node}.####.{output_extension}"
)

natron_cmd = command_builder.CommandBuilder(
"natronrenderer", rez_packages=["natron"], delimiter=None
)

tasks: List[farm.Task] = []
frame_chunks = frames.split_frameset(frame_range, task_size)

for chunk in frame_chunks:
# Copy the initial command
chunk_cmd = natron_cmd.deepcopy()

chunk_range = str(chunk).replace("x", ":")
chunk_cmd.param("w", [write_node, output_path, chunk_range])
chunk_cmd.value(scene_file.as_posix())

command = farm.wrap_with_mount(
chunk_cmd, action_query.context_metadata["project_nas"]
)

task = farm.Task(str(chunk))
task.addCommand(command)
tasks.append(task)

return {"tasks": tasks, "file_name": scene_file.stem}

Go further 🚀

Some ideas to improve the submitter: